dart-lang / native

Dart packages related to FFI and native assets bundling.
BSD 3-Clause "New" or "Revised" License
157 stars 44 forks source link

[ffigen][objective_c] test/native_objc_test/global_test.dart flaky #1435

Open dcharkes opened 3 months ago

dcharkes commented 3 months ago

While reviewing https://github.com/dart-lang/native/pull/1434/:

00:03 +13 -1: test/native_objc_test/automated_ref_count_test.dart: Automatic reference counting assign properties ref count correctly [E]                                                                                                                                                   
  Expected: <1>
    Actual: <0>

After rerunning the test passed.

cc @liamappelbe

liamappelbe commented 3 months ago

I've fixed all the flakes I could find, but there are some that I can't repro locally. I've done my best to fix those too, but I might have missed some. Please reopen or file another issue if there are more flakes on CI.

liamappelbe commented 3 months ago

global_test and global_native_test are still flaky. Can't repro locally.

https://github.com/dart-lang/native/actions/runs/10605979638/job/29397792224

liamappelbe commented 1 month ago

Might have gotten less flaky, but it's not fixed: https://github.com/dart-lang/native/actions/runs/11470847451/job/31920895855?pr=1673

liamappelbe commented 1 month ago

It passes 10 runs in a row pretty easily: https://github.com/dart-lang/native/actions/runs/11471378290/job/31922174382?pr=1674

I wonder if the build or other setup is flaky. I've changed the workflow so that it completes all 10 runs even if one fails. Maybe we'll see that it's still ~30% flaky, but when it does flake it fails all 10 runs. That would explain why my "run 10 times" strategy failed to detect the flake in #1666.

liamappelbe commented 1 month ago

Yep, finally got a flake, and all 10 test runs failed. Probably the build is flaky. Next step is to try and get the build artifacts from CI and see whats going on.