Fix intermittents tests for macos desktop

google / webcrypto.dart

Cross-platform implementation of Web Cryptography APIs

https://pub.dev/packages/webcrypto

Apache License 2.0

81 stars 48 forks source link

Fix intermittents tests for macos desktop #76

Open jonasfj opened 10 months ago

jonasfj commented 10 months ago

.github/workflows/test.yml contains:

      # TODO: Enable macos desktop when supported
      #- run: flutter test integration_test/webcrypto_test.dart -d macos
      #  working-directory: ./example

We really should enable integration tests on macos desktop. If anyone is interested in figuring how to get this working on Github Actions that would be great.

Contributions are highly appreciated, I tried in https://github.com/google/webcrypto.dart/pull/74, but there is probably a tiny thing not working.

CoreNion commented 10 months ago

On the actual machine, the test appears to get stuck on the step 'test app can compute a hash' if the example app is not the active window. The test passes when the app is open, which is why I did not notice the issue. This part runs testWidgets(), but it is likely not being handled by the app GUI on GitHub Action and is causing the test to hang?

jsiedentop commented 10 months ago

I can confirm that the test app can compute a hash hangs on my mac if the example application is in the background during the test (Mac M1, macOS 14.1 23B74 darwin-arm64). However I was able to run the test.yml Action successfully on GitHub, without any relevant change:

jonasfj commented 10 months ago

Merged https://github.com/google/webcrypto.dart/pull/77 and closing this for now.

If someone is interested in trying to figure out if this is an flaky, and/or what we could do to make it work better, please go ahead.

Contributions are welcome, please ping me @jonasfj for reviews (I'm sometimes slow to notice PRs).

jsiedentop commented 10 months ago

At least it seems, that the library itself seems to work like expected.

If the cause of the flaky test is really, that the application looses focus, then I would investigate:

if the test should work without focus?
why the app looses focus at all?
is there a way to rewrite the test to make it more stable.

I would argue, that it’s a separate topic. I keep it in mind and maybe will come back to this, if it turns out to be a big problem. However, sporadic errors that occur in the pipeline could be quite difficult to isolate.

jonasfj commented 10 months ago

I just landed https://github.com/google/webcrypto.dart/pull/78 and will publish.

But I had to manually rerun the tests to get it to pass.

First time it timed out after 12 minutes.
Second time it finished in 2 minutes.

Ideas for what we can do to make flutter test integration_test/webcrypto_test.dart -d macos reliable are welcome.

The flutter part of the example app we have here are really dumb and simple. Nothing fancy, so this could be a bug in Flutter somewhere. Or it could be a bug in flutter integration test logic.