search

Concordium / concordium-ledger-app

Apache License 2.0
4 stars 1 forks source link

Sporadically failing end2end tests #62

Closed orhoj closed 8 months ago

orhoj commented 2 years ago

Bug Description Sometimes the end to end tests fail even though there is no issue with the production code. This seems to be because of a bug in Speculos (or Zemu) where some screens are sometimes not rendered correctly, i.e. the second line of text on a screen is blank even though it should contain some text.

Steps to Reproduce Run end to end tests as a Github action a number of times and one of them will fail.

Expected Result The tests should be stable, i.e. no changes to the code base should result in the tests staying green.

Actual Result The test suite fails on a mismatch of images due to a snapshot having a blank line that we expected to contain some text.

shjortConcordium commented 1 year ago

If the issue is with Zemu and not Speculos itself, then the solution might be to migrate the tests to use https://github.com/LedgerHQ/ragger instead.

orhoj commented 1 year ago

It seems that our main issue is that waitUntilScreenIsNot can resolve before e.g. the "Review transaction" screen is fully shown, and it might only show the top line "Review". So the screenshot comparison fails as the screenshot is not as expected. To fix it we have to wait in a different way as it seems that function is inconsistent, especially when run as a GitHub action. There is a timing issue present.

orhoj commented 8 months ago

This was mostly solved by adding a wait. If the problem arises again, then the tests should be redone to not depend on waitUntilScreenIsNot, but instead to wait for the expected next screen. But that is a cumbersome task.