This PR attempts to cleanup, refactor, and optimize the settings of our end to end tests to minimized flakiness and errors, both locally and on github. Particular attention is paid to reducing the odds that a dev needs to "rerun" the tests in order to force a github pass, as well as to make sure the tests simply work on run when executing locally. With the browser switch already in place, I used the video/trace logs produced by the e2e's (which are absolutely fantastic and get captured for any end to end test that fails at least once) to create a grid-search optimization table to find the best combination of parameters with which to run the tests. This PR introduces the following changes:
pins the number of retries for each test to 2, since some tests involving public database calls are flaky and work on subsequent reruns (expected behavior)
pins the number of parallel workers for all tests and runs to 2, since this reduces timeout probability on github while minimizing impact to runner actions
eliminates manual field setting after using the FILL FIELDS patient option in test workflows
marks the customize query tests as natively .slow() to allow for additional timeout leniency and suppress warnings (which are now gone across the board for these tests)
augments global .expect() timeouts to allow for different browsers to take more time with display visibility
fixes the disabling logic of the submit query button after the query select dropdown--previously, the way we had our state setting and isSubscribed variables configured means that a query could be selected but the DB query valueset fetch hasn't returned yet. This is now impossible, and the button is fully disabled until a query is selected and the app is no longer loading value sets, a state which is possible only after all isSusbcribed followers have been set
refactors large scale table row count of 32 rows into concrete sections of specific counts per clinical service type, both to reduce flakiness and allow rendering time and to pinpoint an error source if the test does break due to the public server not returning medication requests / diagnostic reports
Related Issue
No issue was created for this, but this is my final attempt at cleaning up local e2e development troubles once and for all
Additional Information
I ran the tests on this github page four separate times after my latest commit and all four passed with no retries or breaks in between, so this configuration seems like something github can handle.
Reducing e2e Flakiness
Summary
This PR attempts to cleanup, refactor, and optimize the settings of our end to end tests to minimized flakiness and errors, both locally and on github. Particular attention is paid to reducing the odds that a dev needs to "rerun" the tests in order to force a github pass, as well as to make sure the tests simply work on run when executing locally. With the browser switch already in place, I used the video/trace logs produced by the e2e's (which are absolutely fantastic and get captured for any end to end test that fails at least once) to create a grid-search optimization table to find the best combination of parameters with which to run the tests. This PR introduces the following changes:
.slow()to allow for additional timeout leniency and suppress warnings (which are now gone across the board for these tests).expect()timeouts to allow for different browsers to take more time with display visibilityisSubscribedvariables configured means that a query could be selected but the DB query valueset fetch hasn't returned yet. This is now impossible, and the button is fully disabled until a query is selected and the app is no longer loading value sets, a state which is possible only after allisSusbcribedfollowers have been setRelated Issue
No issue was created for this, but this is my final attempt at cleaning up local e2e development troubles once and for all
Additional Information
I ran the tests on this github page four separate times after my latest commit and all four passed with no retries or breaks in between, so this configuration seems like something github can handle.