At present E2E tests are more of a burden on the team than they are helping with quality, due to instability which I believe was mainly caused by parallel workers, but these problems were hidden by the two-retries policy.
This PR fixes some of the stability issues but also argues for following the testing philosophy "fail fast".
Rather than "retrying flaky tests" I propose seeing any test failure, even intermittent, as a sign of a potential problem. The right thing to do with a potential problem is not, as it were, to sweep it under the rug, but to fix it for the future.
If a test is truly flaky then it needs to be fixed or removed - E2E tests need to pull their weight, given their cost in terms of longer feedback loops.
Proposal
This PR
rolls back the "multiple workers in CI " change made in #31 (which is redundant with sharding anyway)
disables retries - if tests fail, we want to diagnose and fix the failure
saves Playwright traces (which were super helpful in diagnosing these problems)
saves server logs alongside traces for ease of debugging
adds a QR fix to the occasional failure in "Add dummy data", caused by FactoryGirl sometimes creating duplicate domains
Purpose
At present E2E tests are more of a burden on the team than they are helping with quality, due to instability which I believe was mainly caused by parallel workers, but these problems were hidden by the two-retries policy.
This PR fixes some of the stability issues but also argues for following the testing philosophy "fail fast".
Rather than "retrying flaky tests" I propose seeing any test failure, even intermittent, as a sign of a potential problem. The right thing to do with a potential problem is not, as it were, to sweep it under the rug, but to fix it for the future.
If a test is truly flaky then it needs to be fixed or removed - E2E tests need to pull their weight, given their cost in terms of longer feedback loops.
Proposal
This PR
Fixes #409 (hopefully)