search

ecamp / ecamp3

eCamp v3 is a web-based app for camp and course planning. The application is specialized for camps and courses of youth associations and for Y+S offers in the sport of camp sports/trekking.
https://ecamp3.ch
GNU Affero General Public License v3.0
110 stars 48 forks source link

Improve stability of e2e tests #5493

Open BacLuc opened 1 week ago

BacLuc commented 1 week ago

Ways we could improve the stability of e2e tests

Improve the run environment

Thesis: The cypress github action is flaky, the docker images are better

Does not seem that way.

Thesis: they are flaky because they don't have enough RAM

Does not seem that way

Thesis: they are flaky because they don't have enough CPU

The feeling is that the tests would then just run longer and run into timeouts. But this seems not the case.

Thesis: when we use the cypress retry mechanism, they will be more stable

It does not solve the problem, but renovate can merge and we can work.

Thesis: the frankenphp container stops responding

(Which leads to the connection refused for the login request errors) -> try enable strace: https://github.com/BacLuc/ecamp3/tree/debug-frankenphp-strace add debug log to caddy: https://github.com/BacLuc/ecamp3/tree/enable-caddy-debug

Improve the single tests

The inspection of the test errors is in this Issue: #5322

Thesis: Not all tests are flaky

manuelmeister commented 1 week ago

Thesis: they are flaky because browsers are flaky => use stable browser version Thesis: they are flaky because Cypress is flaky => use Playwright?