search

ecamp / ecamp3

eCamp v3 is a web-based app for camp and course planning. The application is specialized for camps and courses of youth associations and for Y+S offers in the sport of camp sports/trekking.
https://ecamp3.ch
GNU Affero General Public License v3.0
123 stars 54 forks source link

Improve stability of e2e tests #5493

Open BacLuc opened 5 months ago

BacLuc commented 5 months ago

Ways we could improve the stability of e2e tests

Improve the run environment

Thesis: The cypress github action is flaky, the docker images are better

Does not seem that way.

Thesis: they are flaky because they don't have enough RAM

Does not seem that way

Thesis: they are flaky because they don't have enough CPU

The feeling is that the tests would then just run longer and run into timeouts. But this seems not the case.

Thesis: when we use the cypress retry mechanism, they will be more stable

It does not solve the problem, but renovate can merge and we can work.

Thesis: the frankenphp container stops responding

(Which leads to the connection refused for the login request errors) -> try enable strace: https://github.com/BacLuc/ecamp3/tree/debug-frankenphp-strace add debug log to caddy: https://github.com/BacLuc/ecamp3/tree/enable-caddy-debug

-> this seems to be confirmed. (See https://github.com/BacLuc/ecamp3/actions/runs/10013887138 in the failing runs in the step added by this PR: https://github.com/ecamp/ecamp3/pull/5558) I have 3 Solutions for this problem, they are listed in these 3 posts:

Vote with a :+1: or :-1:

Improve the single tests

The inspection of the test errors is in this Issue: #5322

Thesis: Not all tests are flaky

manuelmeister commented 4 months ago

Thesis: they are flaky because browsers are flaky => use stable browser version Thesis: they are flaky because Cypress is flaky => use Playwright?

BacLuc commented 4 months ago

Solution 1 for Thesis: "the frankenphp container stops responding"

Revert back to a classic php fpm setup

The thesis is, that the frankenphp is not yet production ready and crashes from time to time. -> Then we should step back to the php fpm setup from before.

BacLuc commented 4 months ago

Solution 2 for Thesis: the frankenphp container stops responding

Use the prod frankenphp api image for the e2e tests

This seems to work (and hopefully so, we want to deploy this in prod). Needs first #5277 (but is prepared)

Seems to work well: https://github.com/BacLuc/ecamp3/actions/runs/10014833334 But is a little complex:

still crashes, but less often: https://github.com/BacLuc/ecamp3/actions/runs/10014833334/job/27685355483

BacLuc commented 4 months ago

Solution 3 for Thesis: the frankenphp container stops responding

Add retries in cypress and restart the api container

This also seems to work. Firefox crashes a lot if you use a global beforeEach. @BacLuc would like to remove firefox from the tests for now if we use this approach.

Seems to not work perfectly, but better as before (https://github.com/bacluc-test-org/ecamp3/actions/runs/9998897898):