axelboc
commented
3 weeks ago Puppeteer should help simplify Cypress' internals and reduce incompatibilities, but it is not a testing framework. The only other well-established E2E testing framework I know is Microsoft's Playwright but I don't know that it would make any difference for us in practice. IMO, it's just that end-to-end tests are challenging by nature and we lack the experience to set them up properly.
There are currently two main issues:
- I'm struggling to find a way to make tests independent from one another - i.e. to reset the server's state before each test. We currently don't have a solution for this, which means that each test has to account for the possibility that the UI may be in different states when the test starts (e.g. a sample is already mounted; another user is already in control; etc.) The heavily stateful nature of MXCuBE-Web makes this particularly difficult to deal with.
- WebSockets don't seem to work on the CI, which is the main reason why some of the tests are slow and flaky -- this is highly problematic and definitely needs to be investigated further. Perhaps this would be solved by switching to Playwright, but who knows... and who knows which other issues we would face...
E2E tests are a must-have, but since they are slow and challenging, they are usually few and tend to complement a more thorough suite of feature tests.
Feature tests are tests that run in a mock browser environment like JSDOM and where the server is entirely mocked. They are much faster since they don't run in a real browser and API calls are instantaneous, and it's easier to make tests independent from one another since it's a matter of resetting the API mocks between each test. However, running in a mock browser environment brings its own set of problems...
Examples of feature (and unit) testing frameworks include Jest and Vitest. The latter integrates very well with Vite, obviously, which is super convenient and avoids a lot of setup issues.
Feature testing MXCuBE would be particularly challenging because of how many API calls and WebSocket messages need to be mocked. The chance of writing tests that would mock things properly and in the right order would be very slim. We would likely encounter a lot of false positive and false negative tests (i.e. tests that fail or pass when they shouldn't).
I still think we need feature tests (in addition to E2E test) -- I'm just trying to convey that front-end testing is particularly hard, no matter the framework and the kind of tests.
fabcor-maxiv
marcus-oscarsson
I noticed that we seem to have issues with Cypress (flaky behaviour, performance issues, and so on), but I am not familiar with the topic at all so I do not know if we are encountering a normal amount of issues or if it is worth opening a discussion and reconsider how we handle end-to-end tests.
@axelboc @walesch-yan @marcus-oscarsson I have of course seen that you guys have done some good work on this lately, so really I don't know, I can not judge the situation, if it is just a short bad stroke of luck. : )
I randomly saw this announcement the other day: https://hacks.mozilla.org/2024/08/puppeteer-support-for-firefox/ which also pushed me a bit to open this ticket. : )