speed up CD?

[jstet]

CD currently takes ~5min plus vercel build ~1min.

I feel like thats to slow for if we want to change small things in short time spans.

Possible solutions:

• https://playwright.dev/docs/ci#on-pushpull_request-sharded
• https://blog.karmacomputing.co.uk/make-playwright-faster-with-containers-and-build-caching-github-actions/

[KonradUdoHannes]

We can try a few things to improve runtime because that is always nice. However end2end testing is nothing that is designed to fast in general and any progress might deteriorate quickly as we add more tests. The way testing often works and the way I like to think about it is as follows.

• unit tests fast and can be run often as part of quick iterations
• end2end tests are potentially slow and should not/can not be part of a quick iteration process but rather are a final check after a few iterations (that passed unit tests)

With this in mind I wonder how much effort we should spend on this. I think we already have a setup where quick iterations run unit tests in CI an make a cloudflare deployment to observe. So ideally we are not often in a situation where we feel like we have to wait for the static build to deploy. If there are often reasons to wait for the build and that disrupts the workflow we can maybe also look at those and see whether we find other solutions for them.

[KonradUdoHannes]

Independently of my above comment there are two other things that we can potentially do to improve CD runtime, but both possibly a bit more complex.

• currently we make the static build twice. Once in CD for testing against in and once on vercel. We could export the build as an artifact and deliver the finished build to vercel for deployment. This should save some time, and ensure that the deployed build is exactly the one that was tested against.
• Some end2end tests are often "flaky" this means they do not always pass on the first try and are automatically rerun a few times. If its possible to reduce flakyness of the tests, it should also reduce overall runtime (in addition to increasing test reliability).

[KonradUdoHannes]

Another point to consider is that we are currently end2end testing against 3 browsers.

• chromium
• firefox
• webkit

Which means that every test runs three times. This is of course quite a factor for runtime. If we are more concerned with general issues rather than browser specific issues we could also reduce the number of browser to 1 or 2.

[jstet]

I agree and feel like this doesnt have a high priority. However "sharding" the tests doesnt seem like too much work. If you want, i can look into this.

[jstet]

Independently of my above comment there are two other things that we can potentially do to improve CD runtime, but both possibly a bit more complex.

* currently we make the static build twice. Once in CD for testing against in and once on vercel. We could export the build as an artifact and deliver the finished build to vercel for deployment. This should save some time, and ensure that the deployed build is exactly the one that was tested against.

* Some end2end tests are often "flaky" this means they do not always pass on the first try and are automatically rerun a few times. If its possible to reduce flakyness of the tests, it should also reduce overall runtime (in addition to increasing test reliability).

I also thought about this, but did not find a good solution after researching for a bit.

[jstet]

Its fine as it is atm