linusha
commented
6 months ago Upon further investigation, all of the above information seems to be incorrect. We found out that retrying the layout tests reliably fixes the problem. This is highly curious, as we tried to capture the console output of the failing pipeline runs which threw Error rendering morphs, which is usually unrecoverable and all subsequent tests run in the same lively session.
As of now, we will try to reactivate the caches and we think that the culprit is inside of the text layout tests.
Ever since merging #1481 our CI pipelines failed more or less reliably.
We often had trouble reproducing this locally. Although it was not impossible to reproduce it locally, it did work more often than not and inside of a running lively environment the tests never failed.
We found out that disabling the caching inside of the actions, thus enforcing a clean reinstallation of lively reliably seems to solve the issue. We then tried to move the lively.morphic tests, which are the failing tests, to the start of the pipeline for easier testing. This lead to the tests failing again.
We briefly experimented with setting
headless: 'shell'instead ofheadless: 'new'. However, this would need to be investigated further.Overall the situation is highly unsatisfactory and we have no real indicator what the exact cause for the problems is. For now, we just hope that the problem does not persist, but we should investigate how we can harden the pipeline. Enabling the caching again would also be quite nice as the runtime of the pipelines skyrockets drastically, which is especially annoying for the PR checks. In the worst case we could try to force the installation only for the daily checks and for
lively.morpic.