search

source-academy / frontend

Frontend of Source Academy, an online experiential environment for computational thinking (React, Redux, Saga, Blueprint)
https://sourceacademy.org
Apache License 2.0
101 stars 164 forks source link

Inconsistent snapshot test between different test runs #2975

Closed RichDom2185 closed 1 month ago

RichDom2185 commented 2 months ago

Following changes to test files in #2952 and #2961, I've been finding myself having to re-trigger workflows often because some tests would just randomly fail.

When they do fail, it's always due to a snapshot mismatch, specifically regarding the icon component SVGs not being populated.

Restarting the workflow solves it.

This affects local test environments as well the CI environment. Sometimes the PR passes CI and gets merged only for it to fail the master branch CI later (until the workflow is reran), which is annoying.

Reference logs:

https://github.com/source-academy/frontend/actions/runs/8912047561/attempts/1 https://github.com/source-academy/frontend/actions/runs/8912188643/attempts/1

RichDom2185 commented 2 months ago

To test locally, simply run yarn test interactively and press a to run all tests repeatedly. Sometimes at least one snapshot will fail.

RichDom2185 commented 2 months ago

Another interesting one:

2 different run attempts of the same commit, different snapshots failing:

https://github.com/source-academy/frontend/actions/runs/8922258620/attempts/1?pr=2968 https://github.com/source-academy/frontend/actions/runs/8922258620/attempts/2?pr=2968

chownces commented 2 months ago

I think this has got to do with the async nature of the rendering... Let's KIV this as discussed offline, since we have to migrate the tests over when moving to Vite (hopefully soon...)