bbc / VideoContext

An experimental HTML5 & WebGL video composition and rendering API.
http://bbc.github.io/VideoContext/
Apache License 2.0
1.32k stars 156 forks source link

Fix Cypress tests #185

Closed richski closed 4 years ago

richski commented 4 years ago

This PR investigated and implements some solutions for the recent failed local and CI testing issues.

Issues

1. All effects regression tests failing locally

Status: Fixed

Issue: Since the Cypress version bump, all the effects tests are failing locally when running yarn cypress.

Solution: Generate and push new snapshots for all the effects at the 1 second mark - https://github.com/bbc/VideoContext/pull/185/commits/7e77845ffbd4ac6ddf69b3641d78d6c9aec73bbc.

2. star wipe test timeout on Travis

Status: Improved

Issue: Despite the original cypress fixes PR #182 passing, the merge into develop still failed. This time with the star wipe test timing out.

Solution: Normally the star wipe passes, but sometimes travis seems to take a bit longer with all the tests. I've increased the Cypress timeout duration, which will hopefully remove or reduce the frequency of this error - https://github.com/bbc/VideoContext/pull/185/commits/f9ae0c4546dce1f92c41eb724e90614d155e0d94.

Click for more info
This issue was first seen on `develop` **Travis log:** https://travis-ci.org/bbc/VideoContext/jobs/604064544 **Cypress log:** https://dashboard.cypress.io/#/projects/mdase9/runs/33/specs It was also seen on this PR before the solution was applied **Cypress log:** https://dashboard.cypress.io/#/projects/mdase9/runs/38/specs **Error output:** ```bash 1) star wipe: CypressError: cy.then() timed out after waiting '10000ms'. ```

3. All tests failed on Travis

Status: Fixed

Issue: All tests failed on one build - https://travis-ci.org/bbc/VideoContext/builds/604052229

Solution: Incorrect package registry was used in the yarn.lock. This has since been corrected on develop -https://github.com/bbc/VideoContext/commit/f9dfd1aec4e0c0eb8b1ecdf6b04e0849617ec962.

4. All regression tests timed out on Travis

Status: Ignore

Issue: Regression tests ran for 50 mins before timing out and reporting an error.

Solution: Looks like Travis was having a moment, so probably safe to ignore.

Click for more info
**Travis output:** ```bash Setting environment variables from .travis.yml $ export CYPRESS_RECORD_KEY=[secure] $ export TEST_SUITE=regression $ nvm install 11.10.1 Downloading and installing node v11.10.1... Downloading https://nodejs.org/dist/v11.10.1/node-v11.10.1-linux-x64.tar.xz... curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly terminated. Binary download from https://nodejs.org/dist/v11.10.1/node-v11.10.1-linux-x64.tar.xz failed, trying source. grep: /home/travis/.nvm/.cache/bin/node-v11.10.1-linux-x64/node-v11.10.1-linux-x64.tar.xz: No such file or directory Provided file to checksum does not exist. Binary download failed, trying source. Detected that you have 2 CPU core(s) Number of CPU core(s) less than or equal to 2, running in single-threaded mode Clang v3.5+ detected! CC or CXX not specified, will use Clang as C/C++ compiler! Downloading https://nodejs.org/dist/v11.10.1/node-v11.10.1.tar.xz... Computing checksum with sha256sum Checksums matched! tar: Ignoring unknown extended header keyword 'SCHILY.dev' tar: Ignoring unknown extended header keyword 'SCHILY.ino' tar: Ignoring unknown extended header keyword 'SCHILY.nlink' ``` **Travis log:** https://travis-ci.org/bbc/VideoContext/jobs/604407165

Notes

~The regression tests are currently failing on Travis due to the cypress.io account hitting its monthly 500 build limit. This should reset tomorrow (November 1st).~