Smoke testing is currently broken on GitHub Actions

[jameshadfield]

Context

We use smoke testing (npm run smoke-test:ci) to test, at a high level, that certain datasets are rendered as expected. For instance, the title of the "/ncov/global" dataset is "Genomic epidemiology of novel coronavirus - Global subsampling" and we smoke-test this by ensuring that the phrase "novel coronavirus" can be found when the page is rendered (as defined here https://github.com/nextstrain/auspice/blob/master/test/smoke-test/urls.txt#L4)

Current Behavior

Running locally passes, as expected:

npm run get-data # get latest datasets
npm run smoke-test:ci # passes

However on GitHub Actions (as defined here) the test fails with error:

FAIL test/smoke-test/urls.test.js (17.482s)
  smoke test rendering of paths in urls.txt
    ✕ it appropriately renders /ncov/global (1151ms)
    ✓ it appropriately renders /zika (7652ms)
    ✓ it appropriately renders /zika?c=region&l=clock&legend=open&m=div&r=region (2715ms)
    ✓ it appropriately renders /page-does-not-exist (1083ms)

  ● smoke test rendering of paths in urls.txt › it appropriately renders /ncov/global

    could not find text "novel coronavirus" on page /ncov/global

      42 |           expect(snapshotText).toEqual(expect.stringContaining(assertion));
      43 |         } catch (_err) {
    > 44 |           throw Error(`could not find text "${assertion}" on page ${testCase.path}`);
         |                 ^
      45 |         }
      46 |       }
      47 |     });

      at Object.it (test/smoke-test/urls.test.js:44:17)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 3 passed, 4 total
Snapshots:   0 total
Time:        17.551s

You can see the log of a failing GitHub Action here.

It's perhaps relevant that when the test fails on a PR, the heroku review app which is spun-up renders the ncov/global dataset as expected, i.e. with "novel coronavirus" in the title! This matches local behavior.

Expected behavior

Smoke tests on GitHub actions behave the same as running locally

Possible solution

The snapshot testing was OS dependent (font rendering was different at the pixel level), so perhaps that's applicable here?

Additional context

The title of the ncov/global dataset hasn't changed for 4 months. The smoke-test failures have been happening consistently on auspice master and various PRs. Thus, it's not a transient failure due to grabbing a ncov dataset with an unexpected title!

[eharkins]

@jameshadfield should we close this? Seems like they are working again: https://github.com/nextstrain/auspice/runs/1554302456?check_suite_focus=true

[jameshadfield]

This is still a bug -- running them on GitHub actions seem to result in stochastic pass/fail results -- for instance https://github.com/nextstrain/auspice/pull/1246/checks failed ~2 days ago.