search

chanzuckerberg / cellxgene

An interactive explorer for single-cell transcriptomics data
https://chanzuckerberg.github.io/cellxgene/
MIT License
626 stars 116 forks source link

Fix cellxgene desktop GHA intermittent failures #2433

Open atolopko-czi opened 3 years ago

atolopko-czi commented 3 years ago

The cellxgene desktop GHA compatibility matrix tests have intermittently failing jobs, all failing at the smoke-test-annotations step. E.g. see https://github.com/chanzuckerberg/cellxgene/actions/runs/1243692946. May be timing-related.

atolopko-czi commented 3 years ago

As of PR #2431.

seve commented 3 years ago

Looks like the objective of the test, properly truncated values, is working properly, where it's going wrong is that the number of cells attached to the truncated value is varying from the expected screenshot. It might just be a matter of updating the snapshots.

atolopko-czi commented 3 years ago

Since smoke-test-annotations appears to pass reliably for the push-test workflow, I tried running just the smoke-test-annotations from within the compatibility-test workflow. But alas it still fails for some steps. See https://github.com/chanzuckerberg/cellxgene/runs/3655021242?check_suite_focus=true.

atolopko-czi commented 3 years ago

I agree with Seve's assessment on this:

We might want to invest a good amount of time into this. The good thing is we realized this isn't necessarily a compatibility test issue, but an at large smoke test issue.
I don't think this is blocking 0.19 since this isn't an underlying issue of the FE, but of the tests themselves which aren't attached to a release version.

atolopko-czi commented 3 years ago

Recording Seve's input that the failing tests appear to be related lasso'ing, and the resultant selection of cells.

atolopko-czi commented 3 years ago

Retries reduce the number of failures. Adding retries allowed some smoke-test-annotations steps to pass, after initial failure. Not a proper solution, but just confirming that the failures are in fact intermittent and non-deterministic.

atolopko-czi commented 3 years ago

Also fails on push-tests.