Eric-Arellano
commented
1 year ago Also no idea why the CI snapshots keep failing when the "actual" images look exactly the same as the originals 😅
It looks like for some reason the new snapshots were 1 pixel longer in https://github.com/Qiskit/qiskit_sphinx_theme/pull/530. I don't know why that changed, but I think it explains this.
In terms of why things kept failing, it's because some tests have multiple screenshots in the same test. When a screenshot fails, then the test early exits. So it won't try out the snapshots for all subsequent ones. So, once you fix the first failure, only then do you see the second failure, and so on.
It's common to have to update snapshot images that fail in CI, even when they've already been run in docker and updated locally. In addition I've found that CI will fail repeatedly even after I update the snapshots based on the results from the previous failure, with different images failing each time. I'm not sure if this is the intended behaviour or a bug in our docker setup, but this makes it cumbersome to do development work and would be good to streamline this process if posible