Use Docker to ensure local tests and CI pipeline tests agree?

[papadeltasierra]

Refs:

• https://github.com/papadeltasierra/typeshed/actions/runs/10358301701
• https://github.com/papadeltasierra/typeshed/actions/runs/10358301703

Both these CI pipelines show errors despite me running tests/runtests.py locally beforehand and seeing no errors.

I assume this is the result of some sort of versioning issue between the testing tools being used. Would it be possible to provide a Docker container that has the tooling matching the CI pipeline so that running tests locally matches test results in the CI pipeline?

[AlexWaygood]

runtests.py is meant to be a convenience tool for contributors, but isn't meant to exhaustively run every check in our CI pipeline. While most of our tests can be run locally, it wouldn't really be practical to run all of them locally for every PR; we have an awful lot of CI here. In general, we don't expect PRs to necessarily pass every check immediately after you open them, and we don't look down on PRs that have checks failing at first :-)

[papadeltasierra]

Fair enough but one issue I hit was that running the tests locally flagged an error "in reverse" to the CI i.e. I had to ADD a type hint to pass locally, then REMOVE it to pass the CI tests. Surely the test I'm running locally should match one of the tests run in CI and I should not be seeing this? I do appreciate that "CI might spot something that local tests missed" but this was not one of those ;-).

[AlexWaygood]

Hmm, that sounds strange. Can you be more specific? Which tests were complaining, and which type hint was it?

[papadeltasierra]

I tried to reproduce this this evening but I'm guessing there has been an update to one of the checkers because my code, which was "clean" locally yesterday, now has 9000 errors! Hopefully they are all the result of one simple required change and I will be able to get back to you later.