Run tests in a parallel matrix

bonjourmauko commented 3 years ago

Hello!

(Working) prototype for solving #1025 and #1035, poke #1027, please chime in! 😄

⚠ This is not a proposal but a demonstration of the different tools, so we can discuss and choose what make more sense in terms both of performance and readability to the community.

Trying it out

$ make test.matrix
...

real    2m13.152s
user    10m12.086s
sys 1m48.311s

New features

Run openfisca test tests in parallel with make, nox, and invoke
Run tests against Python 3.7+
Run tests against NumPy 1.18+

Notes

We don't have control over the way subprocesses are launched and run, so reporting could be a bit confusing
Performance gains are relative to the number of cores in your machine
OpenFisca is not threadsafe, so each subprocess will run a single thread
If one day we solve concurrency problems, ... that could go faster.

TODO

[x] Run each test suite with invoke
[x] Run the matrix of tests with nox
[x] Parallelize with make
[x] Parallelize with pytest-parallel
[ ] Parallelize with pytest-split
[x] Run .py tests in parallel
[ ] Run .yaml tests in parallel
[ ] Provide decent reporting
[ ] What to do in CI #1030
[x] Benchmark

MattiSG commented 3 years ago

Thanks @maukoquiroga! The promise of such a speed improvement is exciting, and I would be very appreciative of a refactor / overhaul of the current test runner 😃

However, I must say that the changeset is pretty daunting, and seems to have impacts way beyond the test runner, with some core functions having their visibility changed… Could we maybe convene of a video discussion with other interested @openfisca/international-contrib contributors, in order to understand the goals, side effects and side benefits? 🙂

In terms of pure performance and parallelisation, it might also be that #1030 allows us to use https://github.com/nektos/act as suggested by @benoit-cty, and we could have local parallelisation with the exact same setup as CI, which I would be very excited about 😃

bonjourmauko commented 3 years ago

However, I must say that the changeset is pretty daunting, and seems to have impacts way beyond the test runner, with some core functions having their visibility changed…

Some of them were absolutely required, in which case I've tried to avoid a breaking change by exposing changes through __init__.py files, some maybe just because I just was rushing to make it work (hence the draft status).

Indeed a partial refactoring of the test runner was imposed in order to make this work, which could definitely go in a PR on its own, with proper deprecations, and agreed upon breaking changes.

In terms of pure performance and parallelisation, it might also be that #1030 allows us to use https://github.com/nektos/act as suggested by @benoit-cty, and we could have local parallelisation with the exact same setup as CI, which I would be very excited about 😃

❤️

Could we maybe convene of a video discussion with other interested @openfisca/international-contrib contributors, in order to understand the goals, side effects and side benefits? 🙂

Definitely.

bonjourmauko commented 2 years ago

Closing for now.

openfisca / openfisca-core