Making Nevergrad's CI parallel

facebookresearch / nevergrad

A Python toolbox for performing gradient-free optimization

https://facebookresearch.github.io/nevergrad/

MIT License

3.96k stars 354 forks source link

Making Nevergrad's CI parallel #1385

Open teytaud opened 2 years ago

teytaud commented 2 years ago

Types of changes

[ ] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

How Has This Been Tested (if it applies)

Checklist

[x] The documentation is up-to-date with the changes I made.
[x] I have read the CONTRIBUTING document and completed the CLA (see CLA).
[x] All tests passed, and additional code has been covered with new tests.

teytaud commented 2 years ago

The idea here is that it is both faster and more economical / ecological, because bugs are found after less tests when the order is not sequential. This is because most errors typically breaks all tests in one of the block of tests, so a randomized (or parallel) order will be more likely to find early. It would be great to have a clever order though (I'm trying to do that, ideally I'd like to do one test per test file first).

teytaud commented 2 years ago

Hum I see that we can go up to 300. Should we use 10 ?

jrapin commented 2 years ago

does it actually go faster? 22min is the standard time I think, so this does not change anything right now

teytaud commented 2 years ago

I guess it's much more complicated than that... we need to split, otherwise CircleCI just runs all tests on all machines, which is not what we want to do :-)

teytaud commented 2 years ago

(it's cool for testing flakiness though...) i'll see what I can do for this.

jrapin commented 2 years ago

(it's cool for testing flakiness though...) i'll see what I can do for this.

In term of environment it's a disaster though, i'd rather avoid it

jrapin commented 2 years ago

the best option for shortening the CI is to shorten (not remove) the slowest tests. I'm pretty sure we can go below 10min this way in a day or so