Hyperopts

[martin1tab]

This PR modifies the following in ramp hyperopt: 1) --n-trials instead of --n-iter 2) Summary.csv will contain one line per fold, not the mean. 3) There is --resume so if the long run crashes, it is possible to ressume. 4) There are now engines of the type "ray_", like ray_hebo. hebo, ax, blend_search.

[codecov[bot]]

Codecov Report

Base: 83.15% // Head: 75.69% // Decreases project coverage by -7.45% :warning:

Coverage data is based on head (c0d2f11) compared to base (4c71c23). Patch coverage: 22.32% of modified lines in pull request are covered.

:exclamation: Current head c0d2f11 differs from pull request most recent head a3e2565. Consider uploading reports for the commit a3e2565 to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #313 +/- ## ========================================== - Coverage 83.15% 75.69% -7.46% ========================================== Files 137 139 +2 Lines 4748 4855 +107 ========================================== - Hits 3948 3675 -273 - Misses 800 1180 +380 ``` | [Impacted Files](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds) | Coverage Δ | | |---|---|---| | [rampwf/hyperopt/cli/hyperopt.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L2NsaS9oeXBlcm9wdC5weQ==) | `0.00% <0.00%> (ø)` | | | [rampwf/tests/kits/titanic/problem.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL3Rlc3RzL2tpdHMvdGl0YW5pYy9wcm9ibGVtLnB5) | `100.00% <ø> (ø)` | | | [rampwf/hyperopt/hyperopt.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L2h5cGVyb3B0LnB5) | `13.03% <8.29%> (-85.62%)` | :arrow_down: | | [rampwf/hyperopt/engines/random\_engine.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L2VuZ2luZXMvcmFuZG9tX2VuZ2luZS5weQ==) | `18.75% <18.75%> (ø)` | | | [rampwf/hyperopt/engines/generic\_engine.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L2VuZ2luZXMvZ2VuZXJpY19lbmdpbmUucHk=) | `42.85% <42.85%> (ø)` | | | [rampwf/utils/cli/show.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL3V0aWxzL2NsaS9zaG93LnB5) | `76.40% <92.00%> (ø)` | | | [rampwf/\_\_init\_\_.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [rampwf/hyperopt/\_\_init\_\_.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [rampwf/hyperopt/engines/\_\_init\_\_.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL2h5cGVyb3B0L2VuZ2luZXMvX19pbml0X18ucHk=) | `100.00% <100.00%> (ø)` | | | [rampwf/score\_types/\_\_init\_\_.py](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds#diff-cmFtcHdmL3Njb3JlX3R5cGVzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | ... and [11 more](https://codecov.io/gh/paris-saclay-cds/ramp-workflow/pull/313/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds) | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=paris-saclay-cds)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[kegl]

It seems windows tests are failing because of the netcdf library, @rth do you have any suggestion what to do?

[rth]

CI fails because there are no binary wheels for Python 3.6 and 3.7 for netcdf4 and it misses the headers to build it from sources (which is particularly difficult on Windows).

What we could do is either,

• pin netcdf4 to an earlier version for now

• or better upgrade CI to test Python 3.8+ (preferable in a separate PR ) which will not have this issue.

  BTW, if possible it might be better to keep ray an optional dependency as itself it has a lot of dependencies (particularly if we don't need it to run submissions on the ramp-board server)

[martin1tab]

@rth @kegl any idea on how to solve this new error ?

import file mismatch: imported module 'rampwf.prediction_types.tests.test_mixed_predictions' has this file attribute: /usr/share/miniconda/envs/testenv/lib/python3.7/site-packages/rampwf/prediction_types/tests/test_mixed_predictions.py which is not the same as the test file we want to collect: /home/runner/work/ramp-workflow/ramp-workflow/rampwf/prediction_types/tests/test_mixed_predictions.py HINT: remove pycache / .pyc files and/or use a unique basename for your test file modules

[kegl]

Do you see the error when you run pytest locally on your machine?

It seems this error was generated when you added the hyperopt test, no? Maybe some globals are clashing with other tests, like PATH?

[kegl]

CI fails because there are no binary wheels for Python 3.6 and 3.7 for netcdf4 and it misses the headers to build it from sources (which is particularly difficult on Windows).

What we could do is either,

* pin netcdf4 to an earlier version for now

* or better upgrade CI to test Python 3.8+ (preferable in a separate PR ) which will not have this issue.

What I propose is to let this error stay for now. @SofianChay will work on the time series tools in the next months, one of the goals is to get rid of netcfd and replace the test with a simpler use case using csvs.

[kegl]

BTW, if possible it might be better to keep ray an optional dependency as itself it has a lot of dependencies (particularly if we don't need it to run submissions on the ramp-board server)

Awesome, that is exactly what I had in mind but don't know how to do it. Yes, ramp-hyperopt is optional, not needed for the server, so we should make ray tune import also optional. @rth Can you do this once @martin1tab fixes the new clash?

[kegl]

@martin1tab We noticed that there were a lot of formatting changes unrelated to the PR (' to ", to mention one; the convention on RAMP is single quote). Pls take these off the PR.