search

jrbourbeau / dask-optuna

Scale Optuna with Dask
https://jrbourbeau.github.io/dask-optuna/
MIT License
35 stars 6 forks source link

AttributeError: 'NoneType' object has no attribute 'optuna_read_trials_from_remote_storage' #20

Closed pseudotensor closed 3 years ago

pseudotensor commented 3 years ago 

2020-11-19 11:17:59,349 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |   File "h2oaicore/models.py", line 1502, in h2oaicore.models.MainModel.hyper_fit
2020-11-19 11:17:59,349 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |   File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/optuna/study.py", line 108, in trials
2020-11-19 11:17:59,349 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |     return self.get_trials()
2020-11-19 11:17:59,349 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |   File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/optuna/study.py", line 145, in get_trials
2020-11-19 11:17:59,350 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |     self._storage.read_trials_from_remote_storage(self._study_id)
2020-11-19 11:17:59,350 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |   File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/dask_optuna/storage.py", line 597, in read_trials_from_remote
2020-11-19 11:17:59,350 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   |     self.client.scheduler.optuna_read_trials_from_remote_storage,
2020-11-19 11:17:59,350 C: NA  D:  NA    M:  NA    NODE:SERVER      10539  DATA   | AttributeError: 'NoneType' object has no attribute 'optuna_read_trials_from_remote_storage'

Running like:

                with client_manager as client:
                    storage = dask_optuna.DaskStorage()
                    study = optuna.create_study(sampler=sampler,
                                                pruner=pruner,
                                                direction=direction,
                                                storage=storage)
                    # Optimize in parallel on dask cluster
                    with joblib.parallel_backend("dask"):
                        study.optimize(objective, n_trials=n_trials)  # timeout can be specified

And once done with all (here 100) trials, I get this error. But I don't always get. So I don't have an MRE.

Any ideas?

pseudotensor commented 3 years ago

Nevermind, I realized the error fell outside of the client context, not in it.