WARNING Trial Runner checkpointing failed: cannot pickle 'dict_values' object

ray-project / tune-sklearn

A drop-in replacement for Scikit-Learn’s GridSearchCV / RandomizedSearchCV -- but with cutting edge hyperparameter tuning techniques.

https://docs.ray.io/en/master/tune/api_docs/sklearn.html

Apache License 2.0

465 stars 52 forks source link

WARNING Trial Runner checkpointing failed: cannot pickle 'dict_values' object #204

Open einarG-1qbit opened 3 years ago

einarG-1qbit commented 3 years ago

Hi, I'm training a custom sklearn model with tune-sklearn and skopt Bayesian. All works fine, except that in my logs I get the warning: WARNING Trial Runner checkpointing failed: cannot pickle 'dict_values' object No other info is provided, just the warning. How do I fix this issue?

I use: tune-sklearn 0.2.1 ray 1.2.0 sklearn 0.0

I had few session crashes on GCP and I tried to restore the state from ray, however no checkpoints were found. I guess this pertains to the warning above.

Yard1 commented 3 years ago

Does it happen with a different search algo?

richardliaw commented 3 years ago

I think checkpoint/restore may not work directly with tune-sklearn. However, I would try:

tune-sklearn (latest master, so clone off of github) pip install ray ==1.3

einarG-1qbit commented 3 years ago

thanks for the quick reply. I just tried TuneSearchCV without the parameter search_optimization (default random search) And there weren't any warnings.