ray-project / tune-sklearn

A drop-in replacement for Scikit-Learn’s GridSearchCV / RandomizedSearchCV -- but with cutting edge hyperparameter tuning techniques.
https://docs.ray.io/en/master/tune/api_docs/sklearn.html
Apache License 2.0
465 stars 52 forks source link

gpu usage #228

Open Eeonum opened 2 years ago

Eeonum commented 2 years ago

Hi I am getting this error when I set use_gpu=True.

(scheduler +6s) Tip: use ray status to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0. (scheduler +6s) Error: No available node types can fulfill resource request {'CPU': 1.0, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue. (scheduler +41s) Error: No available node types can fulfill resource request {'GPU': 1.0, 'CPU': 1.0}. Add suitable node types to this cluster to resolve this issue. Ignore this message if the cluster is autoscaling. You asked for 1.0 cpu and 1.0 gpu per trial, but the cluster only has 10.0 cpu and 0 gpu. Stop the tuning job and adjust the resources requested per trial (possibly via resources_per_trial or via num_workers for rllib) and/or add more resources to your Ray runtime. (scheduler +1m16s) Error: No available node types can fulfill resource request {'GPU': 1.0, 'CPU': 1.0}. Add suitable node types to this cluster to resolve this issue. (scheduler +1m51s) Error: No available node types can fulfill resource request {'CPU': 1.0, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue. Ignore this message if the cluster is autoscaling. You asked for 1.0 cpu and 1.0 gpu per trial, but the cluster only has 10.0 cpu and 0 gpu. Stop the tuning job and adjust the resources requested per trial (possibly via resources_per_trial or via num_workers for rllib) and/or add more resources to your Ray runtime. (scheduler +2m26s) Error: No available node types can fulfill resource request {'CPU': 1.0, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue. (scheduler +3m1s) Error: No available node types can fulfill resource request {'CPU': 1.0, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue. Ignore this message if the cluster is autoscaling. You asked for 1.0 cpu and 1.0 gpu per trial, but the cluster only has 10.0 cpu and 0 gpu. Stop the tuning job and adjust the resources requested per trial (possibly via resources_per_trial or via num_workers for rllib) and/or add more resources to your Ray runtime. (scheduler +3m36s) Error: No available node types can fulfill resource request {'GPU': 1.0, 'CPU': 1.0}. Add suitable node types to this cluster to resolve this issue.

Eeonum commented 2 years ago

was trying to run the following script:

`""" An example training a CatBoostClassifier, performing randomized search using TuneSearchCV. """

from tune_sklearn import TuneSearchCV from sklearn import datasets from sklearn.model_selection import train_test_split from catboost import CatBoostClassifier

digits = datasets.load_digits() x = digits.data y = digits.target x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=.2)

params = { "learning_rate": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], "depth": [3, 4, 5], }

catboostclf = CatBoostClassifier(n_estimators=50, )

digit_search = TuneSearchCV( catboostclf, param_distributions=params, n_trials=3, early_stopping=True, use_gpu=True # Commented out for testing on github actions,

but this is how you would use gpu

)

digit_search.fit(x_train, y_train) print(digit_search.bestparams) print(digit_search.cvresults)`

papelypluma commented 1 year ago

hi @Eeonum by any chance I'd like to ask whether you were able to address this issue on your end. I'm trying to run a similar one, but I keep getting the same error. Thanks!

HoBeedzc commented 1 year ago

hi, @Eeonum by any chance I'd like to ask whether you could address this issue on your end. I'm trying to run a similar one but keep getting the same error. Thanks!