search

hyperopt / hyperopt-sklearn

Hyper-parameter optimization for sklearn
hyperopt.github.io/hyperopt-sklearn
Other
1.57k stars 270 forks source link

Any Advice on Avoiding 'NaN' errors #189

Open dfossl opened 2 years ago

dfossl commented 2 years ago

Sometimes when I run this:

test_model = HyperoptEstimator(regressor=gradient_boosting_regressor('test_regres'),
                            preprocessing=[],
                            algo=tpe.suggest,
                            max_evals=50,
                            trial_timeout=30)
test_model.fit(X_train.to_numpy(), y_train.to_numpy().ravel())

I get an error ValueError: Input contains NaN. during training. It doesn't happen every time and I know that the data has no nan's, infinites, or duplicates. This leads me to believe one of the operations is creating a NaN. Is there anyway to skip these operations or deduce what operation is causing this?

mandjevant commented 2 years ago

Hey there, could you provide me with a sample of your training data? I am unable to reproduce this issue.

RaistlinTAO commented 1 year ago

I just got same error during HyperoptEstimator search, hope the following log helps

2|hpsklearnBlack  | job exception: Input contains NaN.
 98%|█████████▊| 48/49 [00:02<?, ?trial/s, best loss=?]
2|hpsklearnBlack  | Traceback (most recent call last):
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 48, in <module>
2|hpsklearnBlack  |     find_best_model(X_train, y_train, X_test, y_test)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 29, in find_best_model
2|hpsklearnBlack  |     estimator.fit(x, y)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 464, in fit
2|hpsklearnBlack  |     fit_iter.send(increment)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 339, in fit_iter
2|hpsklearnBlack  |     hyperopt.fmin(_fn_with_timeout,
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 540, in fmin
2|hpsklearnBlack  |     return trials.fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 671, in fmin
2|hpsklearnBlack  |     return fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 586, in fmin
2|hpsklearnBlack  |     rval.exhaust()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 364, in exhaust
2|hpsklearnBlack  |     self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 300, in run
2|hpsklearnBlack  |     self.serial_evaluate()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 178, in serial_evaluate
2|hpsklearnBlack  |     result = self.domain.evaluate(spec, ctrl)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 892, in evaluate
2|hpsklearnBlack  |     rval = self.fn(pyll_rval)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 311, in _fn_with_timeout
2|hpsklearnBlack  |     raise fn_rval[1]
2|hpsklearnBlack  | ValueError: Input contains NaN.

According the result from df.isnull().sum() there is no NaN in data, hence, the NaH error occurs during parameter injection.

mandjevant commented 1 year ago

@RaistlinTAO Could I get a snippet of your code please? I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

RaistlinTAO commented 1 year ago

@RaistlinTAO Could I get a snippet of your code please? I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

Of course, happy to help

def find_best_model(x, y, test_x, test_y):
    estimator = HyperoptEstimator(
        regressor=gradient_boosting_regressor("T"),
        algo=tpe.suggest,
        max_evals=800,
        trial_timeout=300)
    estimator.fit(x, y)
    print('HyperoptEstimator Score: ')
    print(estimator.score(test_x, test_y))
    print('Best Model: ')
    print(estimator.best_model())

find_best_model(X_train, y_train, X_test, y_test)
RaistlinTAO commented 1 year ago

Update:

With same code and same settings, it will bypass the error after give it another 3-5 tries.

I think it related to hyperparameter combination. sometimes it just skip the wrong combination or preprocessing

mandjevant commented 1 year ago

Thanks a lot @RaistlinTAO that is what I was thinking as well. I've noticed this behaviour before. Unfortunately, I cannot reproduce the error in our testing environment so it is a difficult issue to fix. Your code snippet will help me to point down the problem and focus my testing.

I will work on a fix for this. But for the time being, retrying a few times should bypass the error.

RaistlinTAO commented 1 year ago

After several attempts on debug in local env I have located the root cause.

In sklearn\ensemble_gb_losses.py update_terminal_regions() statement:

    def _update_terminal_region(...):
        ...
        diff_minus_median = diff - median
        ...

# and 

def update_terminal_regions(...):
        raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()

SAMPLE OUTPUT

E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: overflow encountered in multiply
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: invalid value encountered in add
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
 97%|█████████▋| 32/33 [00:06<?, ?trial/s, best loss=?]
job exception: Input contains NaN.

There are two different outputs indicate that diff/median/learning_rate is NaN under certain circumstances. I ll just leave this here since my hands are tied atm.

crieux commented 1 year ago

Hello, I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
RaistlinTAO commented 1 year ago

Hello, I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Well my suggestions are:

  1. Use Optuna (https://github.com/optuna/optuna), however they have different issues, but it's rare.
  2. Use GridSearchCV, but considering the time/resource consumption.

Pick any solution that fit your needs and have a good one

zhukuixi commented 1 month ago

I think this have sth to do with the gradient. Try to use gradient clip to avoid this problem when it comes to some extreme hyperparameter combination.

Apply gradient clipping to prevent exploding gradients, which can cause NaN values.