Sometimes I get the following errors, I guess it may happen when Celery receives both configuration_recommendation task and periodic tasks in the same time.
[30/May/2020 07:51:18] ERROR [celery.worker.job:282] Task configuration_recommendation[configuration_recommendation-edce126a-41db-4372-8dc8-68c4ceb51750] raised unexpected: ValueError("Input contains NaN, infinity or a value too large for dtype('float64').",)
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(args, kwargs)
File "/app/website/website/tasks/async_tasks.py", line 933, in configuration_recommendation
res = tf_optimize(m.model, X_samples, opt_kwargs)
File "/app/analysis/gpr/optimize.py", line 74, in tf_optimize
assert_all_finite(Xnew_value)
File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 54, in assert_all_finite
_assert_all_finite(X.data if sp.issparse(X) else X)
File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 44, in _assert_all_finite
" or a value too large for %r." % X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Sometimes I get the following errors, I guess it may happen when Celery receives both configuration_recommendation task and periodic tasks in the same time.
[30/May/2020 07:51:18] ERROR [celery.worker.job:282] Task configuration_recommendation[configuration_recommendation-edce126a-41db-4372-8dc8-68c4ceb51750] raised unexpected: ValueError("Input contains NaN, infinity or a value too large for dtype('float64').",) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 438, in __protected_call__ return self.run(args, kwargs) File "/app/website/website/tasks/async_tasks.py", line 933, in configuration_recommendation res = tf_optimize(m.model, X_samples, opt_kwargs) File "/app/analysis/gpr/optimize.py", line 74, in tf_optimize assert_all_finite(Xnew_value) File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 54, in assert_all_finite _assert_all_finite(X.data if sp.issparse(X) else X) File "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py", line 44, in _assert_all_finite " or a value too large for %r." % X.dtype) ValueError: Input contains NaN, infinity or a value too large for dtype('float64').