search

mindsdb / mindsdb_native

Machine Learning in one line of code
http://mindsdb.com
GNU General Public License v3.0
37 stars 28 forks source link

Error when querying time series model #407

Closed paxcema closed 3 years ago

paxcema commented 3 years ago

Your Environment

Please describe your issue and how we can replicate it Predictor successfully trains on a clickhouse datasource with metro_traffic_ts dataset and standard time series options (order by date, window = 5, use_previous_target = True).

However, when trying a single query from the GUI (based on first row of testing data split), prediction fails with the following stack trace:

ERROR:mindsdb-logger-core-logger---:/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/transaction.py:103 - CUDA error: no kernel image is available for execution on the device

ERROR:mindsdb-logger-core-logger---:/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/transaction.py:104 - Could not load mindsdb conformal predictor in the file: /home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb/var/predictors/tstest/icp.pickle

[2021-01-26 14:01:38,257] ERROR in app: Exception on /api/predictors/tstest/predict [POST]
Traceback (most recent call last):
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask_restx/api.py", line 375, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask_restx/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb/api/http/namespaces/predictor.py", line 229, in post
    results = ca.mindsdb_native.predict(name, when_data=when, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb/interfaces/native/native.py", line 71, in predict
    predictions = self.predictor_cache[name]['predictor'].predict(
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/predictor.py", line 424, in predict
    self.transaction.run()
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/transaction.py", line 396, in run
    self.lmd['all_conformal_ranges'][predicted_col] = self.hmd['icp'][predicted_col].predict(X.values)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/nonconformist/icp.py", line 387, in predict
    p = self.nc_function.predict(x[idx, :],
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/nonconformist/nc.py", line 495, in predict
    prediction = self.model.predict(x)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/helpers/conformal_helpers.py", line 75, in predict
    predictions = self.model.predict(when_data=x)
AttributeError: 'NoneType' object has no attribute 'predict'
ERROR:mindsdb.api.http.initialize:Exception on /api/predictors/tstest/predict [POST]
Traceback (most recent call last):
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask_restx/api.py", line 375, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/flask_restx/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb/api/http/namespaces/predictor.py", line 229, in post
    results = ca.mindsdb_native.predict(name, when_data=when, **kwargs)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb/interfaces/native/native.py", line 71, in predict
    predictions = self.predictor_cache[name]['predictor'].predict(
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/predictor.py", line 424, in predict
    self.transaction.run()
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/controllers/transaction.py", line 396, in run
    self.lmd['all_conformal_ranges'][predicted_col] = self.hmd['icp'][predicted_col].predict(X.values)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/nonconformist/icp.py", line 387, in predict
    p = self.nc_function.predict(x[idx, :],
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/nonconformist/nc.py", line 495, in predict
    prediction = self.model.predict(x)
  File "/home/mdbtester/Downloads/mindsdb_env/lib/python3.8/site-packages/mindsdb_native/libs/helpers/conformal_helpers.py", line 75, in predict
    predictions = self.model.predict(when_data=x)
AttributeError: 'NoneType' object has no attribute 'predict'

Somehow, the model attribute inside the conformal predictor object stayed as None. The ICP .pickle file exists, so I suspect it's related to the CUDA error warning.

paxcema commented 3 years ago

Seems my intuition is right. Installing torch==1.7.1+cu110 fixes the issue and I can predict normally.

paxcema commented 3 years ago

Okay, weirdly enough the error arises only the first time one queries the model. Afterwards, predictions work just fine.

The issue has to do with the way Lightwood detects whether there is a usable GPU. Somehow, doing torch.ones(1).cuda() is not always enough to trigger a runtime error, like in this case. Doing torch.ones(1).cuda().__repr__() works, triggering the RuntimeError that signals Lightwood there's no GPU, and solves the issue.

paxcema commented 3 years ago

Closed by L#362.