search

mljar / mljar-supervised

Python package for AutoML on Tabular Data with Feature Engineering, Hyper-Parameters Tuning, Explanations and Automatic Documentation
https://mljar.com
MIT License
3k stars 401 forks source link

AutoMLException: No models produced. #575

Closed Karlheinzniebuhr closed 1 year ago

Karlheinzniebuhr commented 1 year ago

I'm getting this exception since today, with a codebase I've used for months without any issues before. Same dataset and settings work without problem in Explain mode

from supervised.automl import AutoML
mode = 'Perform'
name = 'USDJPY'
model_path_1 = f'{mode}_%_change_high_{name}'
model_path_2 = f'{mode}_%_change_low_{name}'
model_path_3 = f'{mode}_%_change_close_{name}'

automl1 = AutoML(mode=mode, results_path=model_path_1)
automl1.fit(X_train, y_train['label_high'])
automl2 = AutoML(mode=mode, results_path=model_path_2)
automl2.fit(X_train, y_train['label_low'])
automl3 = AutoML(mode=mode, results_path=model_path_3)
automl3.fit(X_train, y_train['label_close'])

# create the markdown report
automl1.report()
automl2.report()
automl3.report()
Linear algorithm was disabled.
AutoML directory: Perform_%_change_high_USDJPY
The task is regression with evaluation metric rmse
AutoML will use algorithms: ['Random Forest', 'LightGBM', 'Xgboost', 'CatBoost', 'Neural Network']
AutoML will ensemble available models
AutoML steps: ['simple_algorithms', 'default_algorithms', 'not_so_random', 'golden_features', 'insert_random_feature', 'features_selection', 'hill_climbing_1', 'hill_climbing_2', 'ensemble']
Skip simple_algorithms because no parameters were generated.
* Step default_algorithms will try to check up to 5 models
1_Default_LightGBM not trained. Force to stop the training. Total time for AutoML training already exceeded.
There was an error during 2_Default_Xgboost training.
Please check Perform_%_change_high_USDJPY\errors.md for details.
There was an error during 3_Default_CatBoost training.
Please check Perform_%_change_high_USDJPY\errors.md for details.
There was an error during 4_Default_NeuralNetwork training.
Please check Perform_%_change_high_USDJPY\errors.md for details.
There was an error during 5_Default_RandomForest training.
Please check Perform_%_change_high_USDJPY\errors.md for details.
Skip not_so_random because of the time limit.
Skip golden_features because no parameters were generated.
Skip insert_random_feature because no parameters were generated.
Skip features_selection because no parameters were generated.
Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?307fe80e-f0ef-46df-9149-958e7718dd3f)
---------------------------------------------------------------------------
AutoMLException                           Traceback (most recent call last)
c:\dev\Python\forex\LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb Cell 15 in <cell line: 8>()
      [5](vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=4) model_path_3 = f'{mode}_%_change_close_{name}'
      [7](vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=6) automl1 = AutoML(mode=mode, results_path=model_path_1)
----> [8](vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=7) automl1.fit(X_train, y_train['label_high'])
      [9](vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=8) automl2 = AutoML(mode=mode, results_path=model_path_2)
     [10](vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=9) automl2.fit(X_train, y_train['label_low'])

File c:\ProgramData\Anaconda3\lib\site-packages\supervised\automl.py:368, in AutoML.fit(self, X, y, sample_weight, cv)
    346 def fit(
    347     self,
    348     X: Union[numpy.ndarray, pandas.DataFrame],
   (...)
    351     cv: Optional[Union[Iterable, List]] = None,
    352 ):
    353     """Fit the AutoML model.
    354 
    355     Arguments:
   (...)
    366         AutoML object: Returns `self`
    367     """
--> 368     return self._fit(X, y, sample_weight, cv)

File c:\ProgramData\Anaconda3\lib\site-packages\supervised\base_automl.py:1126, in BaseAutoML._fit(self, X, y, sample_weight, cv)
...
   1048 generated_params = []
   1049 if step in self._all_params:

AutoMLException: No models produced. 
Please check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.

FULL exception text

{
    "name": "AutoMLException",
    "message": "No models produced. \nPlease check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.",
    "stack": "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mAutoMLException\u001b[0m                           Traceback (most recent call last)\n\u001b[1;32mc:\\dev\\Python\\forex\\LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb Cell 15\u001b[0m in \u001b[0;36m<cell line: 8>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m model_path_3 \u001b[39m=\u001b[39m \u001b[39mf\u001b[39m\u001b[39m'\u001b[39m\u001b[39m{\u001b[39;00mmode\u001b[39m}\u001b[39;00m\u001b[39m_%_change_close_\u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m'\u001b[39m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m automl1 \u001b[39m=\u001b[39m AutoML(mode\u001b[39m=\u001b[39mmode, results_path\u001b[39m=\u001b[39mmodel_path_1)\n\u001b[1;32m----> <a href='vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m automl1\u001b[39m.\u001b[39;49mfit(X_train, y_train[\u001b[39m'\u001b[39;49m\u001b[39mlabel_high\u001b[39;49m\u001b[39m'\u001b[39;49m])\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=8'>9</a>\u001b[0m automl2 \u001b[39m=\u001b[39m AutoML(mode\u001b[39m=\u001b[39mmode, results_path\u001b[39m=\u001b[39mmodel_path_2)\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/dev/Python/forex/LAB_MLJAR_multi_target_regression_plus_n_5m.ipynb#X20sZmlsZQ%3D%3D?line=9'>10</a>\u001b[0m automl2\u001b[39m.\u001b[39mfit(X_train, y_train[\u001b[39m'\u001b[39m\u001b[39mlabel_low\u001b[39m\u001b[39m'\u001b[39m])\n\nFile \u001b[1;32mc:\\ProgramData\\Anaconda3\\lib\\site-packages\\supervised\\automl.py:368\u001b[0m, in \u001b[0;36mAutoML.fit\u001b[1;34m(self, X, y, sample_weight, cv)\u001b[0m\n\u001b[0;32m    346\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mfit\u001b[39m(\n\u001b[0;32m    347\u001b[0m     \u001b[39mself\u001b[39m,\n\u001b[0;32m    348\u001b[0m     X: Union[numpy\u001b[39m.\u001b[39mndarray, pandas\u001b[39m.\u001b[39mDataFrame],\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    351\u001b[0m     cv: Optional[Union[Iterable, List]] \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m,\n\u001b[0;32m    352\u001b[0m ):\n\u001b[0;32m    353\u001b[0m     \u001b[39m\"\"\"Fit the AutoML model.\u001b[39;00m\n\u001b[0;32m    354\u001b[0m \n\u001b[0;32m    355\u001b[0m \u001b[39m    Arguments:\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    366\u001b[0m \u001b[39m        AutoML object: Returns `self`\u001b[39;00m\n\u001b[0;32m    367\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[1;32m--> 368\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_fit(X, y, sample_weight, cv)\n\nFile \u001b[1;32mc:\\ProgramData\\Anaconda3\\lib\\site-packages\\supervised\\base_automl.py:1126\u001b[0m, in \u001b[0;36mBaseAutoML._fit\u001b[1;34m(self, X, y, sample_weight, cv)\u001b[0m\n\u001b[0;32m   1123\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose_print(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mAutoML best model: \u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_best_model\u001b[39m.\u001b[39mget_name()\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m   1125\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m-> 1126\u001b[0m     \u001b[39mraise\u001b[39;00m e\n\u001b[0;32m   1128\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\n\nFile \u001b[1;32mc:\\ProgramData\\Anaconda3\\lib\\site-packages\\supervised\\base_automl.py:1043\u001b[0m, in \u001b[0;36mBaseAutoML._fit\u001b[1;34m(self, X, y, sample_weight, cv)\u001b[0m\n\u001b[0;32m   1041\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39m\"\u001b[39m\u001b[39mhill_climbing\u001b[39m\u001b[39m\"\u001b[39m \u001b[39min\u001b[39;00m step \u001b[39mor\u001b[39;00m step \u001b[39min\u001b[39;00m [\u001b[39m\"\u001b[39m\u001b[39mensemble\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m\"\u001b[39m\u001b[39mstack\u001b[39m\u001b[39m\"\u001b[39m]:\n\u001b[0;32m   1042\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_models) \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m-> 1043\u001b[0m         \u001b[39mraise\u001b[39;00m AutoMLException(\n\u001b[0;32m   1044\u001b[0m             \u001b[39m\"\u001b[39m\u001b[39mNo models produced. \u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39mPlease check your data or\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m   1045\u001b[0m             \u001b[39m\"\u001b[39m\u001b[39m submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m   1046\u001b[0m         )\n\u001b[0;32m   1048\u001b[0m generated_params \u001b[39m=\u001b[39m []\n\u001b[0;32m   1049\u001b[0m \u001b[39mif\u001b[39;00m step \u001b[39min\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_all_params:\n\n\u001b[1;31mAutoMLException\u001b[0m: No models produced. \nPlease check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new."
}
Karlheinzniebuhr commented 1 year ago

Here is the content of errors.md

## Error for 2_Default_Xgboost

list index out of range
Traceback (most recent call last):
  File "c:\ProgramData\Anaconda3\lib\site-packages\supervised\base_automl.py", line 1089, in _fit
    params["final_loss"] = self._models[-1].get_final_loss()
IndexError: list index out of range

Please set a GitHub issue with above error message at: https://github.com/mljar/mljar-supervised/issues/new

## Error for 3_Default_CatBoost

list index out of range
Traceback (most recent call last):
  File "c:\ProgramData\Anaconda3\lib\site-packages\supervised\base_automl.py", line 1089, in _fit
    params["final_loss"] = self._models[-1].get_final_loss()
IndexError: list index out of range

Please set a GitHub issue with above error message at: https://github.com/mljar/mljar-supervised/issues/new

## Error for 4_Default_NeuralNetwork

list index out of range
Traceback (most recent call last):
  File "c:\ProgramData\Anaconda3\lib\site-packages\supervised\base_automl.py", line 1089, in _fit
    params["final_loss"] = self._models[-1].get_final_loss()
IndexError: list index out of range

Please set a GitHub issue with above error message at: https://github.com/mljar/mljar-supervised/issues/new

## Error for 5_Default_RandomForest

list index out of range
Traceback (most recent call last):
  File "c:\ProgramData\Anaconda3\lib\site-packages\supervised\base_automl.py", line 1089, in _fit
    params["final_loss"] = self._models[-1].get_final_loss()
IndexError: list index out of range

Please set a GitHub issue with above error message at: https://github.com/mljar/mljar-supervised/issues/new
pplonski commented 1 year ago

Hi @Karlheinzniebuhr, did you update any Python package or change something in data? There are no parameters generated thats why no models are trained.

Karlheinzniebuhr commented 1 year ago

Hi @Karlheinzniebuhr, did you update any Python package or change something in data? There are no parameters generated thats why no models are trained.

I did use a bigger dataset, I tried limiting the dataset to 1/10th and that worked. But once I used the whole dataset again the same exception occurs. I'm trying to run on linux too now to see if same problem happens

pplonski commented 1 year ago

@Karlheinzniebuhr try to set total_time_limit=6*3600 maybe too small time budget is the reason?

Karlheinzniebuhr commented 1 year ago

Indeed this seems to have been the issue, changed to 24*3600 and so far it is running and saving the progress. Thanks so much!

@Karlheinzniebuhr try to set total_time_limit=6*3600 maybe too small time budget is the reason?

pplonski commented 1 year ago

Great! @Karlheinzniebuhr do you get good results with AutoML?