aimclub / FEDOT

Automated modeling and machine learning framework FEDOT
https://fedot.readthedocs.io
BSD 3-Clause "New" or "Revised" License
619 stars 84 forks source link

[Test]: ets, smoothing pipelines causing integration test failure #1279

Closed Lopa10ko closed 2 months ago

Lopa10ko commented 3 months ago

Forecasting specific integration test fails with smoothing and ets pipelines on the salaries dataset.

https://github.com/aimclub/FEDOT/blob/b711ebe2490f712ddbdd484e1f0fbf5883c0bcd7/test/integration/real_applications/test_examples.py#L86-L88

Current Behavior

Test fails with:

[!WARNING] Exception: Invalid fitness after objective evaluation. Skipping the graph: /n_ets_{'error': 'add', 'trend': 'mul', 'seasonal': None, 'damped_trend': True, 'seasonal_periods': 36.54298926868026}

Possible Solution

Predicted input contains NaNs:

[!WARNING]
Objective evaluation error for graph {'depth': 1, 'length': 1, 'nodes': [ets]} on metric mae: Metric can not be evaluated because of: Input contains NaN.

Here, after statsmodels evaluation, results contains NaN: https://github.com/aimclub/FEDOT/blob/b711ebe2490f712ddbdd484e1f0fbf5883c0bcd7/fedot/core/composer/metrics.py#L114-L133

Steps to Reproduce

In a fedot/core/optimisers/objective/data_objective_eval.py do something like prepared_pipeline.save(path='C:/model_troubled') right before the warning corresponding to a fitness invalidation. Since you saved the troubled pipeline, you can then pass this pipeline to an initial_assumption field in examples/simple/time_series_forecasting/api_forecasting.py::run_ts_forecasting_example

pipeline = Pipeline().load('C:/model_troubled/0_pipeline_saved')
model = Fedot(problem='ts_forecasting',
              task_params=Task(TaskTypesEnum.ts_forecasting,
                               TsForecastingParams(forecast_length=horizon)).task_params,
              timeout=timeout,
              n_jobs=-1,
              metric='mae',
              with_tuning=with_tuning,
              initial_assumption=pipeline)
Lopa10ko commented 3 months ago

possibly related to https://github.com/statsmodels/statsmodels/issues/6243