search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
391 stars 130 forks source link

[timeseries] Fixed metrics calculation. Added GluonTS for baselines. Improved AutoGluonTS implementation. #504

Closed limpbot closed 1 year ago

limpbot commented 1 year ago

Hi @sebhrusen, @PGijsbers, In this PR there are the following changes, I would highly appreciate it if you could review them.

Renaming forecast_range_in_steps to forecast_horizon_in_steps, because it is much more popular in the research field.

Fixed metrics calculation. A) Renamed y_past_period_error to naive_1_error, because we do not know periodicity/seasonality. B) Renamed ncrps (Normalized Continuous Ranked Probability Score) to mwql (Mean Weighted Quantile Loss), because that is what is actually calculated. This is an approximation of ncrps. C) Added item_id column to results, to be able to calculate itemwise_mean. D) Introduced finite_mean to calculate correct metrics, even if there are some NaNs.

Added GluonTS framework, because it allows to calculate multiple timeseries baselines. GluonTS_* 1) Prophet, 2) DeepAR, 3) NBEATS, 4) NPTS, 5) SeasonalNaive, 6) SimpleFeedForward, 7) MQCNN, 8) MQRNN, 9) TFT, 10) ARIMA, 11) ETS, 12) STL-AR, 13) Theta.

Added improvements for AutoGluonTS. A) Do not deepcopy the dataset to save memory. B) Use correct module to retrieve the AutoGluon version.

FYI: @Innixma @canerturkmen @gidler

limpbot commented 1 year ago

Cherry picked the relevant commits.

limpbot commented 1 year ago

Due to new internal discussion, I will close this PR for the time being. Thank you for starting commenting, I will keep your comments in mind!