search

winedarksea / AutoTS

Automated Time Series Forecasting
MIT License
1.05k stars 96 forks source link

Additional metrics #220

Open ideusoes opened 6 months ago

ideusoes commented 6 months ago

As per the autoTS tutorial, metric_weighting allows the user to choose from a list of available metrics, but there is no option to use other metrics if they are not present in the dictionary keys.

I would like to use the MASE metric (Mean Absolute Scaled Error). Is there a plan to allow integration of custom metrics, or perhaps to add metrics requested by users ?

Thank you!

winedarksea commented 6 months ago

No current option to support custom metrics. Yes it is on my to do list but it is tricky because of how it is all set up.

For your immediate needs: SMAPE should correlate well with MASE. uwmse is another scaled error that would work. Between those you should be able to have model selection that is effectively the same as MASE, I should think.

You can access model.initial_results.per_series_mae and scale that pretty easily if you want to view results. For even more custom metrics you can run model.retrieve_validation_forecasts(models=[list of model ids]) and then calculate whatever you want afterwards.