Automated comparison of metrics (from Evaluator) to automatically pick the best forecasting model.

[antonkiselevtdg]

Description

Hi,

Thank you very much for this wonderful tool.

I would like to suggest the following:

I usually take a single data set and test it using several models one by one.

Then I use the Evaluator (after each test) and pick the best according to a specific score that I choose (sMAPE as an example).

It would be great if I could automatically generate the Evaluator table with multiple models in it, so I could asses them and pick the one that I would like to use for my data set based on a specific metric (or a combination of metrics).

This could help automate picking the best model for a given data set.

Also, I have tried to merge the results manually (using simple methods, like the following):

def Merge_Dictionaries(first, second):
    result = {**first_Dict, **second_Dict}
    return result

first_Dict = fbprophet_agg_metrics
second_Dict = npts_agg_metrics

third_Dict = Merge_Dictionaries(first_Dict, second_Dict)
print(third_Dict)

But could not properly do that. It shows only a single dict.

[jaheba]

Hi,

I believe your function isn't doing what you want to do. Since the metrics should be the same, it will just override the initial metrics.

Although some multiple-method support is something we are interested in, I think it would be easiest for you to just implement that bit of logic yourself, and adapt it to your specific needs.