search

robjhyndman / M4metalearning

116 stars 49 forks source link

Error Calculation Question #16

Closed Arsa-Nik closed 4 years ago

Arsa-Nik commented 5 years ago

Regarding error calculation, I was wondering why the OWA error for snaive method is not equal to one for each series. For instance, for the first series of monthly data, the snaive error, $errors['snaive_forec'] is 2.459 instead of 1.

Thanks, Arsa

robjhyndman commented 5 years ago

OWA is a mix of MASE and sMAPE. So it will not be equal to 1 for the snaive method.

Arsa-Nik commented 5 years ago

Aren’t snaive’s mase and smape supposed to be divided by snaive’s (or naive 2 in M4) mase and smape respectively to have relative measure of mase and smape before combining them with equal weight to have OWA?

robjhyndman commented 5 years ago

Naive 2 is not the same as Snaive. See https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

Arsa-Nik commented 5 years ago

Yes they are not the same. I was wondering since naive 2 does not exist among the 9 methods, you might have used Snaive instead to have the benchmark with OWA of 1.

robjhyndman commented 5 years ago

We have not redefined OWA. It is exactly as defined in the M4 competition.

Arsa-Nik commented 5 years ago

I could be completely wrong but the very last line of the code in the link shows the relative mase and mape calculation with respect to naive 2 in OWA. https://github.com/M4Competition/M4-methods/blob/master/Benchmarks%20and%20Evaluation.R

robjhyndman commented 5 years ago

Exactly. So they use naive 2 not snaive. The OWA for naive 2 should be 1. The OWA for snaive should be some other number.

Arsa-Nik commented 5 years ago

yes they use naive 2, however I couldn't find similar calculation relative to naive 2 in 'calc_errors' function. dataset[[i]]$errors <- 0.5 * (rowMeans(lentry$mase_err)/avg_snaive_errors$avg_mase + rowMeans(lentry$smape_err)/avg_snaive_errors$avg_smape)

Arsa-Nik commented 5 years ago

Sorry, I initially thought you are using snaive instead of naive 2 in the denominator and the rest of the calculation is the same as M4 guideline for OWA calculation for each series in 'calc_errors' function. however I realized you are taking the average of average of snaive's mase and smape for all series and using it in the denominator for each series. 1) Does this make OWA for each forecast method for each series vary based on data set size? 2) If we calculate OWA for each forecast method for each series independently and use naive 2 instead of snaive in the denominator, could it potentially affect gradient and hessian and improve the overall accuracy?