search

jeslago / epftoolbox

An open-access benchmark and toolbox for electricity price forecasting
Apache License 2.0
212 stars 79 forks source link

Hyperparameter for Germany #9

Closed rderayati closed 5 months ago

rderayati commented 2 years ago

Hi, I tried to replicate the forecast for the day ahead electricity prices for Germany, but the sMAPEs are too high, contrasting with the markets, like PJM, BE, and NP. Did you also run the Hyperparameter optimization (with max evals 1500) for DE? If yes, why the forecast for the DE market is not accurate?

jeslago commented 2 years ago

Without further details it is not possible to help you. Please, when opening a ticket to report a bug, specify:

  1. What have you done exactly? (Explain with as many details as possible)
  2. Are you using the same dataset as in the study?
  3. What about the other metrics like MAE?
  4. What do you mean too high sMAPE?
  5. Can you post all the metrics?
rderayati commented 2 years ago

Actually, I did not mean to report a bug, I just wanted to ask for some explanation about the outcome.

I ran the recalibrating dnn simpified.py first. I used the "https://sandbox.zenodo.org/api/files/fb5bae17-de91-4ce7-b348-0d62e52824b5/DE.csv" which is the dataset you mentioned for Germany.

Then reran it for other markets such as PJM or BE and for both, sMAPE and MAE were noticeably lower than Germany's results. for example, sMAPE for Germany was above 30 % but for PJM was around 5%.

Considering the above explanation, I wondered if you also ran the Hyperparameter optimization script (with max evals 1500) for DE dataset to get experimental file/DNN_hyperparameters_nl2_datDE_YT2_SF_CW4_1? Because I think one of the reasons for the high error value in German's results can be due to hyperparameters optimization.

jeslago commented 2 years ago

I understand now. So a couple of things:

  1. We did run hyperparameter optimization for all the models and hyperparameters.
  2. Check rMAE rather than sMAPE. sMAPE is not the most reliable metric. MAE depends on the currency and marketplac.
  3. Compare also the values that you obtain to the ones provided in the paper (in terms of rMAE/MAE) so you can also get an idea of how far off you are from the "ideal" one without doing hyperparameter optimization.
jeslago commented 5 months ago

Closing due to inactivity and no clear action items