probabilistic ML - Githubissues

openclimatefix / uk-pv-national-xg

National PV forecasting using Gradient Boosted Methods.

4 stars 3 forks source link

probabilistic ML #66

Closed peterdudfield closed 1 year ago

peterdudfield commented 1 year ago

Detailed Description

It would be great to get probabilistic forecasts

Context

Useful for user to know the uncertainty
A simplier approach would be to look at the errors over the last week and use this to estimate probabilities, but I think this might be to simple and not capture new weather events

Possible Implementation

perhaps we could use? https://xgboost.readthedocs.io/en/latest/python/examples/quantile_regression.html
Report for each horizon how good the quantile regressors are
for live this should come out as the pd.dataframe here.
Different code can sort it being saved in the database.

jacobbieker commented 1 year ago

The XGBoost built in quantile regression isn't available in stable yet, its been done in version 2.0 but not sure when that will be released.

peterdudfield commented 1 year ago

The XGBoost built in quantile regression isn't available in stable yet, its been done in version 2.0 but not sure when that will be released.

Worth a go?

jacobbieker commented 1 year ago

Yeah, we just have to compile it, which seems straightforward enough, especially if we dont need GPU support

jacobbieker commented 1 year ago

Example plot from that link above, it was a bit difficult compiling it and installing it correctly, but did get it working:

xgboost_test

jacobbieker commented 1 year ago

Might need to add something like this to the CI for tests to run with it https://github.com/dmlc/xgboost/blob/339f21e1bfa1b2d356b6b8318ab7cf9fb4baff36/.github/workflows/python_tests.yml#L207-L257

peterdudfield commented 1 year ago

GSP data is here - https://console.cloud.google.com/storage/browser/solar-pv-nowcasting-data/PV/GSP/v5

peterdudfield commented 1 year ago

NWP data will be here - https://console.cloud.google.com/storage/browser/solar-pv-nowcasting-data/NWP/UK_Met_Office/National_averaged

just uploading

jacobbieker commented 1 year ago

Here are some results from training the new model

prob_national_features

peterdudfield commented 1 year ago

Nice would you mind plotting true and forecast + CI (confidence intervals) on the same plot, just so we get an idea of things? Then we should think of some metrics for the CI's

jacobbieker commented 1 year ago

Yep, sounds good!

jacobbieker commented 1 year ago

Here is it combined the losses on a single plot. I'll plot a few examples soon

jacobbieker commented 1 year ago

And an example forecast example_forecast

jacobbieker commented 1 year ago

myplot3 myplot2 myplot1 myplot

peterdudfield commented 1 year ago

nice that looks really great!!!

jacobbieker commented 1 year ago

Q-Q Plots for Test set, median prediction vs truth quantiles: qq_36 qq_35 qq_34 qq_33 qq_32 qq_31 qq_30 qq_29 qq_28 qq_27 qq_26 qq_25 qq_24 qq_23 qq_22 qq_21 qq_20 qq_19 qq_18 qq_17 qq_16 qq_15 qq_14 qq_13 qq_12 qq_11 qq_10 qq_9 qq_8 qq_7 qq_6 qq_5 qq_4 qq_3 qq_2 qq_1 qq_0

peterdudfield commented 1 year ago

What do this graphs show @jacobbieker? Can you summurize the results?

jacobbieker commented 1 year ago

Yeah, sorta. These are Quantile-Quantile plots, with the y_axis being the median forecast from the National XG, and x_axis being the true generation, and these are plotting the quantiles for those data against each other. If they match perfectly, then it would be a straight line. I believe these are saying that the probabilistic national forecast is well calibrated vs the actual forecast, or at least closely follows the same distribution as the test dataset.

peterdudfield commented 1 year ago

Cool - thanks, shall I look at the PR? Is that ready?

jacobbieker commented 1 year ago

Its almost ready, I have to add building xgboost for the tests, and update the tests, which I'll do soon.

peterdudfield commented 1 year ago

and add it to inference? so that it works when running live

jacobbieker commented 1 year ago

Yeah

jacobbieker commented 1 year ago

In reference to this: https://github.com/openclimatefix/uk-pv-national-xg/pull/74#discussion_r1191380436 I've plotted a few selections of the test data and the 90th percentile, both for only daytime, and overall. To me, visually it looks right, in that it is larger for the majority of the points, which seems to contradict the numbers.

Overall, the 90th percentile output probably is not larger than 90% of the actual generation values, but in none of these plots does it look like its only larger than 37% of the values, like the data was suggesting. It does seem to do worse in the 1 hour forecast vs the 0th ones, but not that bad I don't think?

Edit: I found a bug, I was dividing by the whole dataset length on the daytime only values, making them too small. New values are being added to the readme file, but in general, its a lot closer, although the percentiles are still too far in. On daylight-only values, the 10th percentile is larger than around 28% of values, the 50th around 57%, and the 90th larger than 76% of values. So the outputs are still not as wide as we would expect, but a lot closer to what it should be.

The code I used to calculate the numbers from the predictions:

percentile_counts = []
    non_night_percentiles = []
    for idx, alpha in enumerate(ALPHA):
        y_pred_test_alpha = y_pred_test[:, idx]
        percentile_counts.append(np.sum(y_test['target'].values < y_pred_test_alpha))
        non_night_percentiles.append(
            np.sum(
                (y_test['target'].values < y_pred_test_alpha)
                & (y_test['target'].values > 0.01)
            )
        )
    # Get percentage of total test data that is below each percentile
    percentile_counts = np.array(percentile_counts) / len(y_test['target'].values)
    non_night_percentiles = np.array(non_night_percentiles) / len(y_test['target'].values)
    # Print out each of the pinball test losses and percentile counts
    for idx, alpha in enumerate(ALPHA):
        print(
            f"Percentile: {alpha}, test pinball: {np.round(test_pinballs[idx], 5)}, percentile count: {np.round(percentile_counts[idx], 5)} " \
            f"non-night percentile count: {np.round(non_night_percentiles[idx], 5)}"
        )

The plots from plotting the output from the model and the true generation for the 0th hour forecast:

0_actual_vs_90th_percentile_first_200

0_actual_vs_90th_percentile_first_200_daytime 0_actual_vs_90th_percentile_daytime

From the 1 hour forecast:

1_actual_vs_90th_percentile_first_1000 1_actual_vs_90th_percentile_daytime

2 hour forecast:

2_actual_vs_90th_percentile_first_200_daytime 2_actual_vs_90th_percentile_daytime

jacobbieker commented 1 year ago

Overall, it seems that the 10th percentile to 90th percentile only covers around 40-50% of the generation values, rather than the expected 80%, at least on the 2021 test set. This ranges from 24%-76% for 0th hour forecast, down to 30%-73% for the 16 hours ahead. One fix could be training it only on daytime generation, but then it wouldn't work well for night time.

peterdudfield commented 1 year ago

These numbers still seem a bit funny. Can you run them on the train set and see if there are what we expect

jacobbieker commented 1 year ago

Yep

jacobbieker commented 1 year ago

For the 0th hour forecast:

Percentile: 0.1, train pinball: 0.00261, percentile count: 0.55396 non-night percentile count: 0.10889
Percentile: 0.5, train pinball: 0.00444, percentile count: 0.74715 non-night percentile count: 0.49213
Percentile: 0.9, train pinball: 0.00245, percentile count: 0.93095 non-night percentile count: 0.85802

1 hour:

Percentile: 0.1, train pinball: 0.0041, percentile count: 0.55658 non-night percentile count: 0.11401
Percentile: 0.5, train pinball: 0.00686, percentile count: 0.75535 non-night percentile count: 0.51036
Percentile: 0.9, train pinball: 0.00377, percentile count: 0.93095 non-night percentile count: 0.85664

2 Hour:

Percentile: 0.1, train pinball: 0.00526, percentile count: 0.56136 non-night percentile count: 0.12175
Percentile: 0.5, train pinball: 0.00858, percentile count: 0.76184 non-night percentile count: 0.52448
Percentile: 0.9, train pinball: 0.00471, percentile count: 0.93067 non-night percentile count: 0.8534

4 Hour:

Percentile: 0.1, train pinball: 0.00543, percentile count: 0.55652 non-night percentile count: 0.10789
Percentile: 0.5, train pinball: 0.00879, percentile count: 0.74806 non-night percentile count: 0.49525
Percentile: 0.9, train pinball: 0.00486, percentile count: 0.93101 non-night percentile count: 0.85577

8 Hour:

Percentile: 0.1, train pinball: 0.0054, percentile count: 0.55817 non-night percentile count: 0.11464
Percentile: 0.5, train pinball: 0.00897, percentile count: 0.74858 non-night percentile count: 0.49563
Percentile: 0.9, train pinball: 0.00531, percentile count: 0.93448 non-night percentile count: 0.86214

16 hour:

Percentile: 0.1, train pinball: 0.00547, percentile count: 0.55914 non-night percentile count: 0.11774
Percentile: 0.5, train pinball: 0.00886, percentile count: 0.75877 non-night percentile count: 0.51792
Percentile: 0.9, train pinball: 0.00509, percentile count: 0.92862 non-night percentile count: 0.85104

24 Hour:

Percentile: 0.1, train pinball: 0.00518, percentile count: 0.55675 non-night percentile count: 0.11187
Percentile: 0.5, train pinball: 0.00886, percentile count: 0.76167 non-night percentile count: 0.52204
Percentile: 0.9, train pinball: 0.00525, percentile count: 0.93198 non-night percentile count: 0.85803

36 Hours:

Percentile: 0.1, train pinball: 0.00526, percentile count: 0.55755 non-night percentile count: 0.11373
Percentile: 0.5, train pinball: 0.00929, percentile count: 0.75774 non-night percentile count: 0.51536
Percentile: 0.9, train pinball: 0.00556, percentile count: 0.92902 non-night percentile count: 0.85218

jacobbieker commented 1 year ago

Others are running now, but first pass it looks pretty close to right for non-nighttime

jacobbieker commented 1 year ago

After running it, yeah, on the train set, the numbers are right around what we would expect to see.

dantravers commented 1 year ago

Thanks @jacobbieker - nice to find the bug. The numbers make sense now! The train ones are basically bang-on, whcih makes it feel like they calcualtions re now correct.

jacobbieker commented 1 year ago

Next Steps:

Train on 2021 and test on 2020 and see if quantile changes are the same
Process 2016-2021 (Or up to 2023 with new NWP data), and see if more data fixes the issue + model should be better than anyway

jacobbieker commented 1 year ago

Dump from training on 2021 and testing on 2020

/home/jacob/anaconda3/envs/national_xg/lib/python3.10/site-packages/nowcasting_datamodel/models/forecast.py:206: SAWarning: This declarative base already contains a class with the same class name and module name as nowcasting_datamodel.models.forecast.ForecastValueYearMonth, and will be replaced in the string-lookup table.
  class ForecastValueYearMonth(ForecastValueSQLMixin, Base_Forecast):
0
[15:53:10] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00282, percentile count: 0.53132 non-night percentile count: 0.11743
Percentile: 0.5, train pinball: 0.005, percentile count: 0.74744 non-night percentile count: 0.52492
Percentile: 0.9, train pinball: 0.0038, percentile count: 0.92279 non-night percentile count: 0.85077
Number of positive values for actual - 90th percentile: 3084/17568
Number of positive values for actual - 10th percentile: 6861/17568
Number of positive values for actual - median: 4916/17568
Percentile: 0.1, test pinball: 0.00282, percentile count: 0.60946 non-night percentile count: 0.20367
Percentile: 0.5, test pinball: 0.005, percentile count: 0.72017 non-night percentile count: 0.42308
Percentile: 0.9, test pinball: 0.0038, percentile count: 0.82445 non-night percentile count: 0.62612
Number of positive values for actual - 90th percentile: 2994/8008
Number of positive values for actual - 10th percentile: 6377/8008
Number of positive values for actual - median: 4620/8008
Median test MAE: 0.01
[INFO][2023-05-15 03:54:02] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010640953822223813, pinball_test_loss=0.005001878992738392, pinball_train_10_percentile_loss=0.001404232237700506, pinball_test_10_percentile_loss=0.0028194282260794548, pinball_train_90_percentile_loss=0.0006192746190488303, pinball_test_90_percentile_loss=0.0038008921153929788, mae_train_loss=0.0021281907644447626, mae_test_loss=0.010003757985476784, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
1
[15:54:05] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00388, percentile count: 0.53286 non-night percentile count: 0.12085
Percentile: 0.5, train pinball: 0.0074, percentile count: 0.75453 non-night percentile count: 0.54109
Percentile: 0.9, train pinball: 0.00658, percentile count: 0.92312 non-night percentile count: 0.85024
Number of positive values for actual - 90th percentile: 3394/17568
Number of positive values for actual - 10th percentile: 6769/17568
Number of positive values for actual - median: 5056/17568
Percentile: 0.1, test pinball: 0.00388, percentile count: 0.6147 non-night percentile count: 0.21566
Percentile: 0.5, test pinball: 0.0074, percentile count: 0.7122 non-night percentile count: 0.40534
Percentile: 0.9, test pinball: 0.00658, percentile count: 0.80681 non-night percentile count: 0.58429
Number of positive values for actual - 90th percentile: 3329/8008
Number of positive values for actual - 10th percentile: 6281/8008
Number of positive values for actual - median: 4762/8008
Median test MAE: 0.0148
[INFO][2023-05-15 03:55:16] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011104914595641766, pinball_test_loss=0.007401900459080166, pinball_train_10_percentile_loss=0.0016597589380497257, pinball_test_10_percentile_loss=0.0038831984857985538, pinball_train_90_percentile_loss=0.0007042405500994957, pinball_test_90_percentile_loss=0.006581151385691185, mae_train_loss=0.002220982919128353, mae_test_loss=0.014803800918160331, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
2
[15:55:19] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00457, percentile count: 0.53366 non-night percentile count: 0.11879
Percentile: 0.5, train pinball: 0.00915, percentile count: 0.75018 non-night percentile count: 0.52994
Percentile: 0.9, train pinball: 0.00842, percentile count: 0.92733 non-night percentile count: 0.85616
Number of positive values for actual - 90th percentile: 3626/17568
Number of positive values for actual - 10th percentile: 6763/17568
Number of positive values for actual - median: 5178/17568
Percentile: 0.1, test pinball: 0.00457, percentile count: 0.61504 non-night percentile count: 0.20929
Percentile: 0.5, test pinball: 0.00915, percentile count: 0.70526 non-night percentile count: 0.38362
Percentile: 0.9, test pinball: 0.00842, percentile count: 0.7936 non-night percentile count: 0.55445
Number of positive values for actual - 90th percentile: 3568/8008
Number of positive values for actual - 10th percentile: 6332/8008
Number of positive values for actual - median: 4936/8008
Median test MAE: 0.0183
[INFO][2023-05-15 03:56:28] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001144103634254023, pinball_test_loss=0.009148290727160136, pinball_train_10_percentile_loss=0.0018119470262555977, pinball_test_10_percentile_loss=0.004570263163533994, pinball_train_90_percentile_loss=0.0007678085038015272, pinball_test_90_percentile_loss=0.008420526516184515, mae_train_loss=0.002288207268508046, mae_test_loss=0.01829658145432027, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
4
[15:56:30] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00449, percentile count: 0.53413 non-night percentile count: 0.11948
Percentile: 0.5, train pinball: 0.00902, percentile count: 0.74704 non-night percentile count: 0.52498
Percentile: 0.9, train pinball: 0.00823, percentile count: 0.92473 non-night percentile count: 0.85327
Number of positive values for actual - 90th percentile: 3566/17568
Number of positive values for actual - 10th percentile: 6705/17568
Number of positive values for actual - median: 5076/17568
Percentile: 0.1, test pinball: 0.00449, percentile count: 0.61834 non-night percentile count: 0.21616
Percentile: 0.5, test pinball: 0.00902, percentile count: 0.71107 non-night percentile count: 0.3971
Percentile: 0.9, test pinball: 0.00823, percentile count: 0.79702 non-night percentile count: 0.56306
Number of positive values for actual - 90th percentile: 3499/8008
Number of positive values for actual - 10th percentile: 6277/8008
Number of positive values for actual - median: 4828/8008
Median test MAE: 0.01804
[INFO][2023-05-15 03:57:25] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011438906938244222, pinball_test_loss=0.00901948458437547, pinball_train_10_percentile_loss=0.001847795746115881, pinball_test_10_percentile_loss=0.004491232350364743, pinball_train_90_percentile_loss=0.0007802793007906134, pinball_test_90_percentile_loss=0.008231752014050476, mae_train_loss=0.0022877813876488445, mae_test_loss=0.01803896916875094, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
8
[15:57:33] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00464, percentile count: 0.52978 non-night percentile count: 0.11232
Percentile: 0.5, train pinball: 0.00905, percentile count: 0.74056 non-night percentile count: 0.51177
Percentile: 0.9, train pinball: 0.00816, percentile count: 0.92499 non-night percentile count: 0.85382
Number of positive values for actual - 90th percentile: 3457/17568
Number of positive values for actual - 10th percentile: 6631/17568
Number of positive values for actual - median: 5006/17568
Percentile: 0.1, test pinball: 0.00464, percentile count: 0.62255 non-night percentile count: 0.22453
Percentile: 0.5, test pinball: 0.00905, percentile count: 0.71505 non-night percentile count: 0.40659
Percentile: 0.9, test pinball: 0.00816, percentile count: 0.80322 non-night percentile count: 0.5763
Number of positive values for actual - 90th percentile: 3393/8008
Number of positive values for actual - 10th percentile: 6210/8008
Number of positive values for actual - median: 4752/8008
Median test MAE: 0.0181
[INFO][2023-05-15 03:59:01] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011402334015802717, pinball_test_loss=0.009048043443156828, pinball_train_10_percentile_loss=0.0018385429349155262, pinball_test_10_percentile_loss=0.004643507940825476, pinball_train_90_percentile_loss=0.0007916722942069105, pinball_test_90_percentile_loss=0.008163335700549341, mae_train_loss=0.0022804668031605434, mae_test_loss=0.018096086886313655, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
16
[15:59:04] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00456, percentile count: 0.53379 non-night percentile count: 0.1201
Percentile: 0.5, train pinball: 0.00938, percentile count: 0.74517 non-night percentile count: 0.51931
Percentile: 0.9, train pinball: 0.00899, percentile count: 0.92526 non-night percentile count: 0.8537
Number of positive values for actual - 90th percentile: 3519/17568
Number of positive values for actual - 10th percentile: 6754/17568
Number of positive values for actual - median: 5109/17568
Percentile: 0.1, test pinball: 0.00456, percentile count: 0.61555 non-night percentile count: 0.21114
Percentile: 0.5, test pinball: 0.00938, percentile count: 0.70919 non-night percentile count: 0.39256
Percentile: 0.9, test pinball: 0.00899, percentile count: 0.79969 non-night percentile count: 0.56911
Number of positive values for actual - 90th percentile: 3451/8009
Number of positive values for actual - 10th percentile: 6318/8009
Number of positive values for actual - median: 4865/8009
Median test MAE: 0.01877
[INFO][2023-05-15 04:00:15] : Trained model for 16 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011590466974939694, pinball_test_loss=0.009383942765958034, pinball_train_10_percentile_loss=0.0017947936152005292, pinball_test_10_percentile_loss=0.004558433889922194, pinball_train_90_percentile_loss=0.0007731039416195466, pinball_test_90_percentile_loss=0.008986563054281572, mae_train_loss=0.002318093394987939, mae_test_loss=0.018767885531916068, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
24
[16:00:17] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00481, percentile count: 0.53272 non-night percentile count: 0.11987
Percentile: 0.5, train pinball: 0.00957, percentile count: 0.74998 non-night percentile count: 0.53165
Percentile: 0.9, train pinball: 0.00872, percentile count: 0.91851 non-night percentile count: 0.84778
Number of positive values for actual - 90th percentile: 3577/17568
Number of positive values for actual - 10th percentile: 6660/17568
Number of positive values for actual - median: 5031/17568
Percentile: 0.1, test pinball: 0.00481, percentile count: 0.6209 non-night percentile count: 0.2205
Percentile: 0.5, test pinball: 0.00957, percentile count: 0.71363 non-night percentile count: 0.40267
Percentile: 0.9, test pinball: 0.00872, percentile count: 0.79639 non-night percentile count: 0.57111
Number of positive values for actual - 90th percentile: 3435/8009
Number of positive values for actual - 10th percentile: 6243/8009
Number of positive values for actual - median: 4784/8009
Median test MAE: 0.01914
[INFO][2023-05-15 04:01:39] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011689183464406079, pinball_test_loss=0.009572480399415753, pinball_train_10_percentile_loss=0.0018157064921232367, pinball_test_10_percentile_loss=0.0048107734662148445, pinball_train_90_percentile_loss=0.0007840143375788614, pinball_test_90_percentile_loss=0.00872483492972501, mae_train_loss=0.0023378366928812158, mae_test_loss=0.019144960798831507, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
36
[16:01:42] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00479, percentile count: 0.53332 non-night percentile count: 0.12051
Percentile: 0.5, train pinball: 0.01021, percentile count: 0.75419 non-night percentile count: 0.54111
Percentile: 0.9, train pinball: 0.00965, percentile count: 0.92446 non-night percentile count: 0.85195
Number of positive values for actual - 90th percentile: 3521/17568
Number of positive values for actual - 10th percentile: 6705/17568
Number of positive values for actual - median: 5056/17568
Percentile: 0.1, test pinball: 0.00479, percentile count: 0.61834 non-night percentile count: 0.2176
Percentile: 0.5, test pinball: 0.01021, percentile count: 0.7122 non-night percentile count: 0.401
Percentile: 0.9, test pinball: 0.00965, percentile count: 0.79958 non-night percentile count: 0.56879
Number of positive values for actual - 90th percentile: 3454/8010
Number of positive values for actual - 10th percentile: 6267/8010
Number of positive values for actual - median: 4798/8010
Median test MAE: 0.02042

jacobbieker commented 1 year ago

Training on 2018-2020 and test on 2021 seems to give better quantiles on test, although still not matching train.

0
[17:09:50] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00113, percentile count: 0.5581 non-night percentile count: 0.11856
Percentile: 0.5, train pinball: 0.00099, percentile count: 0.75538 non-night percentile count: 0.50943
Percentile: 0.9, train pinball: 0.00058, percentile count: 0.93545 non-night percentile count: 0.866
Number of positive values for actual - 90th percentile: 1842/14959
Number of positive values for actual - 10th percentile: 6113/14959
Number of positive values for actual - median: 3715/14959
Percentile: 0.1, test pinball: 0.00214, percentile count: 0.59135 non-night percentile count: 0.21572
Percentile: 0.5, test pinball: 0.00382, percentile count: 0.75165 non-night percentile count: 0.51803
Percentile: 0.9, test pinball: 0.00226, percentile count: 0.87686 non-night percentile count: 0.75564
Number of positive values for actual - 90th percentile: 1775/7264
Number of positive values for actual - 10th percentile: 5697/7264
Number of positive values for actual - median: 3501/7264
[INFO][2023-06-11 05:11:31] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.000991247321062835, pinball_test_loss=0.003824923673035013, pinball_train_10_percentile_loss=0.0011261347527875322, pinball_test_10_percentile_loss=0.0021402175620202414, pinball_train_90_percentile_loss=0.000579076802607529, pinball_test_90_percentile_loss=0.002264250145383788, mae_train_loss=0.00198249464212567, mae_test_loss=0.007649847346070026, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.00765
1
[17:11:33] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00141, percentile count: 0.55734 non-night percentile count: 0.11772
Percentile: 0.5, train pinball: 0.0011, percentile count: 0.76178 non-night percentile count: 0.5224
Percentile: 0.9, train pinball: 0.00073, percentile count: 0.93358 non-night percentile count: 0.862
Number of positive values for actual - 90th percentile: 1970/14959
Number of positive values for actual - 10th percentile: 5879/14959
Number of positive values for actual - median: 3610/14959
Percentile: 0.1, test pinball: 0.00338, percentile count: 0.60699 non-night percentile count: 0.24611
Percentile: 0.5, test pinball: 0.00602, percentile count: 0.75867 non-night percentile count: 0.53214
Percentile: 0.9, test pinball: 0.0036, percentile count: 0.86831 non-night percentile count: 0.7382
Number of positive values for actual - 90th percentile: 1902/7265
Number of positive values for actual - 10th percentile: 5477/7265
Number of positive values for actual - median: 3399/7265
[INFO][2023-06-11 05:13:19] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011013967111184412, pinball_test_loss=0.00602311128233597, pinball_train_10_percentile_loss=0.0014071095129575713, pinball_test_10_percentile_loss=0.0033825036821428307, pinball_train_90_percentile_loss=0.000729664642589215, pinball_test_90_percentile_loss=0.0036011592099093504, mae_train_loss=0.0022027934222368824, mae_test_loss=0.01204622256467194, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01205
2
[17:13:22] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00165, percentile count: 0.56583 non-night percentile count: 0.13352
Percentile: 0.5, train pinball: 0.00113, percentile count: 0.76073 non-night percentile count: 0.51968
Percentile: 0.9, train pinball: 0.00079, percentile count: 0.93083 non-night percentile count: 0.85569
Number of positive values for actual - 90th percentile: 2005/14959
Number of positive values for actual - 10th percentile: 5669/14959
Number of positive values for actual - median: 3519/14959
Percentile: 0.1, test pinball: 0.00465, percentile count: 0.62103 non-night percentile count: 0.2735
Percentile: 0.5, test pinball: 0.00824, percentile count: 0.76476 non-night percentile count: 0.54315
Percentile: 0.9, test pinball: 0.00505, percentile count: 0.86597 non-night percentile count: 0.73008
Number of positive values for actual - 90th percentile: 1961/7265
Number of positive values for actual - 10th percentile: 5278/7265
Number of positive values for actual - median: 3319/7265
Median test MAE: 0.01647
4
[INFO][2023-06-11 05:15:10] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001131806461778474, pinball_test_loss=0.008236535415238545, pinball_train_10_percentile_loss=0.001647585960302832, pinball_test_10_percentile_loss=0.004651346635391553, pinball_train_90_percentile_loss=0.0007898988668456825, pinball_test_90_percentile_loss=0.0050475711607327585, mae_train_loss=0.002263612923556948, mae_test_loss=0.01647307083047709, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[17:15:12] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00166, percentile count: 0.56413 non-night percentile count: 0.13086
Percentile: 0.5, train pinball: 0.00114, percentile count: 0.76008 non-night percentile count: 0.51754
Percentile: 0.9, train pinball: 0.0008, percentile count: 0.93344 non-night percentile count: 0.86128
Number of positive values for actual - 90th percentile: 2066/14959
Number of positive values for actual - 10th percentile: 5659/14959
Number of positive values for actual - median: 3599/14959
Percentile: 0.1, test pinball: 0.00477, percentile count: 0.6217 non-night percentile count: 0.27392
Percentile: 0.5, test pinball: 0.00852, percentile count: 0.75941 non-night percentile count: 0.53255
Percentile: 0.9, test pinball: 0.00524, percentile count: 0.86189 non-night percentile count: 0.72251
Number of positive values for actual - 90th percentile: 2016/7265
Number of positive values for actual - 10th percentile: 5275/7265
Number of positive values for actual - median: 3396/7265
Median test MAE: 0.01704
8
[INFO][2023-06-11 05:17:02] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011383940226437211, pinball_test_loss=0.00852058118304746, pinball_train_10_percentile_loss=0.0016618212298398405, pinball_test_10_percentile_loss=0.004774271670976112, pinball_train_90_percentile_loss=0.0007959981748023115, pinball_test_90_percentile_loss=0.00523771071201421, mae_train_loss=0.0022767880452874423, mae_test_loss=0.01704116236609492, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[17:17:04] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00167, percentile count: 0.56486 non-night percentile count: 0.13191
Percentile: 0.5, train pinball: 0.00114, percentile count: 0.75418 non-night percentile count: 0.50516
Percentile: 0.9, train pinball: 0.00079, percentile count: 0.92999 non-night percentile count: 0.85413
Number of positive values for actual - 90th percentile: 2012/14959
Number of positive values for actual - 10th percentile: 5663/14959
Number of positive values for actual - median: 3532/14959
Percentile: 0.1, test pinball: 0.00489, percentile count: 0.62143 non-night percentile count: 0.27529
Percentile: 0.5, test pinball: 0.00855, percentile count: 0.76389 non-night percentile count: 0.54136
Percentile: 0.9, test pinball: 0.00521, percentile count: 0.8655 non-night percentile count: 0.72966
Number of positive values for actual - 90th percentile: 1964/7265
Number of positive values for actual - 10th percentile: 5265/7265
Number of positive values for actual - median: 3332/7265
Median test MAE: 0.0171
16
[INFO][2023-06-11 05:19:02] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011419371791192908, pinball_test_loss=0.008549507090669357, pinball_train_10_percentile_loss=0.00167450452114518, pinball_test_10_percentile_loss=0.0048922190635565434, pinball_train_90_percentile_loss=0.0007850899426940885, pinball_test_90_percentile_loss=0.0052122115306586355, mae_train_loss=0.0022838743582385816, mae_test_loss=0.017099014181338714, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[17:19:05] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00166, percentile count: 0.56595 non-night percentile count: 0.13302
Percentile: 0.5, train pinball: 0.00114, percentile count: 0.75342 non-night percentile count: 0.50283
Percentile: 0.9, train pinball: 0.00078, percentile count: 0.93388 non-night percentile count: 0.86208
Number of positive values for actual - 90th percentile: 2032/14959
Number of positive values for actual - 10th percentile: 5614/14959
Number of positive values for actual - median: 3589/14959
Percentile: 0.1, test pinball: 0.00493, percentile count: 0.62471 non-night percentile count: 0.2813
Percentile: 0.5, test pinball: 0.00854, percentile count: 0.76008 non-night percentile count: 0.53254
Percentile: 0.9, test pinball: 0.0052, percentile count: 0.86416 non-night percentile count: 0.72642
Number of positive values for actual - 90th percentile: 1984/7252
Number of positive values for actual - 10th percentile: 5212/7252
Number of positive values for actual - median: 3390/7252
[INFO][2023-06-11 05:21:09] : Trained model for 16 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001143120927084893, pinball_test_loss=0.008537761795531878, pinball_train_10_percentile_loss=0.0016640740534759181, pinball_test_10_percentile_loss=0.004928152946556026, pinball_train_90_percentile_loss=0.0007786145341613185, pinball_test_90_percentile_loss=0.0052018175697380485, mae_train_loss=0.002286241854169786, mae_test_loss=0.017075523591063756, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01708
24
[17:21:12] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00165, percentile count: 0.56218 non-night percentile count: 0.12692
Percentile: 0.5, train pinball: 0.00119, percentile count: 0.75726 non-night percentile count: 0.51291
Percentile: 0.9, train pinball: 0.00082, percentile count: 0.93184 non-night percentile count: 0.85831
Number of positive values for actual - 90th percentile: 2142/14959
Number of positive values for actual - 10th percentile: 5683/14959
Number of positive values for actual - median: 3604/14959
Percentile: 0.1, test pinball: 0.00485, percentile count: 0.62009 non-night percentile count: 0.27168
Percentile: 0.5, test pinball: 0.00868, percentile count: 0.75907 non-night percentile count: 0.53276
Percentile: 0.9, test pinball: 0.00533, percentile count: 0.85681 non-night percentile count: 0.71305
Number of positive values for actual - 90th percentile: 2085/7266
Number of positive values for actual - 10th percentile: 5292/7266
Number of positive values for actual - median: 3395/7266
[INFO][2023-06-11 05:23:10] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001186544185965205, pinball_test_loss=0.008676519424807173, pinball_train_10_percentile_loss=0.0016542010491550187, pinball_test_10_percentile_loss=0.004853736523650258, pinball_train_90_percentile_loss=0.000821182447214496, pinball_test_90_percentile_loss=0.005334800608221614, mae_train_loss=0.00237308837193041, mae_test_loss=0.017353038849614345, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01735
36
[17:23:13] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.0018, percentile count: 0.56627 non-night percentile count: 0.13492
Percentile: 0.5, train pinball: 0.00117, percentile count: 0.75939 non-night percentile count: 0.51772
Percentile: 0.9, train pinball: 0.00082, percentile count: 0.93132 non-night percentile count: 0.85687
Number of positive values for actual - 90th percentile: 2078/14959
Number of positive values for actual - 10th percentile: 5592/14959
Number of positive values for actual - median: 3539/14959
Percentile: 0.1, test pinball: 0.00489, percentile count: 0.62618 non-night percentile count: 0.28357
Percentile: 0.5, test pinball: 0.0091, percentile count: 0.76342 non-night percentile count: 0.54221
Percentile: 0.9, test pinball: 0.00559, percentile count: 0.86109 non-night percentile count: 0.72084
Number of positive values for actual - 90th percentile: 2027/7261
Number of positive values for actual - 10th percentile: 5202/7261
Number of positive values for actual - median: 3324/7261
[INFO][2023-06-11 05:25:01] : Trained model for 36 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011716316185477846, pinball_test_loss=0.009104774227201469, pinball_train_10_percentile_loss=0.0018003260658481421, pinball_test_10_percentile_loss=0.0048882410622952946, pinball_train_90_percentile_loss=0.0008197396722700832, pinball_test_90_percentile_loss=0.0055880302006493885, mae_train_loss=0.0023432632370955692, mae_test_loss=0.018209548454402938, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01821

jacobbieker commented 1 year ago

Adding 2 more years of training data seems to have changed the 10th/90th quantiles on the test from covering 30-35% to 40-50% of the test set, when they should be covering ~80%. So it does improve it, and adding more years seems like it would make it closer to what we want, but we only have 2016-2023 years of NWP data. So we can train on 2016-2022 and test on 2023 possibly? Or train on 2016-2021 and test on 2022 to give another 3 more years of training data

jacobbieker commented 1 year ago

Trained with 2016 included as well (no 2017 data yet) Quantiles now cover between 45-55%, so around the 5% uplift per year that adding the other two years was looking at as well. So if this holds up, then training on 2017,2021, and 2022 should give quantiles of between 60-70%, so still 10-20% less than they should, but a lot closer.

0
[10:58:57] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00112, percentile count: 0.55129 non-night percentile count: 0.11297
Percentile: 0.5, train pinball: 0.00103, percentile count: 0.75955 non-night percentile count: 0.52364
Percentile: 0.9, train pinball: 0.00059, percentile count: 0.93549 non-night percentile count: 0.86865
Number of positive values for actual - 90th percentile: 1818/14959
Number of positive values for actual - 10th percentile: 6170/14959
Number of positive values for actual - median: 3701/14959
Percentile: 0.1, test pinball: 0.00207, percentile count: 0.58754 non-night percentile count: 0.20925
Percentile: 0.5, test pinball: 0.00378, percentile count: 0.75259 non-night percentile count: 0.52271
Percentile: 0.9, test pinball: 0.00225, percentile count: 0.87847 non-night percentile count: 0.75854
Number of positive values for actual - 90th percentile: 1754/7264
Number of positive values for actual - 10th percentile: 5744/7264
Number of positive values for actual - median: 3467/7264
Median test MAE: 0.00757
1
[INFO][2023-06-19 11:01:45] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010313230854484088, pinball_test_loss=0.003784108528609259, pinball_train_10_percentile_loss=0.001122361775271996, pinball_test_10_percentile_loss=0.002070283260569465, pinball_train_90_percentile_loss=0.0005934654327998835, pinball_test_90_percentile_loss=0.0022458097430846868, mae_train_loss=0.0020626461708968175, mae_test_loss=0.007568217057218518, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[11:01:48] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00142, percentile count: 0.5549 non-night percentile count: 0.12152
Percentile: 0.5, train pinball: 0.00115, percentile count: 0.7577 non-night percentile count: 0.51853
Percentile: 0.9, train pinball: 0.00075, percentile count: 0.93151 non-night percentile count: 0.85902
Number of positive values for actual - 90th percentile: 1959/14959
Number of positive values for actual - 10th percentile: 6020/14959
Number of positive values for actual - median: 3731/14959
Percentile: 0.1, test pinball: 0.00314, percentile count: 0.59757 non-night percentile count: 0.22849
Percentile: 0.5, test pinball: 0.00585, percentile count: 0.75058 non-night percentile count: 0.51672
Percentile: 0.9, test pinball: 0.00359, percentile count: 0.86904 non-night percentile count: 0.73889
Number of positive values for actual - 90th percentile: 1897/7265
Number of positive values for actual - 10th percentile: 5605/7265
Number of positive values for actual - median: 3511/7265
Median test MAE: 0.0117
2
[INFO][2023-06-19 11:04:48] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001154322442606701, pinball_test_loss=0.005852364995757857, pinball_train_10_percentile_loss=0.0014173748253169466, pinball_test_10_percentile_loss=0.0031419054591439494, pinball_train_90_percentile_loss=0.0007501659937747078, pinball_test_90_percentile_loss=0.003591006193813399, mae_train_loss=0.002308644885213402, mae_test_loss=0.011704729991515714, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[11:04:50] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00171, percentile count: 0.56199 non-night percentile count: 0.13496
Percentile: 0.5, train pinball: 0.0012, percentile count: 0.76088 non-night percentile count: 0.52295
Percentile: 0.9, train pinball: 0.00084, percentile count: 0.93139 non-night percentile count: 0.85711
Number of positive values for actual - 90th percentile: 2099/14959
Number of positive values for actual - 10th percentile: 5916/14959
Number of positive values for actual - median: 3719/14959
Percentile: 0.1, test pinball: 0.00443, percentile count: 0.60452 non-night percentile count: 0.24322
Percentile: 0.5, test pinball: 0.00821, percentile count: 0.75139 non-night percentile count: 0.5181
Percentile: 0.9, test pinball: 0.00532, percentile count: 0.85968 non-night percentile count: 0.71838
Number of positive values for actual - 90th percentile: 2046/7265
Number of positive values for actual - 10th percentile: 5498/7265
Number of positive values for actual - median: 3501/7265
[INFO][2023-06-19 11:07:46] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0011984495362803457, pinball_test_loss=0.008208094115134018, pinball_train_10_percentile_loss=0.0017111383597786585, pinball_test_10_percentile_loss=0.0044258880216503415, pinball_train_90_percentile_loss=0.0008363107708217844, pinball_test_90_percentile_loss=0.005319124166917135, mae_train_loss=0.0023968990725606915, mae_test_loss=0.016416188230268036, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01642
4
[11:07:48] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00172, percentile count: 0.56105 non-night percentile count: 0.13315
Percentile: 0.5, train pinball: 0.00121, percentile count: 0.75852 non-night percentile count: 0.51812
Percentile: 0.9, train pinball: 0.00084, percentile count: 0.93014 non-night percentile count: 0.85499
Number of positive values for actual - 90th percentile: 2107/14959
Number of positive values for actual - 10th percentile: 5889/14959
Number of positive values for actual - median: 3714/14959
Percentile: 0.1, test pinball: 0.00455, percentile count: 0.60632 non-night percentile count: 0.24666
Percentile: 0.5, test pinball: 0.00843, percentile count: 0.75172 non-night percentile count: 0.51989
Percentile: 0.9, test pinball: 0.0054, percentile count: 0.85915 non-night percentile count: 0.71796
Number of positive values for actual - 90th percentile: 2049/7265
Number of positive values for actual - 10th percentile: 5473/7265
Number of positive values for actual - median: 3488/7265
Median test MAE: 0.01686
8
[INFO][2023-06-19 11:10:43] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012050679249539108, pinball_test_loss=0.00842812916283893, pinball_train_10_percentile_loss=0.0017204405113389763, pinball_test_10_percentile_loss=0.004549523598784022, pinball_train_90_percentile_loss=0.0008365365465787252, pinball_test_90_percentile_loss=0.0053977606611368256, mae_train_loss=0.0024101358499078215, mae_test_loss=0.01685625832567786, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[11:10:45] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00171, percentile count: 0.56006 non-night percentile count: 0.13056
Percentile: 0.5, train pinball: 0.00121, percentile count: 0.75743 non-night percentile count: 0.51632
Percentile: 0.9, train pinball: 0.00082, percentile count: 0.93101 non-night percentile count: 0.85656
Number of positive values for actual - 90th percentile: 2075/14959
Number of positive values for actual - 10th percentile: 5811/14959
Number of positive values for actual - median: 3659/14959
Percentile: 0.1, test pinball: 0.00471, percentile count: 0.61154 non-night percentile count: 0.25712
Percentile: 0.5, test pinball: 0.00846, percentile count: 0.7554 non-night percentile count: 0.52512
Percentile: 0.9, test pinball: 0.00532, percentile count: 0.86129 non-night percentile count: 0.72195
Number of positive values for actual - 90th percentile: 2020/7265
Number of positive values for actual - 10th percentile: 5397/7265
Number of positive values for actual - median: 3450/7265
Median test MAE: 0.01692
16
[INFO][2023-06-19 11:13:34] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012051481036356008, pinball_test_loss=0.008460924781208266, pinball_train_10_percentile_loss=0.001709002735841087, pinball_test_10_percentile_loss=0.004708857811255197, pinball_train_90_percentile_loss=0.0008176030703524811, pinball_test_90_percentile_loss=0.0053155217263090295, mae_train_loss=0.0024102962072712016, mae_test_loss=0.016921849562416533, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[11:13:37] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00169, percentile count: 0.5604 non-night percentile count: 0.13155
Percentile: 0.5, train pinball: 0.00122, percentile count: 0.75647 non-night percentile count: 0.51419
Percentile: 0.9, train pinball: 0.00082, percentile count: 0.93026 non-night percentile count: 0.85472
Number of positive values for actual - 90th percentile: 1880/14959
Number of positive values for actual - 10th percentile: 5716/14959
Number of positive values for actual - median: 3497/14959
Percentile: 0.1, test pinball: 0.0048, percentile count: 0.61789 non-night percentile count: 0.26875
Percentile: 0.5, test pinball: 0.00847, percentile count: 0.76623 non-night percentile count: 0.54868
Percentile: 0.9, test pinball: 0.0049, percentile count: 0.87432 non-night percentile count: 0.74779
Number of positive values for actual - 90th percentile: 1829/7252
Number of positive values for actual - 10th percentile: 5303/7252
Number of positive values for actual - median: 3273/7252
Median test MAE: 0.01693
24
[INFO][2023-06-19 11:16:26] : Trained model for 16 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012175564683692187, pinball_test_loss=0.008465742302343912, pinball_train_10_percentile_loss=0.0016889762741540124, pinball_test_10_percentile_loss=0.004801480830866383, pinball_train_90_percentile_loss=0.0008221306182888215, pinball_test_90_percentile_loss=0.004903527604208744, mae_train_loss=0.0024351129367384374, mae_test_loss=0.016931484604687824, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[11:16:28] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00171, percentile count: 0.56258 non-night percentile count: 0.13627
Percentile: 0.5, train pinball: 0.00126, percentile count: 0.75341 non-night percentile count: 0.50769
Percentile: 0.9, train pinball: 0.00086, percentile count: 0.93066 non-night percentile count: 0.85681
Number of positive values for actual - 90th percentile: 1923/14959
Number of positive values for actual - 10th percentile: 5660/14959
Number of positive values for actual - median: 3452/14959
Percentile: 0.1, test pinball: 0.00488, percentile count: 0.62163 non-night percentile count: 0.27842
Percentile: 0.5, test pinball: 0.0087, percentile count: 0.76924 non-night percentile count: 0.55601
Percentile: 0.9, test pinball: 0.00493, percentile count: 0.87145 non-night percentile count: 0.74415
Number of positive values for actual - 90th percentile: 1859/7266
Number of positive values for actual - 10th percentile: 5243/7266
Number of positive values for actual - median: 3226/7266
[INFO][2023-06-19 11:19:43] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012638366997457624, pinball_test_loss=0.008698372211020213, pinball_train_10_percentile_loss=0.0017122634513579232, pinball_test_10_percentile_loss=0.004876328463486481, pinball_train_90_percentile_loss=0.0008647757427491676, pinball_test_90_percentile_loss=0.004931883858473683, mae_train_loss=0.0025276733994915247, mae_test_loss=0.017396744422040426, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.0174
36
[11:19:45] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00182, percentile count: 0.56119 non-night percentile count: 0.13363
Percentile: 0.5, train pinball: 0.00125, percentile count: 0.76006 non-night percentile count: 0.52275
Percentile: 0.9, train pinball: 0.00087, percentile count: 0.92981 non-night percentile count: 0.85447
Number of positive values for actual - 90th percentile: 1847/14959
Number of positive values for actual - 10th percentile: 5546/14959
Number of positive values for actual - median: 3320/14959
Percentile: 0.1, test pinball: 0.00508, percentile count: 0.62925 non-night percentile count: 0.2928
Percentile: 0.5, test pinball: 0.00919, percentile count: 0.77806 non-night percentile count: 0.57155
Percentile: 0.9, test pinball: 0.0052, percentile count: 0.87653 non-night percentile count: 0.75224
Number of positive values for actual - 90th percentile: 1799/7261
Number of positive values for actual - 10th percentile: 5135/7261
Number of positive values for actual - median: 3111/7261
Median test MAE: 0.01838
[INFO][2023-06-19 11:23:02] : Trained model for 36 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.001248302406746576, pinball_test_loss=0.009192272618903836, pinball_train_10_percentile_loss=0.0018219932714225783, pinball_test_10_percentile_loss=0.005083545448372809, pinball_train_90_percentile_loss=0.0008693054901977024, pinball_test_90_percentile_loss=0.00520123916741499, mae_train_loss=0.002496604813493152, mae_test_loss=0.01838454523780767, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

jacobbieker commented 1 year ago

With training on 2016-2020, and 2022 and testing on 2021, the results aren't going up as well as hoped. Range is still 43-57%, not the 60-65% I was thinking it would improve to:

0
[13:26:49] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00107, percentile count: 0.54925 non-night percentile count: 0.11581
Percentile: 0.5, train pinball: 0.00107, percentile count: 0.75897 non-night percentile count: 0.52373
Percentile: 0.9, train pinball: 0.00058, percentile count: 0.93605 non-night percentile count: 0.87016
Number of positive values for actual - 90th percentile: 1771/14959
Number of positive values for actual - 10th percentile: 6162/14959
Number of positive values for actual - median: 3703/14959
Percentile: 0.1, test pinball: 0.00202, percentile count: 0.58807 non-night percentile count: 0.21311
Percentile: 0.5, test pinball: 0.00372, percentile count: 0.75246 non-night percentile count: 0.52203
Percentile: 0.9, test pinball: 0.00219, percentile count: 0.88161 non-night percentile count: 0.76487
Number of positive values for actual - 90th percentile: 1708/7264
Number of positive values for actual - 10th percentile: 5716/7264
Number of positive values for actual - median: 3472/7264
Median test MAE: 0.00744
1
[INFO][2023-07-03 01:30:15] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010700763538689128, pinball_test_loss=0.003722452997571725, pinball_train_10_percentile_loss=0.0010749812949920406, pinball_test_10_percentile_loss=0.002017115131622491, pinball_train_90_percentile_loss=0.0005842200814287701, pinball_test_90_percentile_loss=0.002187505050996092, mae_train_loss=0.0021401527077378256, mae_test_loss=0.00744490599514345, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[13:30:18] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00136, percentile count: 0.55233 non-night percentile count: 0.12138
Percentile: 0.5, train pinball: 0.00121, percentile count: 0.76367 non-night percentile count: 0.53363
Percentile: 0.9, train pinball: 0.00075, percentile count: 0.92866 non-night percentile count: 0.85348
Number of positive values for actual - 90th percentile: 1961/14959
Number of positive values for actual - 10th percentile: 6050/14959
Number of positive values for actual - median: 3751/14959
Percentile: 0.1, test pinball: 0.003, percentile count: 0.59556 non-night percentile count: 0.22574
Percentile: 0.5, test pinball: 0.00573, percentile count: 0.74925 non-night percentile count: 0.51659
Percentile: 0.9, test pinball: 0.00349, percentile count: 0.86891 non-night percentile count: 0.73806
Number of positive values for actual - 90th percentile: 1903/7265
Number of positive values for actual - 10th percentile: 5625/7265
Number of positive values for actual - median: 3512/7265
[INFO][2023-07-03 01:33:51] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012110806628264728, pinball_test_loss=0.005731783640055762, pinball_train_10_percentile_loss=0.0013634210146319591, pinball_test_10_percentile_loss=0.003001686500327208, pinball_train_90_percentile_loss=0.0007478248562712943, pinball_test_90_percentile_loss=0.0034894114173973276, mae_train_loss=0.0024221613256529455, mae_test_loss=0.011463567280111523, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01146
2
[13:33:53] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00167, percentile count: 0.55911 non-night percentile count: 0.13507
Percentile: 0.5, train pinball: 0.00126, percentile count: 0.7602 non-night percentile count: 0.52578
Percentile: 0.9, train pinball: 0.00086, percentile count: 0.92953 non-night percentile count: 0.85489
Number of positive values for actual - 90th percentile: 2202/14959
Number of positive values for actual - 10th percentile: 5991/14959
Number of positive values for actual - median: 3873/14959
Percentile: 0.1, test pinball: 0.00427, percentile count: 0.59951 non-night percentile count: 0.23441
Percentile: 0.5, test pinball: 0.00821, percentile count: 0.74109 non-night percentile count: 0.49745
Percentile: 0.9, test pinball: 0.00546, percentile count: 0.8528 non-night percentile count: 0.70406
Number of positive values for actual - 90th percentile: 2150/7265
Number of positive values for actual - 10th percentile: 5562/7265
Number of positive values for actual - median: 3651/7265
[INFO][2023-07-03 01:37:48] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012602450433705187, pinball_test_loss=0.008207599540796725, pinball_train_10_percentile_loss=0.0016688700373852124, pinball_test_10_percentile_loss=0.0042674686642798845, pinball_train_90_percentile_loss=0.0008581383413663751, pinball_test_90_percentile_loss=0.005457999793961636, mae_train_loss=0.0025204900867410373, mae_test_loss=0.01641519908159345, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01642
4
[13:37:51] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00168, percentile count: 0.56134 non-night percentile count: 0.13928
Percentile: 0.5, train pinball: 0.00127, percentile count: 0.75868 non-night percentile count: 0.52182
Percentile: 0.9, train pinball: 0.00086, percentile count: 0.92931 non-night percentile count: 0.85414
Number of positive values for actual - 90th percentile: 2181/14959
Number of positive values for actual - 10th percentile: 5972/14959
Number of positive values for actual - median: 3860/14959
Percentile: 0.1, test pinball: 0.0044, percentile count: 0.60078 non-night percentile count: 0.23799
Percentile: 0.5, test pinball: 0.00839, percentile count: 0.74196 non-night percentile count: 0.49952
Percentile: 0.9, test pinball: 0.00557, percentile count: 0.8542 non-night percentile count: 0.7064
Number of positive values for actual - 90th percentile: 2133/7265
Number of positive values for actual - 10th percentile: 5536/7265
Number of positive values for actual - median: 3636/7265
[INFO][2023-07-03 01:41:24] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012736175343051814, pinball_test_loss=0.008391728546495234, pinball_train_10_percentile_loss=0.0016842752176778816, pinball_test_10_percentile_loss=0.004401864070569362, pinball_train_90_percentile_loss=0.0008551936829977618, pinball_test_90_percentile_loss=0.005572974404012634, mae_train_loss=0.0025472350686103627, mae_test_loss=0.016783457092990468, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01678
8
[13:41:27] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00168, percentile count: 0.56002 non-night percentile count: 0.13657
Percentile: 0.5, train pinball: 0.00127, percentile count: 0.7611 non-night percentile count: 0.52673
Percentile: 0.9, train pinball: 0.00084, percentile count: 0.93129 non-night percentile count: 0.85811
Number of positive values for actual - 90th percentile: 2146/14959
Number of positive values for actual - 10th percentile: 5877/14959
Number of positive values for actual - median: 3840/14959
Percentile: 0.1, test pinball: 0.0045, percentile count: 0.60713 non-night percentile count: 0.24997
Percentile: 0.5, test pinball: 0.0084, percentile count: 0.7433 non-night percentile count: 0.50241
Percentile: 0.9, test pinball: 0.00545, percentile count: 0.85654 non-night percentile count: 0.71108
Number of positive values for actual - 90th percentile: 2099/7265
Number of positive values for actual - 10th percentile: 5449/7265
Number of positive values for actual - median: 3615/7265
[INFO][2023-07-03 01:45:18] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012674934396124339, pinball_test_loss=0.008396571267757101, pinball_train_10_percentile_loss=0.0016788063004774527, pinball_test_10_percentile_loss=0.004495701843400433, pinball_train_90_percentile_loss=0.0008435518045509662, pinball_test_90_percentile_loss=0.005448363649214408, mae_train_loss=0.0025349868792248677, mae_test_loss=0.016793142535514203, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median test MAE: 0.01679
12
[13:45:20] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00169, percentile count: 0.55775 non-night percentile count: 0.13138
Percentile: 0.5, train pinball: 0.00128, percentile count: 0.75993 non-night percentile count: 0.52436
Percentile: 0.9, train pinball: 0.00086, percentile count: 0.93019 non-night percentile count: 0.85596
Number of positive values for actual - 90th percentile: 2036/14959
Number of positive values for actual - 10th percentile: 5862/14959
Number of positive values for actual - median: 3725/14959
Percentile: 0.1, test pinball: 0.00451, percentile count: 0.60813 non-night percentile count: 0.25138
Percentile: 0.5, test pinball: 0.00839, percentile count: 0.75099 non-night percentile count: 0.5186
Percentile: 0.9, test pinball: 0.00522, percentile count: 0.86389 non-night percentile count: 0.72521
Number of positive values for actual - 90th percentile: 1995/7260
Number of positive values for actual - 10th percentile: 5435/7260
Number of positive values for actual - median: 3495/7260
Median test MAE: 0.01678
24
[INFO][2023-07-03 01:48:53] : Trained model for 12 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012839625273683732, pinball_test_loss=0.008391585003216975, pinball_train_10_percentile_loss=0.0016854178215470442, pinball_test_10_percentile_loss=0.004509724982652325, pinball_train_90_percentile_loss=0.0008568094229523167, pinball_test_90_percentile_loss=0.005221557915448461, mae_train_loss=0.0025679250547367464, mae_test_loss=0.01678317000643395, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[13:48:56] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00172, percentile count: 0.55901 non-night percentile count: 0.13449
Percentile: 0.5, train pinball: 0.00133, percentile count: 0.76038 non-night percentile count: 0.52582
Percentile: 0.9, train pinball: 0.00088, percentile count: 0.92999 non-night percentile count: 0.85565
Number of positive values for actual - 90th percentile: 1992/14959
Number of positive values for actual - 10th percentile: 5764/14959
Number of positive values for actual - median: 3607/14959
Percentile: 0.1, test pinball: 0.00467, percentile count: 0.61468 non-night percentile count: 0.26369
Percentile: 0.5, test pinball: 0.00853, percentile count: 0.75887 non-night percentile count: 0.53482
Percentile: 0.9, test pinball: 0.00504, percentile count: 0.86684 non-night percentile count: 0.73479
Number of positive values for actual - 90th percentile: 1927/7266
Number of positive values for actual - 10th percentile: 5350/7266
Number of positive values for actual - median: 3380/7266
Median test MAE: 0.01706
36
[INFO][2023-07-03 01:52:32] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0013338061040532553, pinball_test_loss=0.008530901830825225, pinball_train_10_percentile_loss=0.0017155930575816249, pinball_test_10_percentile_loss=0.004668257820471507, pinball_train_90_percentile_loss=0.000883639364811557, pinball_test_90_percentile_loss=0.005037923392537432, mae_train_loss=0.0026676122081065105, mae_test_loss=0.01706180366165045, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
[13:52:35] WARNING: /home/jacob/Development/xgboost/src/learner.cc:753: 
Parameters: { "scale_pos_weight" } are not used.

Percentile: 0.1, train pinball: 0.00181, percentile count: 0.56006 non-night percentile count: 0.13668
Percentile: 0.5, train pinball: 0.00133, percentile count: 0.76093 non-night percentile count: 0.5274
Percentile: 0.9, train pinball: 0.00089, percentile count: 0.92834 non-night percentile count: 0.85261
Number of positive values for actual - 90th percentile: 1930/14959
Number of positive values for actual - 10th percentile: 5670/14959
Number of positive values for actual - median: 3470/14959
Percentile: 0.1, test pinball: 0.00481, percentile count: 0.62096 non-night percentile count: 0.27723
Percentile: 0.5, test pinball: 0.00908, percentile count: 0.76803 non-night percentile count: 0.55282
Percentile: 0.9, test pinball: 0.00534, percentile count: 0.87098 non-night percentile count: 0.74191
Number of positive values for actual - 90th percentile: 1874/7261
Number of positive values for actual - 10th percentile: 5248/7261
Number of positive values for actual - median: 3247/7261
Median test MAE: 0.01817
[INFO][2023-07-03 01:56:30] : Trained model for 36 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0013279032855523414, pinball_test_loss=0.00908407317006935, pinball_train_10_percentile_loss=0.0018096007974641357, pinball_test_10_percentile_loss=0.004814743433827803, pinball_train_90_percentile_loss=0.0008944979497848297, pinball_test_90_percentile_loss=0.005340563678581044, mae_train_loss=0.0026558065711046828, mae_test_loss=0.0181681463401387, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

jacobbieker commented 1 year ago

Cross validation, testing on 1 year, and training on the rest, for 2016-2022. There doesn't seem to be any real difference on the cross validation, if anything, testing on 2021 might be slightly more optimistic compared to other years.

0

[INFO][2023-07-03 07:59:16] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010550639840426152, pinball_test_loss=0.0037893298269305238, pinball_train_10_percentile_loss=0.001077463850161541, pinball_test_10_percentile_loss=0.0022186065315161377, pinball_train_90_percentile_loss=0.0005756842648535435, pinball_test_90_percentile_loss=0.0023311790724128223, mae_train_loss=0.0021101279680852303, mae_test_loss=0.0075786596538610475, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median MAE: 0.00758
Mean MAE: 0.00774
Std MAE: 0.00051
Mean Pinball Median: 0.49991
Std Pinball Median: 0.02385
Mean Pinball 10th percentile: 0.21619
Std Pinball 10th percentile: 0.01451
Mean Pinball 90th percentile: 0.72409
Std Pinball 90th percentile: 0.0279
Pinball Medians: [0.48077505298213746, 0.4874340286504147, 0.5429756220155818, 0.485530140275496, 0.4712945590994371, 0.5220264317180616, 0.5093668040464594]
Pinball 10th percentiles: [0.22176808961550107, 0.2119879366675044, 0.24918321186227696, 0.20510552255781625, 0.20537836147592245, 0.2131057268722467, 0.2068190333458224]
Pinball 90th percentiles: [0.6874053890402664, 0.7046996732847449, 0.7552148781100779, 0.7195753822823202, 0.6959349593495935, 0.7648678414096917, 0.7409142000749345]
Pinball Ranges: [0.4656373  0.49271174 0.50603167 0.51446986 0.4905566  0.55176211
 0.53409517]
1

Median MAE: 0.0121
Mean MAE: 0.01219
Std MAE: 0.00094
Mean Pinball Median: 0.49761
Std Pinball Median: 0.03574
Mean Pinball 10th percentile: 0.24395
Std Pinball 10th percentile: 0.03457
Mean Pinball 90th percentile: 0.69554
Std Pinball 90th percentile: 0.03127
Pinball Medians: [0.5205935796486978, 0.528273435536567, 0.5364413169137975, 0.4578648136449779, 0.43477173233270794, 0.5165863730213351, 0.48875983514424876]
Pinball 10th percentiles: [0.29421562689279224, 0.2764513696908771, 0.27783362653933147, 0.2169298799747315, 0.2046278924327705, 0.22573984858912594, 0.21187710753091046]
Pinball 90th percentiles: [0.6883706844336765, 0.713998492083438, 0.7158833877858759, 0.6732785849652558, 0.6350218886804253, 0.7380591878871301, 0.7041963282128137]
Pinball Ranges: [0.39415506 0.43754712 0.43804976 0.4563487  0.430394   0.51231934
 0.49231922]
2
[INFO][2023-07-03 08:22:24] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012045550234201239, pinball_test_loss=0.005886998386154272, pinball_train_10_percentile_loss=0.0013648913485722958, pinball_test_10_percentile_loss=0.0032549798597562565, pinball_train_90_percentile_loss=0.0007372956891999191, pinball_test_90_percentile_loss=0.003809915106989967, mae_train_loss=0.0024091100468402477, mae_test_loss=0.011773996772308544, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01724
Mean MAE: 0.01742
Std MAE: 0.00147
Mean Pinball Median: 0.488
Std Pinball Median: 0.07414
Mean Pinball 10th percentile: 0.2563
Std Pinball 10th percentile: 0.06107
Mean Pinball 90th percentile: 0.66742
Std Pinball 90th percentile: 0.05387
Pinball Medians: [0.5631626779763708, 0.5996481528022116, 0.5118120130686102, 0.42238221548566374, 0.36910569105691055, 0.4974535443909153, 0.4524166354439865]
Pinball 10th percentiles: [0.3352014541048167, 0.34757476752953004, 0.2824830359386781, 0.20272830617658205, 0.1856160100062539, 0.23441156228492774, 0.20606968902210565]
Pinball 90th percentiles: [0.7043320205998183, 0.740512691631063, 0.6725307866298065, 0.623468485537451, 0.5661038148843027, 0.7040605643496215, 0.6609216935181716]
Pinball Ranges: [0.36913057 0.39293792 0.39004775 0.42074018 0.3804878  0.469649
 0.454852  ]
4
[INFO][2023-07-03 08:45:54] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012566411983616936, pinball_test_loss=0.008501757542043954, pinball_train_10_percentile_loss=0.0016690717023773577, pinball_test_10_percentile_loss=0.004541239745485791, pinball_train_90_percentile_loss=0.0008389615227801574, pinball_test_90_percentile_loss=0.006025412010102769, mae_train_loss=0.002513282396723387, mae_test_loss=0.017003515084087907, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01735
Mean MAE: 0.01752
Std MAE: 0.00139
Mean Pinball Median: 0.48897
Std Pinball Median: 0.06672
Mean Pinball 10th percentile: 0.25631
Std Pinball 10th percentile: 0.05519
Mean Pinball 90th percentile: 0.66709
Std Pinball 90th percentile: 0.04684
Pinball Medians: [0.5488710410668283, 0.590726313144006, 0.5126916310630811, 0.4311324327736397, 0.3771106941838649, 0.499518238128011, 0.4627201198950918]
Pinball 10th percentiles: [0.319745415972117, 0.33990952500628296, 0.28914300075395827, 0.20792829188233808, 0.19111944965603503, 0.23799036476256022, 0.2083177219932559]
Pinball 90th percentiles: [0.6872253371722988, 0.7338527268157828, 0.6691379743654184, 0.636535790935488, 0.5779862414008755, 0.7064005505849965, 0.6584863244660921]
Pinball Ranges: [0.36747992 0.3939432  0.37999497 0.4286075  0.38686679 0.46841019
 0.4501686 ]
8
[INFO][2023-07-03 09:09:24] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012600357333270877, pinball_test_loss=0.008674636532452047, pinball_train_10_percentile_loss=0.0016875834758432317, pinball_test_10_percentile_loss=0.00461961522857021, pinball_train_90_percentile_loss=0.0008385207358480372, pinball_test_90_percentile_loss=0.006020705398326387, mae_train_loss=0.0025200714666541754, mae_test_loss=0.017349273064904094, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01715
Mean MAE: 0.01739
Std MAE: 0.00099
Mean Pinball Median: 0.48715
Std Pinball Median: 0.05295
Mean Pinball 10th percentile: 0.25444
Std Pinball 10th percentile: 0.04409
Mean Pinball 90th percentile: 0.66836
Std Pinball 90th percentile: 0.03998
Pinball Medians: [0.505379602970147, 0.569364161849711, 0.517089721035436, 0.434036106552203, 0.3944965603502189, 0.5024088093599449, 0.4872611464968153]
Pinball 10th percentiles: [0.2876193362630702, 0.32445338024629305, 0.2896456396079417, 0.21461936624163616, 0.1944965603502189, 0.24996558843771507, 0.22030723117272386]
Pinball 90th percentiles: [0.6640400060615245, 0.7122392560944961, 0.6823322442824831, 0.6374195177376594, 0.5902439024390244, 0.7110805230557468, 0.6811539902585237]
Pinball Ranges: [0.37642067 0.38778588 0.3926866  0.42280015 0.39574734 0.46111493
 0.46084676]
12
[INFO][2023-07-03 09:32:59] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012586437719674258, pinball_test_loss=0.008577267065011888, pinball_train_10_percentile_loss=0.0016860567623376726, pinball_test_10_percentile_loss=0.004750062878693247, pinball_train_90_percentile_loss=0.0008281783249143637, pinball_test_90_percentile_loss=0.005655593532143917, mae_train_loss=0.0025172875439348516, mae_test_loss=0.017154534130023776, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

[INFO][2023-07-03 09:56:26] : Trained model for 12 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012774249067674674, pinball_test_loss=0.00845450839859749, pinball_train_10_percentile_loss=0.0016880324521668347, pinball_test_10_percentile_loss=0.004728071141670281, pinball_train_90_percentile_loss=0.0008396215046314899, pinball_test_90_percentile_loss=0.0054995647254289706, mae_train_loss=0.002554849813534935, mae_test_loss=0.01690901679719498, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
Median MAE: 0.01691
Mean MAE: 0.01746
Std MAE: 0.00109
Mean Pinball Median: 0.4895
Std Pinball Median: 0.04458
Mean Pinball 10th percentile: 0.25294
Std Pinball 10th percentile: 0.03501
Mean Pinball 90th percentile: 0.66953
Std Pinball 90th percentile: 0.03755
Pinball Medians: [0.47940642035130226, 0.5510050251256281, 0.5283990952500628, 0.44933030073287844, 0.4117058529264632, 0.518595041322314, 0.4880930058128633]
Pinball 10th percentiles: [0.2628709872804361, 0.30967336683417085, 0.2901482784619251, 0.23313115996967398, 0.2054777388694347, 0.2513774104683196, 0.2178886180386274]
Pinball 90th percentiles: [0.6364324651726226, 0.6938442211055277, 0.6871073133953255, 0.6474601971190296, 0.6071785892946473, 0.7252066115702479, 0.6894805925370335]
Pinball Ranges: [0.37356148 0.38417085 0.39695903 0.41432904 0.40170085 0.4738292
 0.47159197]
24

Median MAE: 0.01768
Mean MAE: 0.01808
Std MAE: 0.00155
Mean Pinball Median: 0.4914
Std Pinball Median: 0.05257
Mean Pinball 10th percentile: 0.25764
Std Pinball 10th percentile: 0.03301
Mean Pinball 90th percentile: 0.67382
Std Pinball 90th percentile: 0.05624
Pinball Medians: [0.40888417222559126, 0.5229300163337103, 0.5456259426847662, 0.4759074238016947, 0.4210855427713857, 0.5348197082301128, 0.5305164319248826]
Pinball 10th percentiles: [0.22377198302001214, 0.3028018595300917, 0.30165912518853694, 0.24939926647274566, 0.2088544272136068, 0.2636939168731076, 0.25333333333333335]
Pinball 90th percentiles: [0.5708004851425106, 0.6802362105792185, 0.705379587732529, 0.6699127355507778, 0.6198099049524762, 0.7347921827690613, 0.7357746478873239]
Pinball Ranges: [0.3470285  0.37743435 0.40372046 0.42051347 0.41095548 0.47109827
 0.48244131]
36
[INFO][2023-07-03 10:19:58] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0013182253728295837, pinball_test_loss=0.008839582194506672, pinball_train_10_percentile_loss=0.0017034921036401117, pinball_test_10_percentile_loss=0.0050669811571025115, pinball_train_90_percentile_loss=0.0008656499083700535, pinball_test_90_percentile_loss=0.005267883046415519, mae_train_loss=0.0026364507456591675, mae_test_loss=0.017679164389013343, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01906
Mean MAE: 0.01927
Std MAE: 0.00153
Mean Pinball Median: 0.49714
Std Pinball Median: 0.05919
Mean Pinball 10th percentile: 0.26343
Std Pinball 10th percentile: 0.0324
Mean Pinball 90th percentile: 0.67256
Std Pinball 90th percentile: 0.06084
Pinball Medians: [0.40279041552926903, 0.5341708542713568, 0.5307199396909159, 0.4893213698976368, 0.41553082405902214, 0.5528164164715604, 0.5546155292348186]
Pinball 10th percentiles: [0.22596299666363362, 0.30263819095477384, 0.3004146249528835, 0.26260583849361807, 0.21007877954232837, 0.2772345406968737, 0.2650874224478285]
Pinball 90th percentiles: [0.5573248407643312, 0.6804020100502512, 0.7008418142982786, 0.6786301023631998, 0.615605852194573, 0.741908827985126, 0.7332205301748449]
Pinball Ranges: [0.33136184 0.37776382 0.40042719 0.41602426 0.40552707 0.46467429
 0.46813311]

dantravers commented 1 year ago

Thanks Jacob – that’s great to see that investigated.

From: Jacob Bieker @.> Sent: Tuesday, July 4, 2023 9:41 AM To: openclimatefix/uk-pv-national-xg @.> Cc: dantravers @.>; Comment @.> Subject: Re: [openclimatefix/uk-pv-national-xg] probabilistic ML (Issue #66)

[INFO][2023-07-03 07:59:16] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010550639840426152, pinball_test_loss=0.0037893298269305238, pinball_train_10_percentile_loss=0.001077463850161541, pinball_test_10_percentile_loss=0.0022186065315161377, pinball_train_90_percentile_loss=0.0005756842648535435, pinball_test_90_percentile_loss=0.0023311790724128223, mae_train_loss=0.0021101279680852303, mae_test_loss=0.0075786596538610475, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...)) Median MAE: 0.00758 Mean MAE: 0.00774 Std MAE: 0.00051 Mean Pinball Median: 0.49991 Std Pinball Median: 0.02385 Mean Pinball 10th percentile: 0.21619 Std Pinball 10th percentile: 0.01451 Mean Pinball 90th percentile: 0.72409 Std Pinball 90th percentile: 0.0279 Pinball Medians: [0.48077505298213746, 0.4874340286504147, 0.5429756220155818, 0.485530140275496, 0.4712945590994371, 0.5220264317180616, 0.5093668040464594] Pinball 10th percentiles: [0.22176808961550107, 0.2119879366675044, 0.24918321186227696, 0.20510552255781625, 0.20537836147592245, 0.2131057268722467, 0.2068190333458224] Pinball 90th percentiles: [0.6874053890402664, 0.7046996732847449, 0.7552148781100779, 0.7195753822823202, 0.6959349593495935, 0.7648678414096917, 0.7409142000749345] Pinball Ranges: [0.4656373 0.49271174 0.50603167 0.51446986 0.4905566 0.55176211 0.53409517] 1

Median MAE: 0.0121 Mean MAE: 0.01219 Std MAE: 0.00094 Mean Pinball Median: 0.49761 Std Pinball Median: 0.03574 Mean Pinball 10th percentile: 0.24395 Std Pinball 10th percentile: 0.03457 Mean Pinball 90th percentile: 0.69554 Std Pinball 90th percentile: 0.03127 Pinball Medians: [0.5205935796486978, 0.528273435536567, 0.5364413169137975, 0.4578648136449779, 0.43477173233270794, 0.5165863730213351, 0.48875983514424876] Pinball 10th percentiles: [0.29421562689279224, 0.2764513696908771, 0.27783362653933147, 0.2169298799747315, 0.2046278924327705, 0.22573984858912594, 0.21187710753091046] Pinball 90th percentiles: [0.6883706844336765, 0.713998492083438, 0.7158833877858759, 0.6732785849652558, 0.6350218886804253, 0.7380591878871301, 0.7041963282128137] Pinball Ranges: [0.39415506 0.43754712 0.43804976 0.4563487 0.430394 0.51231934 0.49231922] 2 [INFO][2023-07-03 08:22:24] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012045550234201239, pinball_test_loss=0.005886998386154272, pinball_train_10_percentile_loss=0.0013648913485722958, pinball_test_10_percentile_loss=0.0032549798597562565, pinball_train_90_percentile_loss=0.0007372956891999191, pinball_test_90_percentile_loss=0.003809915106989967, mae_train_loss=0.0024091100468402477, mae_test_loss=0.011773996772308544, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01724 Mean MAE: 0.01742 Std MAE: 0.00147 Mean Pinball Median: 0.488 Std Pinball Median: 0.07414 Mean Pinball 10th percentile: 0.2563 Std Pinball 10th percentile: 0.06107 Mean Pinball 90th percentile: 0.66742 Std Pinball 90th percentile: 0.05387 Pinball Medians: [0.5631626779763708, 0.5996481528022116, 0.5118120130686102, 0.42238221548566374, 0.36910569105691055, 0.4974535443909153, 0.4524166354439865] Pinball 10th percentiles: [0.3352014541048167, 0.34757476752953004, 0.2824830359386781, 0.20272830617658205, 0.1856160100062539, 0.23441156228492774, 0.20606968902210565] Pinball 90th percentiles: [0.7043320205998183, 0.740512691631063, 0.6725307866298065, 0.623468485537451, 0.5661038148843027, 0.7040605643496215, 0.6609216935181716] Pinball Ranges: [0.36913057 0.39293792 0.39004775 0.42074018 0.3804878 0.469649 0.454852 ] 4 [INFO][2023-07-03 08:45:54] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012566411983616936, pinball_test_loss=0.008501757542043954, pinball_train_10_percentile_loss=0.0016690717023773577, pinball_test_10_percentile_loss=0.004541239745485791, pinball_train_90_percentile_loss=0.0008389615227801574, pinball_test_90_percentile_loss=0.006025412010102769, mae_train_loss=0.002513282396723387, mae_test_loss=0.017003515084087907, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01735 Mean MAE: 0.01752 Std MAE: 0.00139 Mean Pinball Median: 0.48897 Std Pinball Median: 0.06672 Mean Pinball 10th percentile: 0.25631 Std Pinball 10th percentile: 0.05519 Mean Pinball 90th percentile: 0.66709 Std Pinball 90th percentile: 0.04684 Pinball Medians: [0.5488710410668283, 0.590726313144006, 0.5126916310630811, 0.4311324327736397, 0.3771106941838649, 0.499518238128011, 0.4627201198950918] Pinball 10th percentiles: [0.319745415972117, 0.33990952500628296, 0.28914300075395827, 0.20792829188233808, 0.19111944965603503, 0.23799036476256022, 0.2083177219932559] Pinball 90th percentiles: [0.6872253371722988, 0.7338527268157828, 0.6691379743654184, 0.636535790935488, 0.5779862414008755, 0.7064005505849965, 0.6584863244660921] Pinball Ranges: [0.36747992 0.3939432 0.37999497 0.4286075 0.38686679 0.46841019 0.4501686 ] 8 [INFO][2023-07-03 09:09:24] : Trained model for 4 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012600357333270877, pinball_test_loss=0.008674636532452047, pinball_train_10_percentile_loss=0.0016875834758432317, pinball_test_10_percentile_loss=0.00461961522857021, pinball_train_90_percentile_loss=0.0008385207358480372, pinball_test_90_percentile_loss=0.006020705398326387, mae_train_loss=0.0025200714666541754, mae_test_loss=0.017349273064904094, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01715 Mean MAE: 0.01739 Std MAE: 0.00099 Mean Pinball Median: 0.48715 Std Pinball Median: 0.05295 Mean Pinball 10th percentile: 0.25444 Std Pinball 10th percentile: 0.04409 Mean Pinball 90th percentile: 0.66836 Std Pinball 90th percentile: 0.03998 Pinball Medians: [0.505379602970147, 0.569364161849711, 0.517089721035436, 0.434036106552203, 0.3944965603502189, 0.5024088093599449, 0.4872611464968153] Pinball 10th percentiles: [0.2876193362630702, 0.32445338024629305, 0.2896456396079417, 0.21461936624163616, 0.1944965603502189, 0.24996558843771507, 0.22030723117272386] Pinball 90th percentiles: [0.6640400060615245, 0.7122392560944961, 0.6823322442824831, 0.6374195177376594, 0.5902439024390244, 0.7110805230557468, 0.6811539902585237] Pinball Ranges: [0.37642067 0.38778588 0.3926866 0.42280015 0.39574734 0.46111493 0.46084676] 12 [INFO][2023-07-03 09:32:59] : Trained model for 8 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012586437719674258, pinball_test_loss=0.008577267065011888, pinball_train_10_percentile_loss=0.0016860567623376726, pinball_test_10_percentile_loss=0.004750062878693247, pinball_train_90_percentile_loss=0.0008281783249143637, pinball_test_90_percentile_loss=0.005655593532143917, mae_train_loss=0.0025172875439348516, mae_test_loss=0.017154534130023776, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

[INFO][2023-07-03 09:56:26] : Trained model for 12 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012774249067674674, pinball_test_loss=0.00845450839859749, pinball_train_10_percentile_loss=0.0016880324521668347, pinball_test_10_percentile_loss=0.004728071141670281, pinball_train_90_percentile_loss=0.0008396215046314899, pinball_test_90_percentile_loss=0.0054995647254289706, mae_train_loss=0.002554849813534935, mae_test_loss=0.01690901679719498, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...)) Median MAE: 0.01691 Mean MAE: 0.01746 Std MAE: 0.00109 Mean Pinball Median: 0.4895 Std Pinball Median: 0.04458 Mean Pinball 10th percentile: 0.25294 Std Pinball 10th percentile: 0.03501 Mean Pinball 90th percentile: 0.66953 Std Pinball 90th percentile: 0.03755 Pinball Medians: [0.47940642035130226, 0.5510050251256281, 0.5283990952500628, 0.44933030073287844, 0.4117058529264632, 0.518595041322314, 0.4880930058128633] Pinball 10th percentiles: [0.2628709872804361, 0.30967336683417085, 0.2901482784619251, 0.23313115996967398, 0.2054777388694347, 0.2513774104683196, 0.2178886180386274] Pinball 90th percentiles: [0.6364324651726226, 0.6938442211055277, 0.6871073133953255, 0.6474601971190296, 0.6071785892946473, 0.7252066115702479, 0.6894805925370335] Pinball Ranges: [0.37356148 0.38417085 0.39695903 0.41432904 0.40170085 0.4738292 0.47159197] 24

Median MAE: 0.01768 Mean MAE: 0.01808 Std MAE: 0.00155 Mean Pinball Median: 0.4914 Std Pinball Median: 0.05257 Mean Pinball 10th percentile: 0.25764 Std Pinball 10th percentile: 0.03301 Mean Pinball 90th percentile: 0.67382 Std Pinball 90th percentile: 0.05624 Pinball Medians: [0.40888417222559126, 0.5229300163337103, 0.5456259426847662, 0.4759074238016947, 0.4210855427713857, 0.5348197082301128, 0.5305164319248826] Pinball 10th percentiles: [0.22377198302001214, 0.3028018595300917, 0.30165912518853694, 0.24939926647274566, 0.2088544272136068, 0.2636939168731076, 0.25333333333333335] Pinball 90th percentiles: [0.5708004851425106, 0.6802362105792185, 0.705379587732529, 0.6699127355507778, 0.6198099049524762, 0.7347921827690613, 0.7357746478873239] Pinball Ranges: [0.3470285 0.37743435 0.40372046 0.42051347 0.41095548 0.47109827 0.48244131] 36 [INFO][2023-07-03 10:19:58] : Trained model for 24 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0013182253728295837, pinball_test_loss=0.008839582194506672, pinball_train_10_percentile_loss=0.0017034921036401117, pinball_test_10_percentile_loss=0.0050669811571025115, pinball_train_90_percentile_loss=0.0008656499083700535, pinball_test_90_percentile_loss=0.005267883046415519, mae_train_loss=0.0026364507456591675, mae_test_loss=0.017679164389013343, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85, early_stopping_rounds=None, enable_categorical=False, eval_metric=None, feature_types=None, gamma=0, gpu_id=-1, grow_policy='depthwise', importance_type=None, interaction_constraints='', learning_rate=0.005, max_bin=256, max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None, max_depth=80, max_leaves=0, min_child_weight=5, missing=nan, monotone_constraints=None, multi_strategy=None, n_estimators=1250, n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

Median MAE: 0.01906 Mean MAE: 0.01927 Std MAE: 0.00153 Mean Pinball Median: 0.49714 Std Pinball Median: 0.05919 Mean Pinball 10th percentile: 0.26343 Std Pinball 10th percentile: 0.0324 Mean Pinball 90th percentile: 0.67256 Std Pinball 90th percentile: 0.06084 Pinball Medians: [0.40279041552926903, 0.5341708542713568, 0.5307199396909159, 0.4893213698976368, 0.41553082405902214, 0.5528164164715604, 0.5546155292348186] Pinball 10th percentiles: [0.22596299666363362, 0.30263819095477384, 0.3004146249528835, 0.26260583849361807, 0.21007877954232837, 0.2772345406968737, 0.2650874224478285] Pinball 90th percentiles: [0.5573248407643312, 0.6804020100502512, 0.7008418142982786, 0.6786301023631998, 0.615605852194573, 0.741908827985126, 0.7332205301748449] Pinball Ranges: [0.33136184 0.37776382 0.40042719 0.41602426 0.40552707 0.46467429 0.46813311]

— Reply to this email directly, view it on GitHub https://github.com/openclimatefix/uk-pv-national-xg/issues/66#issuecomment-1619804699 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AJGGYO4CZFCECBOC347MUIDXOPJI5ANCNFSM6AAAAAAXCGA3V4 . You are receiving this because you commented.Message ID: @.***>

jacobbieker commented 1 year ago

This run is from the same cross validation, but only testing on hours 10-14 of the day, trained on all of them, it doesn't seem to be dramatically different than for all times, in some cases like 1-2% better. This has fewer forecast times trained here, as something went wrong with the plotting, but unrelated to the numbers presented here.

0

---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
Median MAE: 0.02079
Mean MAE: 0.0212
Std MAE: 0.00134
Mean Pinball Median: 0.50105
Std Pinball Median: 0.02301
Mean Pinball 10th percentile: 0.19109
Std Pinball 10th percentile: 0.02105
Mean Pinball 90th percentile: 0.7037
Std Pinball 90th percentile: 0.02193
Pinball Medians: [0.5028288543140028, 0.5391591096455071, 0.5206270627062707, 0.47799442896935934, 0.46451082488352974, 0.5011221545367105, 0.5011317338162065]
Pinball 10th percentiles: [0.21534653465346534, 0.21434460016488047, 0.21424642464246424, 0.181058495821727, 0.17785694710879693, 0.16319333119589613, 0.17157084653689453]
Pinball 90th percentiles: [0.6806930693069307, 0.72959604286892, 0.7098459845984598, 0.6974930362116991, 0.6725130172650041, 0.7367746072459121, 0.6989588048890901]
Pinball Ranges: [0.46534653 0.51525144 0.49559956 0.51643454 0.49465607 0.57358128
 0.52738796]
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
1
[INFO][2023-07-04 10:25:49] : Trained model for 0 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0010550639840426152, pinball_test_loss=0.010327169586862257, pinball_train_10_percentile_loss=0.001077463850161541, pinball_test_10_percentile_loss=0.005952934916610267, pinball_train_90_percentile_loss=0.0005756842648535435, pinball_test_90_percentile_loss=0.006716461909136405, mae_train_loss=0.0021101279680852303, mae_test_loss=0.020654339173724515, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

[INFO][2023-07-04 10:51:06] : Trained model for 1 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012045550234201239, pinball_test_loss=0.015830112661725588, pinball_train_10_percentile_loss=0.0013648913485722958, pinball_test_10_percentile_loss=0.008488996478511836, pinball_train_90_percentile_loss=0.0007372956891999191, pinball_test_90_percentile_loss=0.010842991173873466, mae_train_loss=0.0024091100468402477, mae_test_loss=0.031660225323451176, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
Median MAE: 0.03172
Mean MAE: 0.03242
Std MAE: 0.00258
Mean Pinball Median: 0.49556
Std Pinball Median: 0.03555
Mean Pinball 10th percentile: 0.22446
Std Pinball 10th percentile: 0.03157
Mean Pinball 90th percentile: 0.66689
Std Pinball 90th percentile: 0.03185
Pinball Medians: [0.4867068415455512, 0.5458817025981205, 0.5370936371214227, 0.46143057503506313, 0.4410951327433628, 0.5118741976893453, 0.48482102401449934]
Pinball 10th percentiles: [0.2626728110599078, 0.2542841348811498, 0.26451792164490134, 0.20420757363253858, 0.18998893805309736, 0.19801026957638, 0.19755323969188945]
Pinball 90th percentiles: [0.6370081531371854, 0.7075732448866777, 0.6985273687135315, 0.6549789621318373, 0.6183628318584071, 0.6970474967907574, 0.6547349342999547]
Pinball Ranges: [0.37433534 0.45328911 0.43400945 0.45077139 0.42837389 0.49903723
 0.45718169]
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
2
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
Median MAE: 0.04154
Mean MAE: 0.04217
Std MAE: 0.00375
Mean Pinball Median: 0.47035
Std Pinball Median: 0.05599
Mean Pinball 10th percentile: 0.23208
Std Pinball 10th percentile: 0.04055
Mean Pinball 90th percentile: 0.62246
Std Pinball 90th percentile: 0.04304
Pinball Medians: [0.4830508474576271, 0.558119411595039, 0.5146759662888695, 0.4194117647058824, 0.3799709724238026, 0.49458483754512633, 0.44264569842738205]
Pinball 10th percentiles: [0.26676492262343404, 0.29074127487741563, 0.27201394943330426, 0.2011764705882353, 0.1776487663280116, 0.21824745651460453, 0.19796484736355227]
Pinball 90th percentiles: [0.6090641120117907, 0.6867608883761177, 0.6512641673931997, 0.5941176470588235, 0.5492017416545718, 0.6599934361667213, 0.606845513413506]
Pinball Ranges: [0.34229919 0.39601961 0.37925022 0.39294118 0.37155298 0.44174598
 0.40888067]

---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
Median MAE: 0.04154
Mean MAE: 0.04217
Std MAE: 0.00375
Mean Pinball Median: 0.47035
Std Pinball Median: 0.05599
Mean Pinball 10th percentile: 0.23208
Std Pinball 10th percentile: 0.04055
Mean Pinball 90th percentile: 0.62246
Std Pinball 90th percentile: 0.04304
Pinball Medians: [0.4830508474576271, 0.558119411595039, 0.5146759662888695, 0.4194117647058824, 0.3799709724238026, 0.49458483754512633, 0.44264569842738205]
Pinball 10th percentiles: [0.26676492262343404, 0.29074127487741563, 0.27201394943330426, 0.2011764705882353, 0.1776487663280116, 0.21824745651460453, 0.19796484736355227]
Pinball 90th percentiles: [0.6090641120117907, 0.6867608883761177, 0.6512641673931997, 0.5941176470588235, 0.5492017416545718, 0.6599934361667213, 0.606845513413506]
Pinball Ranges: [0.34229919 0.39601961 0.37925022 0.39294118 0.37155298 0.44174598
 0.40888067]
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
4
[INFO][2023-07-04 11:15:28] : Trained model for 2 hour forecast. training_results=ExperimentSummary(pinball_train_loss=0.0012566411983616936, pinball_test_loss=0.020771658745204435, pinball_train_10_percentile_loss=0.0016690717023773577, pinball_test_10_percentile_loss=0.010905167658555849, pinball_train_90_percentile_loss=0.0008389615227801574, pinball_test_90_percentile_loss=0.015555463848280209, mae_train_loss=0.002513282396723387, mae_test_loss=0.04154331749040887, model=XGBRegressor(base_score=None, booster='gbtree', callbacks=None,
             colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.85,
             early_stopping_rounds=None, enable_categorical=False,
             eval_metric=None, feature_types=None, gamma=0, gpu_id=-1,
             grow_policy='depthwise', importance_type=None,
             interaction_constraints='', learning_rate=0.005, max_bin=256,
             max_cat_threshold=64, max_cat_to_onehot=None, max_delta_step=None,
             max_depth=80, max_leaves=0, min_child_weight=5, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1250,
             n_jobs=-1, num_parallel_tree=1, objective='reg:quantileerror', ...))

---------ONLY DONE DURING 10-14 TIMEPERIOD-----------
Median MAE: 0.02487
Mean MAE: 0.02532
Std MAE: 0.00298
Mean Pinball Median: 0.48687
Std Pinball Median: 0.05249
Mean Pinball 10th percentile: 0.25406
Std Pinball 10th percentile: 0.03661
Mean Pinball 90th percentile: 0.68071
Std Pinball 90th percentile: 0.06582
Pinball Medians: [0.4119672836848902, 0.4487270155586987, 0.569247546346783, 0.49451754385964913, 0.4475879579252811, 0.548775748153906, 0.4872746553552492]
Pinball 10th percentiles: [0.2152389151958674, 0.24186704384724186, 0.3351508542348237, 0.24890350877192982, 0.24047878128400435, 0.2693354061406918, 0.22746553552492046]
Pinball 90th percentiles: [0.5630650021523892, 0.6212871287128713, 0.7560886950199928, 0.7149122807017544, 0.661588683351469, 0.7555382821609017, 0.6924708377518558]
Pinball Ranges: [0.34782609 0.37942008 0.42093784 0.46600877 0.4211099  0.48620288
 0.4650053 ]
---------ONLY DONE DURING 10-14 TIMEPERIOD-----------

dantravers commented 1 year ago

Thanks @jacobbieker - sounds like we still have the issue then. Separate question - if I remember correctly the test results didn't particularly differ across different forecast horizons. Is that still the case? I.e. comparing the range covered for a 2 hour, 8 hour, 24 hour forecast horizon?

jacobbieker commented 1 year ago

Yep, these results before are for different horizons, and they are all essentially the same, up to a 10% difference in range (i.e. covers 55% instead of 45%), but not larger than that. Most are fairly similar ranges across the horizons, and none come close to what the quantiles should cover.

jacobbieker commented 1 year ago

I've now pushed a table of results to the experiments folder, its reproduced here of the mean percentile from cross validation for every forecast horizon.

Horizon	0.1	0.5	0.9
0	0.21619	0.49991	0.72409
1	0.24395	0.49761	0.69554
2	0.2563	0.488	0.66742
3	0.25588	0.49109	0.66645
4	0.25631	0.48897	0.66709
5	0.25536	0.49024	0.66989
6	0.25666	0.49079	0.669
7	0.25537	0.48862	0.66705
8	0.25444	0.48715	0.66836
9	0.25524	0.48759	0.66886
10	0.25443	0.48719	0.66661
11	0.25302	0.48774	0.66849
12	0.25294	0.4895	0.66953
13	0.25381	0.48953	0.66902
14	0.25379	0.48751	0.67067
15	0.25463	0.49091	0.66932
16	0.25517	0.48787	0.66843
17	0.25394	0.48815	0.66976
18	0.25636	0.48961	0.67008
19	0.25522	0.48977	0.66885
20	0.25505	0.48906	0.66857
21	0.25536	0.49072	0.671
22	0.25627	0.48986	0.66942
23	0.25585	0.49085	0.67046
24	0.25764	0.4914	0.67382
25	0.25694	0.49036	0.66981
26	0.25876	0.49247	0.67047
27	0.26142	0.49424	0.67243
28	0.25904	0.4941	0.6705
29	0.26	0.49409	0.67048
30	0.25909	0.49636	0.67121
31	0.26019	0.49471	0.67071
32	0.25914	0.49492	0.67213
33	0.26286	0.49671	0.6728
34	0.26168	0.49789	0.67299
35	0.26133	0.48521	0.67172
36	0.26343	0.49714	0.67256

jacobbieker commented 1 year ago

Also, to look at the percentiles numerically, for how much we might need to scale values, for timestep 1, the generation quantiles are:

        index    target
0  0.24395  0.081982
1  0.10000  0.033984
2  0.49761  0.195541
3  0.50000  0.196777
4  0.69554  0.306373
5  0.90000  0.473351

peterdudfield commented 1 year ago

Is there then an easy way to calculate how much we should scale the plevels by in order to for these targets?

jacobbieker commented 1 year ago

I was thinking I can calculate the numerical amount needed for each forecast horizon to scale the values correctly. For the timestep 1 above, that would be scaling the 10th percentile prediction by 0.41, the median prediction by 1.006, and the 90th percentile prediction by 1.545

jacobbieker commented 1 year ago

These quantiles are computed across all the years of data, so should give our best estimate for how much to scale correctly

peterdudfield commented 1 year ago

nice, yea, doing it for all timestemps will give you ad idea of the spread. Maybe we can get away with just using two scale factos, one up and one down for all timesteps, but lets see

jacobbieker commented 1 year ago

Yeah, I'll calculate it and see how much they change, the 0th hour prediction is the most different from the rest, but yeah, see how much of a difference it makes

jacobbieker commented 1 year ago

The scaling factors for each timestep, and the mean/stddev are below:

All:
[[0.47418373 1.00019988 1.45901899]
 [0.41452197 1.00632088 1.54501274]
 [0.39161805 1.03203044 1.63497287]
 [0.39209704 1.02341976 1.63850767]
 [0.39170967 1.02939344 1.63598316]
 [0.39318545 1.02599887 1.62689891]
 [0.39106556 1.02425171 1.63017245]
 [0.39319158 1.03060615 1.63585427]
 [0.39466618 1.03444454 1.63140405]
 [0.3934146  1.03320967 1.6300984 ]
 [0.39480151 1.03440758 1.63694832]
 [0.39707246 1.0327787  1.63098248]
 [0.39715211 1.02805242 1.62717832]
 [0.39581467 1.02794837 1.62952877]
 [0.39585526 1.03353782 1.62414992]
 [0.39399505 1.02411633 1.62830027]
 [0.39334753 1.03255128 1.63141406]
 [0.39540726 1.03177741 1.62706089]
 [0.39165648 1.02770355 1.62612289]
 [0.39343695 1.02721978 1.63042219]
 [0.39361931 1.0291603  1.63116299]
 [0.39324358 1.02418464 1.62368886]
 [0.39186305 1.02723264 1.62841203]
 [0.39253025 1.02401655 1.62505552]
 [0.38937413 1.0227155  1.61544505]
 [0.39067209 1.02571482 1.62713719]
 [0.38710318 1.02021205 1.624785  ]
 [0.38230225 1.01551947 1.61946814]
 [0.38657031 1.01586616 1.62500811]
 [0.38489215 1.01589174 1.62510484]
 [0.38641189 1.01001207 1.62321952]
 [0.38450906 1.01381723 1.62409143]
 [0.38631777 1.01316135 1.62020018]
 [0.38002044 1.00872639 1.61757753]
 [0.3819141  1.00546322 1.61673452]
 [0.38236209 1.0403159  1.6209885 ]
 [0.37895418 1.00735968 1.61806424]]

Mean:
[0.3932663  1.02322536 1.6201669 ]

Stddev:
[0.01485366 0.00966338 0.03048486]

peterdudfield commented 1 year ago

Hmm first two are a lot lower,

perhaps to keep it simple, we should just start with lower=0.4 and upper=1.6

jacobbieker commented 1 year ago

Yeah, that sounds good to me