After #178, we now can mask out segments for validation and testing, but since those segment id's and dates still exist in prepped.npz, when we join them to the observation file during combine_metrics they'll wind up included in the training and validation performance metrics. TLDR, we need to pass those hold-out segments to combine_metrics and mask them out so they're only included in the metrics for the appropriate partition.
SimonTopp
commented
2 years ago
After #178, we now can mask out segments for validation and testing, but since those segment id's and dates still exist in
prepped.npz, when we join them to the observation file duringcombine_metricsthey'll wind up included in the training and validation performance metrics. TLDR, we need to pass those hold-out segments tocombine_metricsand mask them out so they're only included in the metrics for the appropriate partition.