Multiple train/test splits result in discontinuous batches

[SimonTopp]

https://github.com/USGS-R/river-dl/blob/a7629eb2aa1bd4c33a7c00fc6d650d7c6bd9ee3f/river_dl/preproc_utils.py#L112-L117

Here, if we have discontinuous training and testing groups (i.e. two sets of date ranges for both), and batches are set to anything other than 365, then I think this results in one batch that starts in the first date range and ends in the second. I think we should first group by water year, then split into batches and just pad and/or drop the last one. What do you all think?

[jsadler2]

Interesting. It's been a while since I wrote this (or thought about this ... or used this 😄). Have you confirmed that this is what happens?

[jdiaz4302]

This may be of interest as confirmation that multiple train/test splits result in discontinuous ~batches~ sequences.

[janetrbarclay]

Further confirmation if you look at the temps in a single sample (these are observed temps, # in the title is the seg_id)

[jdiaz4302]

Using the existing reduce_training_data_continuous function from the river_dl/preproc_utils.py file can help get continuous batches with nan values. For example, here is the 365-day sequence for pretraining and finetuning Ys when I applied it to only the finetuning Y (the gap in the finetuning Y is where the nans have been placed - summer):

If you apply reduce_training_data_continuous to the x variables, you end up with nan in the predictions and subsequently the loss function. Taking this approach in https://github.com/USGS-R/river-dl/pull/142 by applying reduce_training_data_continuous to only the Y array (and not the pretraining Y or X arrays) led to much worse RMSE (factor of 2). I assume this is because the model is exposed to out of bound x values that have no corresponding out of bound Y values (set to nan) but still associated with the 365-day sequence of other values, so it may lead to some misleading learning.

[jds485]

I think this issue has been addressed from #218