Time masking fails with set_t.csv containing sub-hourly resolutions

[GraemeHawker]

Using a set_t.csv file with 15 minute intervals and applying the following time mask: time: masks: {function: week, options: {day_func: 'extreme', tech: 'demand_power', how: 'max'}} function: resample function_options: {'resolution': '6H'}

Generates an InvalidIndexError due to duplication, as the time masking functions appear to ignore any sub-hourly component of the timestamp. Changing set_t.csv to hourly timesteps for otherwise identical model corrects the issue.

[GraemeHawker]

Hourly timesteps appear to be hardcoded at https://github.com/calliope-project/calliope/blob/2ea865754b202ea4a19a7f5960fcd7b8481a1f6a/calliope/time_masks.py#L62 and https://github.com/calliope-project/calliope/blob/2ea865754b202ea4a19a7f5960fcd7b8481a1f6a/calliope/time_masks.py#L163-L164

[brynpickering]

Should be possible to update the code to use pandas.infer_freq on the time dimension to get the frequency of the input data, feeding that back in where necessary. If you have a dev version you could try it (in the first instance, possibly freq=pd.infer_freq(series.index) and in the second freq=pd.infer_freq(data.t.index)) Otherwise I'll probably only get to it at the end of the week!

[GraemeHawker]

I was thinking something like that would work, but only if the frequency of the input data is constant. While this is probably the case for 99% of models, I assume that the ability to define each entry in set_t.csv (rather than defining a single start timestamp, a frequency and a number of steps as per a pandas index) means that we shouldn't assume it?

In which case perhaps the solution is to draw the timestamps from the raw index that fall between start_hour and end_hour (also adjusted appropriately)

Is the assumption of constant frequency input data already made implicitly elsewhere?

(happy to branch and create pull request once I've figured it out!)

[brynpickering]

Thanks for the PR Graeme! At the moment, an inconsistent time series isn't really picked up. Only the first two steps of the whole series set the time resolution. E.g. if you have 1 hour timesteps overnight and 15 minute timesteps in the day as your input then calliope would see it as all 1 hour timesteps (i.e. time resolution = 1 and timesteps per day = 24).

However, if you give regular input and apply masking then you will get a correctly formatted inconsistent timestep. E.g. if one week of high wind availability is set to be 15 minute resolution then the time resolution of steps in that week will be 0.25.