search

VForWaTer / scripts

Notebook collection for different purposes
0 stars 0 forks source link

Time scale resolution of LUBW gauge data #7

Closed AlexDo1 closed 3 years ago

AlexDo1 commented 3 years ago

The gauge data provided by LUBW switches from 1-hour resolution to 15-minute resolution at timestamp 2016-05-01 00:00:00. It looks like this is the same for each dataset/station.

We need an idea how to handle this in metacatalog.

Currently, time series entries are assigned a scale and it is not possible to vary that scale within an entry, so we would need to create multiple (partial) entries to store the data cleanly.

e.datasource.create_scale(
        resolution='15min', 
        extent=(dat.index[0], dat.index[-1]), 
        support=1.0,
        scale_dimension='temporal'
    )

@mmaelicke, @sihassl

mmaelicke commented 3 years ago

@sihassl, following the data model, we would need to split up the timeseries. Question is, if the equipment (and maybe even the position) was changed, or if just the data storage system was changed. Should be proceed like that or just store '15min' as the highest available resolution. Otherwise, we would need a video call, before we can proceed with uploads.

mmaelicke commented 3 years ago

@AlexDo1 we just discussed this issue.

We need to split the metadata into two Entry. You can just copy over the metadata and just change the temporal scale to the subset time-step and range. Please do also split the timeseries and assign each chunk to the correct Entry. Finally, the two Entries are grouped by a new EntryGroup Type called 'Split dataset' for now. The Entries themselves are not partial as they do also make sense on their own. In your private metacatalog instance, you can just add a new EntryGroup type and write the upload script for it. Then open an issue on metacatalog to add the new EntryGroup type globally. Thanks for pointing this out