search

VForWaTer / metacatalog

Modular metadata management platform for environmental data.
https://vforwater.github.io/metacatalog
GNU General Public License v3.0
3 stars 1 forks source link

Add data import API endpoint #93

Open mmaelicke opened 4 years ago

mmaelicke commented 4 years ago

Some missing tasks from #92 still need to be implemented. API endpoint needs to infer the scales from given datasource type and data type and create DataSource, DataSourceType and DataType from the data. Then, import data

mmaelicke commented 3 years ago

@AlexDo1, this is an issue, I would like to implement with the next minor version. As you are working on upload scripts a lot, I would like to hear your opinion on this issue. Given that a user provides CSV data and we are only talking timeseries right now, do you think this can be done? For other datatypes (i.e. Raster images), a possible pathway would be to store the actual data into netCDF by default.

AlexDo1 commented 3 years ago

I don't quite understand yet, do you want to infer the temporal_scale from the given timeseries data (that shouldn't be a problem if there are no gaps in the data) and also generate the DataSource, DataSourceType and DataType automatically?

mmaelicke commented 3 years ago

something like:

api.import_data(session, entry, DataFrame)

The function needs to check if a Datasource exists -> append data, else create datasource On append -> check integrity (temporal scale is matching) On import -> infer DataType from index (datetime, multi-index, etc) and DataSourceType (timeseries goes into the timeseries table and i.e. *.geotiff goes onto the harddrive.

I think for this to be implemented we need additional IOExtension functionality: store binary files (like Raster) and save netcdf. Generally, a timeseries is allowed to have gaps. That does not affect the temporal scale.

A roadmap could be to identfy, what could easily be done and open new issues for this on the 0.5 and 0.6 milestone. The harder stuff, or anything that needs discussion goes into the 1.0.0 milestone.

AlexDo1 commented 2 years ago

I think this is also still relevant and I can work on the issue. What do you think @mmaelicke?