search

quince-science / QuinCe

QuinCe is an online tool for processing and quality control of data from scientific instruments, with a primary focus on oceanic data.
https://quince.science
GNU General Public License v3.0
7 stars 8 forks source link

Assign instruments to projects #1144

Open squaregoldfish opened 5 years ago

squaregoldfish commented 5 years ago

When an instrument is set up, we can define which projects its data should go to (ICOS, SOCAT, CMEMS etc). Then exporter scripts can interrogate that to find out which data sets to extract.

A bit mask is probably the best data structure.

We may need multiple bit masks for different stages (e.g. some projects may only want final data, some might only need NRT, etc)

squaregoldfish commented 5 years ago

An alternative (probably addition) is to add the project info to the export manifest.

squaregoldfish commented 5 years ago

Maybe better than a bit mask will be a database table of known projects with a 1:∞ link table to the instruments.

squaregoldfish commented 5 years ago

This can also be used to track failed uploads. Each data set can have an upload status variable, with a status for each project it's attached to.

squaregoldfish commented 5 years ago

For the API, Maren currently uses these calls:

Screen Shot 2019-09-25 at 16 03 58

Base new API calls on this workflow.

squaregoldfish commented 5 years ago

Also add into the project definitions which export format should be used. This can then be used to configure Nuka for the versions with non-salinity flags, and Maren's script won't need any special logic.

squaregoldfish commented 4 years ago

Possible database structures: ink-image

If we store exports, do we want to store the PIDs/DOIs?

squaregoldfish commented 4 years ago

Higher priority tasks have put this on hold.

squaregoldfish commented 4 years ago

The export scripts will have to become more intelligent, knowing which projects a dataset has been exported for, and which not.