search

quince-science / QuinCe

QuinCe is an online tool for processing and quality control of data from scientific instruments, with a primary focus on oceanic data.
https://quince.science
GNU General Public License v3.0
7 stars 8 forks source link

Processing job results not written in batches #1364

Open squaregoldfish opened 5 years ago

squaregoldfish commented 5 years ago

The Auto QC and Data Reduction jobs collect queries together and run them in batches. Although this looks as though it works, the database is still writing them as individual queries. Work out why.

squaregoldfish commented 5 years ago

Looks like this only applies to Auto QC

squaregoldfish commented 5 years ago

The Auto QC jobs are all updates. Is it possible these aren't batchable? If so, is deleting and re-inserting the way forward?

squaregoldfish commented 5 years ago

We should also be able to work out if any of the records have changed, and therefore not even try to update the records. This would be a bigger improvement than batching, probably.