search

HakaiInstitute / hakai-ctd-qc

Series of tests applied to the Hakai CTD profile data based on the QARTOD tests and other Hakai Specific ones.
0 stars 0 forks source link

Hakai QC tool speed increase #13

Closed JessyBarrette closed 1 year ago

JessyBarrette commented 1 year ago

The QCing process is pretty fast to run the qc with about 10 profiles per seconds qced. The slowest part of the whole process is related to querying the database to retrieve a set number of profiles. The tool right now retrieves specific profiles by their hakai_id. We could potentially increase that speed by:

  1. Query via an indexed variable
  2. Run the different queries in different threads by using python multithread
JessyBarrette commented 1 year ago

As of now the whole process to qc all the profiles and takes about 6h we can close this issue for now