Hakai Profile QC initial deployment

[JessyBarrette]

Hakai Profile QC initial deployment

The hakai profile qc tool has been for a while used to generate the research and provisional datasets made available on the Hakai ERDDAP by generating NetCDF files. We need to include the qc results within the hakai database.

Any errors are tracked through Sentry: https://sentry.io/organizations/hakai-institute/projects/ctd-auto-qc/?project=6685251

Initial deployment steps on development environment

• [x] Create an app on hak4i.org to qc data in batch mode without taking over the the production server https://captain.server.hak4i.org/#/apps/details/hakai-profile-qc-batch-mode
• [x] Test run data from hak4i.org on test suite standalone
• [x] Test run data from hak4i.org on test suite and submit data to hakaidev DB through the goose.hakai.org/api
• [x] Test run data from hak4i.org on QU39 and submit data to hakaidev DB through the goose.hakai.org/api
• [x] Test run data from hak4i.org on all the profiles from hakaidev DB through the goose.hakai.org/api
• [x] Review goose.hakai.org/portal to make sure all the data is appropriately represented
• [x] Confirm all is good with hakai people.

Deployment on production server

• [x] Make a new deployment of the repo on hak4i.org for production purposes -> https://captain.server.hak4i.org/#/apps/details/hakai-profile-qc-batch-mode-production
• [x] Make a dry-run
• [x] #15
  • [x] unqced profiles
  • [x] manually qced profiles
• [x] Confirm everything has been appropriately qced
• [x ] Add a cron job ~on hecate~ to run the hakai qc tool on regular interval to qc the latest data submitted
  • [x] Can be deployed on ~hecate.hakai.org directly or just~ re deploy hak4i.org?
• [x] Connect the Hakai ERDDAP CTD profile Datasets to the respective appropriate views
  • [x] Development ERDDAP: http://goose.hakai.org/erddap
  • [x] Production ERDDAP: http://catalogue.hakai.org/erddap
• [x] COMPLETED

Related issues

The present issues are just present here to help track but do not need to get resolved prior to the deployment

13

14

[JessyBarrette]

A full QC of the hakaidev CTD profiles was completed last night with sucess and suggest no issues. The whole process was completed in ~6hours (~7000 profiles).

[JessyBarrette]

@fostermh I started populating the hakaidev CTD profiles schema with the automated qc flags generated by this repo.

All the flags are now made available on the goose.hakai.org/portal

I'm wondering if you are aware of any issues related to those latest changes? As far as I can see everything seems good and manage to handle all those flags accordingly.

[JessyBarrette]

Well unfortunately the dry run ended up not been a dry run due to a bad handling of the environment variables. QC flags are now already available on hakai db ctd schema for a major part of the CTD dataset.

[fostermh]

fun

[JessyBarrette]

Another dry run was completed on all the qced and unqced data ~10k profiles which last over 4 hours. We omitted for now the manually already qced data. This makes me wonder if want to add flags to those specific profiles. My first thought is yes we want that.

[JessyBarrette]

Since a major part of the flags are now already available the hakai ctd, @fostermh do you see any major issues, or I'm good to roll out the whole process?

[fostermh]

do you have an example file / cast I could look at that has had qc run on it in production?

[JessyBarrette]

Sure:	hakai_id	cause
01907467_2015-06-10T15:54:38Z	instrument hit bottom lowest data is flagged
80217_2021-04-08T16:10:59.000Z	density inversion and bottom hit
080217_2019-09-04T17:36:45.166Z	cap was left on the oxygen sensor which is making the whole profile bad

You will see those in the _flag and _flag_level_1 columns

looking at this now I'm realizing that some of the thresholds are too sensitive for the bottom hit detection and do_cap. The tests will need to be refined in the near future.

[fostermh]

it seems ok but the level 1 flags are not making it to the excel or csv downloads at the moment so I will need to sort that out

[JessyBarrette]

Yeah I added that as an issue here #14 not sure if we really want to add it to the portal output. People can still retrieve if through the api or ERDDAP.

[JessyBarrette]

All the deployment is now pretty much there. Only the cron job to hecate is left and will left for the new year.

[JessyBarrette]

This is an old issue that should have been closed a while back