search

HakaiInstitute / hakai-datasets

Hakai Datasets that are going into https://catalogue.hakai.org/erddap/
0 stars 0 forks source link

add NT instrumentation view #134

Closed fostermh closed 1 year ago

fostermh commented 1 year ago

also fix hakai instrumentation view processing stage

fostermh commented 1 year ago

still to do would be to create the xml file's for the new datasets. also, an 'organization' field was added to the post_qc view so this should be added or dropped from the xml file for the hakai instrumentation dataset

JessyBarrette commented 1 year ago

Hum not sure if that's for the same purpose but I've made similar views and datasets but in the cioos-pacific-erddap. The different components are already within the dev branch

Le jeu. 2 févr. 2023 6:42 p.m., Matthew Foster @.***> a écrit :

@.**** commented on this pull request.

In views/HakaiWaterPropertiesInstrumentProfileResearch.sql https://github.com/HakaiInstitute/hakai-datasets/pull/134#discussion_r1095205646 :

@@ -7,14 +7,13 @@ FROM ctd.ctd_post_qc_data d WHERE (

  • d.cast_processing_stage >= '8_binAvg' :: ctd.processing_stage
  • OR d.cast_processing_stage >= '8_rbr_processed' :: ctd.processing_stage
  • d.cast_processing_stage >= '10_qc_pi' :: ctd.processing_stage

the processing_stage seemed wrong to me. but maybe it was ment to be the same for reasearch and provisional, not sure

— Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/pull/134#pullrequestreview-1282073068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHICYON7XRAEWZV364QRHIDWVRA7VANCNFSM6AAAAAAUPUHS7Q . You are receiving this because your review was requested.Message ID: @.***>

JessyBarrette commented 1 year ago

@fostermh I assumed back then that we wanted those non hakai datasets within the CIOOS Pacific ERDDAP? Is that true or we would want them in Hakai's one

fostermh commented 1 year ago

given that we are hosting the original data source in the hakai database I assumed we would serve it from a hakai erddap along with a metadata record in the hakai catalogue. all of which would flow up into cioos pacific. Perhaps a discussion to have with the group. As you say, from a technical perspective either way would work.

raytula commented 1 year ago

It makes sense to me that it would flow from Hakai to CIOOS.

JessyBarrette commented 1 year ago

Ok I'll move all the datasets from cioos-pacific to hakai erddap.

JessyBarrette commented 1 year ago

@fostermh I added little change to your table view of the NT tables to include both the static records (when cast_type is static) and downcasts.

I generated a new table for provisional and research datasets for all the different organizations.

There will likely be a separate discussion with each different dataset and the related people.

fostermh commented 1 year ago

sorry if this is a silly comment, I have not been involved in the development of the datasets to date, but shouldn't the parks canada, skeena, and UofW datasets be in their own branches? How do you prevent accidentally deploying datasets to erddap if you merge all this in before it is approved by the respective organizations?

JessyBarrette commented 1 year ago

all those goes to the dev branch which is presented on the goose.hakai.org/erddap.

going from dev to master has often been a bit tricky and I have been doing some git workaround to help merging one to the other without merging all.

Nate and I talked before Christmas about a few ways to improve the workflow by adding a development folder where only the dev stuff will live. We haven't yet implemented that but maybe it will be time with all the different projects happening in parallel right now.

fostermh commented 1 year ago

one option would be to maintain a branch for each dataset you are working on. it would be a 'hot fix' in the world of git flow and would branch off of master. When ready to test you merge it into development. once approved and your happy, the same branch is then merged into master. I expect Nate has other suggestions as well.

JessyBarrette commented 1 year ago

yeah I'm happy to fit which everway. The other issue I forgot to mention is that the database view and tables from this repo are drop nightly and regenerated from the master branch only and our development erddap is also pulling from the hakai database (not hakaidev)

JessyBarrette commented 1 year ago

one option would be to maintain a branch for each dataset you are working on. it would be a 'hot fix' in the world of git flow and would branch off of master. When ready to test you merge it into development. once approved and your happy, the same branch is then merged into master. I expect Nate has other suggestions as well.

That's a good idea, I may close this present PR and reapply the proposed changes here following this suggested method

JessyBarrette commented 1 year ago

Thinking about it better, I think it will be easier to add those present changes to the dev branch and proceed to modify the repository workflow as discussed with @n-a-t-e before the break.

Until the new workflow is establish (which shouldn't take long), merging this PR and #138 should make the different datasets available on the development erddap.