search

HakaiInstitute / hakai-datasets

Hakai Datasets that are going into https://catalogue.hakai.org/erddap/
0 stars 0 forks source link

Dataset Submission: HakaiChlorophyllSampleResearch #32

Closed JessyBarrette closed 1 week ago

JessyBarrette commented 3 years ago

Hakai Dataset Submission

Below are listed all the different steps related to the initial submission of a dataset.

A more detailed written and visual description of every step is available respectively here and here.

Submission steps

Initial Submission (Data Administrator)

ERDDAP Dataset Creation (Data Integrator)

Dataset Review (Data Administrator)

CIOOS Metadata Form

Dataset Completion (Data Integrator)

JessyBarrette commented 3 years ago

This ERDDAP dataset will be generated by querying the Hakai database for the nutrient data with a filter on only the approved dataset.

JessyBarrette commented 3 years ago

@jdelbel We would need to create a Metadata record specific to the Hakai Chlorophyll data here.

jdelbel commented 3 years ago

Thanks Jessy!

I will start looking into this. Do you have a link to an example metadata record to guide me? I'm sure I can find one on CIOOS, but if you have one handy that would be great.

Justin

On Tue, 27 Apr 2021 at 09:55, Jessy Barrette @.***> wrote:

@jdelbel https://github.com/jdelbel We would need to create a Metadata record specific to the Hakai Chlorophyll data here https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/issues/32#issuecomment-827759765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJPW44VFXBFC7XJJYNHDXYTTK3T7RANCNFSM43VLMPHQ .

-- Justin Del Bel Belluz, MSc. Research Technician - Bio-Optical Oceanography Hakai Institute 100 - 1002 Wharf Street Victoria, BC Canada V8W 1T4 www.hakai.org

JessyBarrette commented 3 years ago

Sure, you can have a look at the CTD provisional dataset form

JessyBarrette commented 3 years ago

This dataset will essentially be a link to the endpoint /eims/views/output/chlorophyll on the database with a filter on the flags (AV, potentially SVC) and maybe specific sites. Finally, we'll need to pivot the data on the filter variable.

JessyBarrette commented 3 years ago

@jdelbel Here's a proposition, we'll go ahead with setting up the ERDDAP dataset, and filter the data as of now either all or just SVC and AV data. This dataset will be available temporarily on the development erddap.

You'll be able to easily review the final result. Once you are happy with the data itself available on ERDDAP, we can push it to the production server.

jdelbel commented 3 years ago

Sounds good. Going to start on this next week. It's 95% ready, so might not actually take long.

On Fri, 30 Apr 2021 at 11:33, Jessy Barrette @.***> wrote:

@jdelbel https://github.com/jdelbel Here's a proposition, we'll go ahead with setting up the ERDDAP dataset, and filter the data as of now either all or just SVC and AV data. This dataset will be available temporarily on the development erddap https://goose.hakai.org/erddap/tabledap/index.html?page=1&itemsPerPage=1000 .

You'll be able to easily review the final result. Once you are happy with the data itself available on ERDDAP, we can push it to the production server.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/issues/32#issuecomment-830284093, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJPW44Q2F6HEFY5HZBRTHUDTLLZZNANCNFSM43VLMPHQ .

-- Justin Del Bel Belluz, MSc. Research Technician - Bio-Optical Oceanography Hakai Institute 100 - 1002 Wharf Street Victoria, BC Canada V8W 1T4 www.hakai.org

JessyBarrette commented 3 years ago

@jdelbel I made a first draft of a QCing notebook for the Chlorophyll-a data. Very preliminary, but that's something to work with: https://colab.research.google.com/gist/JessyBarrette/4cdb8a41f0bb03c7d5d7a65a6e7dd48d/jessy-run-hakai-nutrients-tests.ipynb

JessyBarrette commented 3 years ago

Due to questionable historical data calibration, we will omit as of now the data collected prior to ~2018. @jdelbel please provide the exact at which we should start presenting the data.

jdelbel commented 3 years ago

Using the calibration column on the portal, only filter for data with the 2018-05-04 and 2019-05-09 calibrations applied.

I am going through the flags on these data today and adding the ADL flag to those run above the sensor range.

On Fri, 21 May 2021 at 13:28, Jessy Barrette @.***> wrote:

Due to questionable historical data calibration, we will omit as of now the data collected prior to ~2018. @jdelbel https://github.com/jdelbel please provide the exact at which we should start presenting the data.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/issues/32#issuecomment-846237657, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJPW44UEAGWHKQYJ6ZFNUVDTO267FANCNFSM43VLMPHQ .

-- Justin Del Bel Belluz, MSc. Research Technician - Bio-Optical Oceanography Hakai Institute 100 - 1002 Wharf Street Victoria, BC Canada V8W 1T4 www.hakai.org

JessyBarrette commented 3 years ago

The CIOOS metadata form is now available here: https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/7U7b8oPpeTN6gjvXlUCTGJr5pga2/-MadZvZJ4ZeikpCgRcV8

JessyBarrette commented 3 years ago

Pheao pigments are still missing a standard_name, thanks to @timvdstap it's in the process. We'll update the Hakai Chlorophyll dataset to reflect that once the standard_name _mass_concentration_of_phaeopigments_in_seawater is available.

JessyBarrette commented 3 years ago

@jdelbel the initial version of the Chlorophyll dataset is available here: https://goose.hakai.org/erddap/tabledap/HakaiChlorophyllSampleResearch.html

Let me your thoughts!

JessyBarrette commented 3 years ago

@jdelbel we would need to specify within the CIOOS record in the title the type of dataset it is Provisional/Research Same applies to the provisional dataset

JessyBarrette commented 3 years ago

@jdelbel The Research dataset erddap and forms are now available here for final revision:

JessyBarrette commented 2 years ago

Metadata record was moved to the Hakai CKAN https://cioos-siooc.github.io/metadata-entry-form/#/en/hakai/7U7b8oPpeTN6gjvXlUCTGJr5pga2/-MadZvZJ4ZeikpCgRcV8

JessyBarrette commented 2 years ago

@jdelbel a DOI is now associated with the Chlorophyll Research dataset https://doi.org/10.21966/wsvt-ew96

I added the DOI to the metadata form and also added a placeholder for the ERDDAP dataset. The link will be broken until we make available the dataset on the production server. I still need to add the DOI to the ERDDAP dataset itself too.