Glider Profile QC example data set

ioos / ioos_qc

:ballot_box_with_check: :ocean: IOOS QARTOD and other Quality Control tests implemented in Python

https://ioos.github.io/ioos_qc/

Apache License 2.0

46 stars 27 forks source link

Glider Profile QC example data set #58

Open kerfoot opened 3 years ago

kerfoot commented 3 years ago

@ocefpaf @kwilcox

As discussed yesterday, we'd like to use this package to start running some standardized QARTOD (and potentially others) QC tests on the data we receive at the IOOS Glider DAC. You suggested that I provide some example files. This leads me to 2 pathways:

DAC compliant NetCDF files
I'm working on a more generalized glider data model for getting raw data from a variety of gliders/formats into a DataFrame and then to XArray which allows for exporting the data sets either as a trajectory or a set of profiles. So this provides the opportunity for data providers to apply their own level of qc prior to submitting to the DAC.

How/where should I provide these examples? Would be great to get some working examples of how the package is used with all 3 of these data types.

ocefpaf commented 3 years ago

How/where should I provide these examples?

I can help you write those and add a notebook example to the IOOS gallery.

Would be great to get some working examples of how the package is used with all 3 of these data types.

Just to clarify, the 3 data types you mean are raw, trajectory, and set of profiles?

kerfoot commented 3 years ago

I think the most useful examples would be:

QC on an individual profile from a DAC compliant NetCDF
QC on a profile contained in either a pandas DataFrame or XArray Dataset

While these 2 types are fairly similar, it would provide users with the ability to QC profiles before or after they write DAC compliant NetCDF files.

I am more than happy to help with this as it's something that I will need to do both as a DAC administrator and a data provider. Let me know how I can help and/or get this started.

It might be best to start with a zoom/webex to discuss the details in order to make sure that I am communicating what I want correctly.

ocefpaf commented 5 months ago

@kerfoot we are revisiting this issue and I would like to ask if you have, or can provide, a simple dataset with known QA/QC problems that we could use as test case/example. While we could create one I believe you are the most experienced glider person here to help us out.

kerfoot commented 5 months ago

@ocefpaf perfect timing! We just had our weekly DAC check in call and are working on filing an issue with a comprehensive example of a single profile. My guess is it will be filed within the next few days or so.

ocefpaf commented 5 months ago

My guess is it will be filed within the next few days or so.

Awesome, we are looking for a GSoC candidate to help out on this front. If a candidate is selected, these examples will be very helpful.