search

ioos / ioos_qc

:ballot_box_with_check: :ocean: IOOS QARTOD and other Quality Control tests implemented in Python
https://ioos.github.io/ioos_qc/
Apache License 2.0
42 stars 27 forks source link

GSoC: Workflow Overview and Missing Functionalities for Glider QA/QC #113

Open Sakshamgupta90 opened 1 month ago

Sakshamgupta90 commented 1 month ago

Hi @ocefpaf @mwengren @MathewBiddle, As I am delving deeper into the project, I believe having a clear understanding of the workflow would greatly enhance my productivity and ensure that my contributions align well with the project's goals. Could you please provide a brief overview of the current workflow for the project? Additionally, it would be very helpful if you could highlight any specific functionalities or checks that are currently missing or need improvement. This will help me to focus my efforts more effectively and contribute in the most meaningful way possible.

ocefpaf commented 1 month ago

@Sakshamgupta90 I pinged you on slack on this topic. We can use the document I shared with you to start narrowing down the test we need to implement and/or modify. We can keep this issue open and start listing them here as we read the doc.

Sakshamgupta90 commented 1 month ago

Hi @ocefpaf, I will make sure to list everything on this PR. And will update you by providing the list of functions that need to implement or modify.

Sakshamgupta90 commented 1 month ago

Hi @ocefpaf, List of All the Tests, that are implemented:

I checked the code base, and found that these 5 tests are not yet implemented:

Out of these 5 Tests, first 2 are from Group1 (i.e. Required Tests) and rest are from Group3 (Suggested Tests).

ocefpaf commented 1 month ago

Timing/Gap and Syntax will require some raw data examples that we request i tomorrow's meeting. The TS one is quite challenging to code and we should start with a simple case for it. The challenges are:

  1. Coastal data may differ from classic TS fits.
  2. There are two options there, fitted equation or a numeric table.
  3. The tests are assuming that the temperature sensor is better than the salinity one, which is usually true, and compare the expected salinity based on temperature alone.

Can you updated your message above to list all tests, checkmark those that already exist and add a link in the code base so we can track them? Some, while may be already coded, may require some adjustments for the glider case and we need to check.

@leilabbb may have some insight for us here on which test is OK and which is not in the existing ones.

ocefpaf commented 1 month ago

@Sakshamgupta90 following up from today's meeting. We should get the notebook and test data from #106 and run all the tests that are already implemented. According to @leilabbb some tests results are not working as expected and we need to fix them.

leilabbb commented 1 month ago

@ocefpaf @Sakshamgupta90 can you put a link to the Slack channel to join the conversation?

ocefpaf commented 1 month ago

Leila I invited you using the slack interface. In any case, see if this link works.

leilabbb commented 3 weeks ago

These are the tests the Glider DAC is interested in:

To be added to the QC workflow: Test 3) Location Test (Required) -> Link

Already in the QC workflow: Test 4) Gross Range Test (Required) -> Link Test 7) Spike Test (Strongly Recommended) -> Link Test 8) Rate of Change Test (Strongly Recommended) -> Link Test 9) Flat Line Test (Strongly Recommended) -> Link

The links below give access to a notebook with the Glider DAC QC workflow, the example netCDF data file used in the notebook and the configuration yaml file needed to run the the QC process.

GLIDER-DAC QC Notebook

Example netCDF File

Configuration Yaml File

IMPORTANT: In the notebook, the last section checks the flags and you will see the differences I discovered implementing the QARTOD tests in two different ways.

Sakshamgupta90 commented 2 weeks ago

Hi @leilabbb @ocefpaf, I've observed differing results between the two methods used to implement the QARTOD flat line tests. I'm a bit confused because both methods show different outcomes. Could you please advise on which method you find most appropriate?

These are the tests the Glider DAC is interested in:

To be added to the QC workflow: Test 3) Location Test (Required) -> Link

I've reviewed the location tests for both general models and glider dac, and their descriptions appear quite similar in terms of checking latitude, longitude, and displacement. Could you clarify what specific adaptations are needed for implementing to make sure I'm in sync?

leilabbb commented 2 weeks ago

METHODE 2: Using the ioos_qc.qartod.flat_line_test is more accurate. It shows more reasonable test results.

For the Location test, it is on the list to add to the Glider DAC QC workflow, so I do not have any comment on that.

Sakshamgupta90 commented 2 weeks ago

For the Location test, it is on the list to add to the Glider DAC QC workflow, so I do not have any comment on that.

You mean the list that I created above? If yes, then I listed Location test in the implemented section.