This is a massive PR! But, all is centered around the new workflow for flagging found in virridy/non_targets_workflow/0_auto_qaqc.Rmd. I suggest reviewing this workflow with a fine comb, as the steps in qaqc pipeline have changed quite drastically.
Namely, I had to bake in the option of performing this workflow across either ALL sondes, or just those in the FCW network, which led to a lot of changes in the majority of our older functions. I also started to explore pulling in data that's been verified to make better flags (see the "define thresholds" section in the Rmd). And, you will also see that there are new flags I've developed including a way to flag DO "noise", capping turbidity at 1000 NTU, and a bad-calibration fixer that is still a work in progress.
There is also a weird thing that needs to be done with how we first pull in our API data (data/api/krw_everything_backup2.0_mdt_dl/). This is because data from 2023 can no longer be pulled from HydroVu (names have changed). So with that, please do not attempt to download any API data before 2024! I had to write code to "munge" the old data correctly, munge_api_data_hacky(), that we should remove entirely once we move past working with older data.
Something else to be mindful of is the idea that this workflow works across all data in one go. We will need to modify 0_auto_qaqc.Rmd and likely some of the functions to make them work when we migrate to doing this process in real-time/tacking on new data with the old data.
This is a massive PR! But, all is centered around the new workflow for flagging found in
virridy/non_targets_workflow/0_auto_qaqc.Rmd. I suggest reviewing this workflow with a fine comb, as the steps in qaqc pipeline have changed quite drastically.Namely, I had to bake in the option of performing this workflow across either ALL sondes, or just those in the FCW network, which led to a lot of changes in the majority of our older functions. I also started to explore pulling in data that's been verified to make better flags (see the "define thresholds" section in the Rmd). And, you will also see that there are new flags I've developed including a way to flag DO "noise", capping turbidity at 1000 NTU, and a bad-calibration fixer that is still a work in progress.
There is also a weird thing that needs to be done with how we first pull in our API data (data/api/krw_everything_backup2.0_mdt_dl/). This is because data from 2023 can no longer be pulled from HydroVu (names have changed). So with that, please do not attempt to download any API data before 2024! I had to write code to "munge" the old data correctly,
munge_api_data_hacky(), that we should remove entirely once we move past working with older data.Something else to be mindful of is the idea that this workflow works across all data in one go. We will need to modify
0_auto_qaqc.Rmdand likely some of the functions to make them work when we migrate to doing this process in real-time/tacking on new data with the old data.