search

bcgov / wqbc

An R package for water quality thresholds and index calculation for British Columbia
http://bcgov.github.io/wqbc/
Apache License 2.0
19 stars 9 forks source link

Examine code to variable errors in standarize_wqdata #143

Closed HeatherGranger closed 4 years ago

HeatherGranger commented 4 years ago

Compare lookup_codes to variables. Errors in finding fecal coliforms, BOD etc. The codes may just pertain to what guidelines are being calculated. Likely expand list of codes to include all usual water quality parameters in EMS.

HeatherGranger commented 4 years ago

I think the best way is to make sure the codes/variables are the same as the codes/variables in the limits reference table. Where is the current codes.csv and limits.csv?

joethorley commented 4 years ago

Yes I agree with making the codes/variables the same as in the limits reference table.

The limits.csv doesn't exist anymore because the limits are pulled from 

limits <- bcdata::bcdc_get_data(record = "85d3990a-ec0a-4436-8ebd-150de3ba0747")

The codes.csv file is in data-raw/codes.csv.

The codes.csv file hasn't changed since 2017

❯ git log --follow data-raw/codes.csv
commit a91a953e815125ec0040df71e732a19cd3681e4f
Author: ateucher <andy.teucher@gov.bc.ca>
Date:   Thu Jun 8 16:53:14 2017 -0700

    Use Col.unit for colour units everywhere (#77)

commit 87dff6f4b27ececb1ea580e6e13c385e49774dc5
Author: Andy Teucher <andy.teucher@gov.bc.ca>
Date:   Thu Mar 30 11:01:59 2017 -0700

    Get codes from ems_codes, add Alkalanity pH 4.5/5.2

commit f37acb440251d29176b3b5744fcd8f6d9e1a1c9f
Author: joethorley <joe@poissonconsulting.ca>
Date:   Fri Feb 3 08:04:25 2017 -0800

    more variable codes plus coliforms and units
joethorley commented 4 years ago

As far as I can determine at the this point codes serves two roles: 1) it provides EC_Codes for EMS_Codes and 2) it allows us to recognise variables that are missing from limits.csv.

HeatherGranger commented 4 years ago

@joethorley do you think the best way forward then, is for me to update the codes.csv with all the variables in the limits reference table? If so, what is the best way for me to get the subset of limits from data bc that are listed in the limits reference table? Other than me going through the databc guideline list and pulling out the aquatic life guidelines!

joethorley commented 4 years ago

I'm starting to think we should simply add those codes with missing limits to limits and just used codes as a look up for EC_Codes vs EMS_Codes?

HeatherGranger commented 4 years ago

As per the Friday Aug 7th Heather/Joe chat - some variables we want to bring into the app and plot don't have limits. Manually adding variables/codes to codes.csv seems the best way to move forward at this time. Heather to test when @joethorley confirms wqbc is ready.

HeatherGranger commented 4 years ago

@joethorley tested shinyrems with new variables added to codes.csv. 'Chlorophyll A' and 'Extinction Depth' not retained in Tidy tab. I re-installed the latest wqbc version before running shinyrems; is there something else I need to do?

joethorley commented 4 years ago

I'll have a look this afternoon. Are your changes on a branch?

HeatherGranger commented 4 years ago

The updated codes.csv was added on the add-codes branch, which we merged into master last week. We were looking at limits-and-codes.R on our call to try and diagnose why the new codes weren't integrating. I believe that's where we left it.

joethorley commented 4 years ago

@HeatherGranger - can you resolve this issue?

HeatherGranger commented 4 years ago

@joethorley No, I'm not sure where in limits-and-codes.R the issue is to ensure the codes.csv is being implemented in wqbc. Tested now and new variables in codes.csv are being removed in the tidy tab of shinyrems.

joethorley commented 4 years ago

Sorry that was a confusing comment.

I meant can you resolve this issue: https://github.com/bcgov/wqbc/issues/150

Its holding things up

HeatherGranger commented 4 years ago

@joethorley Answered!

HeatherGranger commented 4 years ago

@joethorley One more thing on this - as we move forward and I discover other variables we've missed including in codes.csv, is the easiest thing just for me to add to codes.csv and push?

joethorley commented 4 years ago

That is a good question - we should probably discuss what makes most sense as a workflow.

HeatherGranger commented 4 years ago

@joethorley looking to complete this issue before tomorrow afternoon. If @JessicaPenno needs to add variables to codes.csv, what is the best workflow for her?

joethorley commented 4 years ago
  1. clone bcgov/wqbc
  2. create new fork
  3. open codes.csv in Excel and manually add variables
  4. if new units (ie not recognized in R/units.R) let me know and I will update wqbc to recognize them (will require user to pull master and merge into fork)
  5. run data-raw/data-raw.R line by line ensuring their are no errors
  6. rebuild the package
  7. check the package to ensure no errors, warnings or notes
  8. do pull request for changes
  9. rebuild shinyrems
HeatherGranger commented 4 years ago

Thanks! @JessicaPenno these are the steps. I'll close this issue now.