Updated `harmonized_sdd()` function

[juandlt-csu]

I have made the following updates to the harmonize sdd workflow based on my last PR:

• Minor data prep
  • Take absolute values from the original result column
• Remove fails
  • Fail text has been updated to be specific to SDD
  • remove activity and laboratory comment column here.
• Clean up MDLs
  • We just look for "<" to MDL, gets flagged post harmonization
• Clean up "greater than" values
  • We just look for ">" and some other special characters, gets flagged post harmonization
    • Keep the data and remove the equal size, do the same for the stars
  • Have been updated to match those updates in chla
  • The other instances of potentially salvageable data are not considered because they constitute far less than 1% of the total dataset.
• Clean up approximated values/Gap fill harmonized values with depth columns
  • Flagged pre-harmonization
  • There are instances of approximate language, but those are unrelated to SDD. Very few instances of "*"
  • Gap filling harmonized values with depth column information based on depth column criteria
    • flag 0 = value and units not adjusted
    • flag 1 = value filled using activity depth measure column
    • flag 2 = value filled using activity bottom depth measure column
  • Values that have been gap filled will be tier 3.
• Remove remaining NAs
  • If the value or units are empty we drop them. We do not attempt to fill in units with depth column information.
  • In my original PR the idea of retaining bottom values was floating around, but since we are not doing that that is no longer a consideration.
• Harmonize value units
  • In the PR there was an issue with dropping all the values that were NA due to bottom language in the original result column. I have been told that it's okay to drop those values now and to not worry about retaining them and the pipeline reflects that decision.
  • We drop rows where there is no unit to convert.
• Gap fill, harmonize, and flag bottom depth data
  • Flagged pre-harmonization
  • Use this step to gap fill and harmonize bottom depth and ignore the other bits that are in this step for chla.
    • flag 0 = bottom depth value not adjusted and no indication that SD hit bottom
    • flag 1 = bottom depth value adjusted due to indication that SD hit bottom
• Aggregate and tier analytical methods
  • Retain those methods that have "unspecified" in their value. The unrelated text strings have been updated.
  • Using harmonized_local_time to determine time that the observation was taken
  • Sdd_scope_tag has been updated and takes on an approach similar to that of bottom language. There is scope tag and negate scope tag language being considered now.
  • Tiers:
    • 0 = Restrictive. Time reported within the allotted window and method indicates use of viewscope.
    • 1 = Narrowed. Either time reported within the allotted window or method indicates use of viewscope, but not both.
    • 2 = Inclusive. Time not reported/time reported outside of window, and no indication of viewscope used.
    • 3 = Inclusive. harmonized value gap filled with depth column.
• Flag
  • MDLs
    • 0 = value not adjusted and value is greater than 0.01
    • 1 = original NA value adjusted using MDL method
    • 2 = provided value below 0.01; not adjusted
  • Greater than
    • 0 = value less than 31m with NO special characters
    • 1 = value less than 31m with special characters
    • 2 = value greater than 31m with NO special characters
    • 3 = value greater than 31m with special characters
  • Field methods
    • 0 = negate adverse text present (calm day)
    • 1 = neither negate key words nor key words present
    • 2 = key words related to adverse weather conditions present, no negate terms present (windy day)
  • Misc flag
    • 0 = no misc flag
    • 1 = bottom indicated
    • 2 = Harmonized value > bottom value
• Unrealistic values:
  • Not using this step. Values greater than 31m are flagged.
• Generate plots with harmonized dataset
  • Updated to match sdd updates
• Aggregate simultaneous records
  • Updates to simultaneous records have been incorporated
• Export
  • Updated to match sdd updates

[mbrousil]

@juandlt-csu Sorry, I haven't had a chance to review yet! Should be able look through this early next week.

[juandlt-csu]

I have made the following updates to the harmonize SDD workflow based on this PR’s comments.

• Minor data prep:
  • Removed taking the absolute value from this step, and instead added a Clean up special characters step before the Clean up MDLs step.
• Clean up special characters:
  • Look for -, =, and * in the original value column and extract the number values.
  • This step flags this data under miscellaneous flag 3 = Special characters present (*, =,-)
• Clean up “greater than” values
  • Remove = from this string search. Now only searches for >
• Clean up approximated values/Gap fill harmonized values with depth columns
  • Simplified the logic for the flagging criteria.
    • Only fill harmonized value if there is Activity depth data or bottom depth data.
    • Prefer to fill in with activity depth data
    • Only prefer bottom depth data if there is bottom language and no negate language
• Gap fill, harmonize, and flag bottom depth data
  • Re-joined the conversion table based on harmonized bottom depth for bottom depth gap fill unit conversion. Previously we had used the conversion column from the harmonized units conversion step under the assumption that these units would be the same.
• Flag
  • Converted adverse text to environmental conditions text. Field flag is now used to determine if there was environmental information stored in the comment columns instead of determining if there were good/bad weather conditions.
• Unrealistic values:
  • Cutting off those values that are greater than 62m (double our cutoff for flagging suspiciously high values)
• General:
  • Made those updates that deal with downstream effects of these changes