Questions to address

[fawda123]

This is a collection of miscellaneous questions we need to address for the package:

• Is there a complete list of Characteristic Name that I can use to check if entries are correct? https://github.com/massbays-tech/MassWateR/blob/d959b4cbbd51ddf7e70f4d6df1afcfedd7e64d5d/R/check_results.R#L46
• Similarly, are there acceptable Result Unit values for each Characteristic Name?
• What about entries in the Relative Depth Category column? It currently checks for Surface, Bottom, and < 1m / 3.3ft https://github.com/massbays-tech/MassWateR/blob/d959b4cbbd51ddf7e70f4d6df1afcfedd7e64d5d/R/check_results.R#L45
• Parameter names in DQO file do not match those in Results file, should they? The names in DQO files were changed to match.
• Do I have the correct activity types subsetted for the completeness checks in qc_completeness? https://github.com/massbays-tech/MassWateR/blob/d959b4cbbd51ddf7e70f4d6df1afcfedd7e64d5d/R/qc_completeness.R#L109

[ben-wetherill]

List of characteristic names - Good question. There is a full list that WQX allows with 5800 entries in it. We should probably select the short list of characteristics that is included in our solution. However, the names must match the WQX list.

Result unit values - Similarly WQX has 360 options in its list. We will need to define our short list, but they must match WQX.

Relative Depth Category - Wee need to think about this.

I don't think the DQO file parameter names must match the Results file. The Results file is designed for WQX. The DQO file needs to be as user friendly as possible. But we will need a mapping of DQO names to WQX names.

I think our full set of allowed activity types is in our results file.

[ben-wetherill]

Here is the logic for the Results file (as of 3/9/22). The activity type mapping table at the bottom is critical.

The Results file now includes a few columns that we hadn’t discussed before – some clarifying info is below:

• There are two different ways (both supported by WQX) that sampling depth can be reported. We will likely limit what depths are incorporated into the tool’s QC and analysis functions (TBD but probably only “surface” or < 1m / 3.3ft), but since users will use the tool to format data and upload to WQX, they are welcome to report it all.
• Within the Result Value column, users can report data that are above quantitation limits (AQL) or below detection limits (BDL) by using those codes. When R sees either code, it should first look to the new column “Quantitation Limit”, and substitute the AQL or BDL with the value listed. If blank, it should look to the DQO table and replace with the appropriate value. We anticipate there may be cases where the limits vary by site, so users need flexibility to manually enter it.
• The QC Reference Value column is where users can enter the QC results from field duplicates, lab duplicates, and lab spikes. It removes the burden of creating a new row for each of these records. However, the tool will need to create those rows for the user in the WQX Formatting package. The “new” row would have the same location, date, time, etc as the original but would have a new Activity Type (e.g. If original row was “Field Msr/Obs” then the new row for the QC data is “Quality Control Field Replicate Msr/Obs”).
• The Result Attribute column is optional and can be used to group results using whatever coding system the users wants, for example, samples that reflect Wet or Dry conditions.

Activity Type mapping: Input Activity Type -------------------------- WQX output new row Activity Type Field Msr/Obs ------------------------------- Quality Control Field Replicate Msr/Obs Sample-Routine ----------------------------- Quality Control Sample-Field Replicate Quality Control Sample-Field Blank ---------- NA Quality Control Sample-Lab Duplicate ------- Quality Control Sample-Lab Duplicate Quality Control Sample-Lab Blank ----------- NA Quality Control Sample-Lab Spike ----------- Quality Control Sample-Reference Sample

[ben-wetherill]

Regarding the activity types in the completeness check... One lab spike activity type is missing ("Quality Control Sample-Reference Sample"), but this will only be in the WQX output file. Also note that the field duplicate activity types will not show up in the Results file. They will only be in the WQX output file. In the code where you are doing sum(), I assume you are getting the count of records not the sum of values. The completeness check is a count of records.

I tried to make some edits directly in the code, but I got a message saying I must be on a branch to make or propose changes. I'm not sure what that means.

[ben-wetherill]

Regarding the Relative Depth Category column... I think the actual name for this column in WQX is "Activity Relative Depth Name", so we should probably change to that. The allowable values are "Surface", "Midwater", "Bottom", and many others. But we are only planning to analyze "Surface". "< 1m / 3.3ft" is not an option for Relative Depth. That was referring to values in the Activity Depth/Height Measure Column. If we see values that are less than 1m or 3.3 ft, then they will be considered the same as "Surface" and included in our analysis. Though, I'm thinking we might need to change this to <=0.5m/2ft, because 1m/3ft is sometimes a non-surface measurement.

[ben-wetherill]

Here is a file of parameter names and mappings, but it may not be the full list yet Parameter Mapping.xlsx .

[carrjill]

@fawda123 standby for an updated Parameter Mapping file.

Re: Relative Depth, users' data can include Surface, Bottom, Near Bottom and Midwater. As Ben said the tool will most likely use data that are either (a) logged as Surface in the Relative Depth column, or (b) recorded as < 1m / 3.3ft in the Activity Depth column.

Re: DQO file parameter names, I do think these should match the results, especially now that we're providing simplified options. I can't see how Marcus would map every possible user-provided format to the Results.

@ben-wetherill We should keep the 1m / 3.3ft depth limit because it is the standard for EPA and DPH bacteria sampling programs. Renaming the Relative Depth column now to match WQX is fine, or we can do it in the WQX template package step which will involve a lot of additional column naming.

[ben-wetherill]

Here is an updated Parameter Mapping file. You can probably ignore the Parameter Group column for now. That is primarily just to help us organize our thinking, but it might come in handy when we are thinking about analysis graphs. Parameter Mapping_4-29-22.xlsx

[fawda123]

Thanks @ben-wetherill and @carrjill, I'm trying to digest all of the comments in this thread. In hindsight, one question/comment should match to one issue so we're not tracking multiple problems in one thread. That's my fault for adding multiple questions in the original issue.

For the list of characteristic names, I've added a new dataset to the package that uses the Parameter Mapping file from 4/29 that Ben posted in his last message. The parameters in the "Simple Parameter" column are now used as the common set that are checked on data import of the results file. I also went through and changed the results and DQO files (using the new ones Ben sent via email on 4/26) to have consistent characteristic names that match the parameter mapping file. If I'm understanding correctly, we are now allowing our template to use these simple names with the hopes that users will follow suit when they're importing their own data. I'll have to map the names to the WQX format for any data intended for upload after going through the package workflow. Will do that later...

I also added "Gage" and "Air Temp" to the parameter mapping file because these variables were included in the original results file. So, now these are "acceptable" entries from our master list.

[fawda123]

I also added some additional checks in check_results() for the result unit values. Importing the results file will also now check for:

• Missing values in the Result Unit column (excluding pH)
• More than one Result Unit entered for each parameter. We may want to chat about whether this is approriate, this is more a check for any downstream analyses, but I realize groups may have different units for the same parameter.
• The Result Units match those in the parameter mapping file. I had to change the entries in the original results file so the checks would pass. We should chat about whether or not this is appropriate, i.e., are we expecting users to follow the conventions in the parameter mapping file for all results and DQO inputs.

[ben-wetherill]

Clarification on parameter mapping - Jill and I were thinking that the mapping between Simple and WQX parameters should be just like synonyms. In other words, in any file, the users should be able to use either form and R should convert to the preferred form for each output. So, users' input files (Results and DQOs) would have either Simple or WQX names (no other names allowed), depending on user preference. This could even be a mixture, meaning users might use Simple names in the DQO files and WQX names in the results file. R should think of them as synonyms. Our QC Review output and analytical outputs should probably use the Simple names, and our WQX output must use the WQX names. Is that reasonable?

Regarding Gage and Air Temp - I sort of left these out on purpose, because these are in that gray area of parameters that we might load into WQX but we don't track them for data quality. They are low priority and probably useless to DEP, but OARS includes them in the WQX file just because they always have. My guess is that many groups may have a parameter or two like this. It wouldn't be hard to argue to remove them completely from the WQX upload, but I'm thinking it might be simplest for us to just allow them to pass through. Maybe it's okay not to have them in the DQO file and just have a warning. What do you think?

Regarding Result Units - I think your checks are good. I'm thinking it is reasonable to expect users to stick to a single unit of measure for each parameter. For example, if their DO DQO is in mg/L, then their results file can't be in ug/L. If we didn't require this then we would need to maintain all of the conversion factors. Some are easy, but some are not.

[ben-wetherill]

I guess this issue can be closed, right?