leppott / ContDataQC

Quality control checks on continuous data. Example data is from a HOBO data logger with 30 minute intervals.
https://leppott.github.io/ContDataQC/
MIT License
20 stars 10 forks source link

Add tracking feature for logger deployments #107

Closed leppott closed 3 years ago

leppott commented 3 years ago

Is your feature request related to a problem? Please describe. Once aggregate files no long know start/stop dates of logger deployments. Need a way to track.

Describe the solution you'd like Add a column during the QC process to populate with start and end of deployments.

Describe alternatives you've considered Only add "start" and "stop" to the column. Easier to filter. Having the date range would add too much to the file sizes.

Additional context If track this information can later on look for differences between deployments.

leppott commented 3 years ago

Modified ContDataQC and the subfunctions QC and QC.file to accept a new parameter.

fun.AddDeployCol
Boolean for adding column name. Default = FALSE. Can be customized in config.R; ContData.env$myName.LoggerDeployment.

Populated with NA except for first row (start) and last row (end).

image

leppott commented 3 years ago

Each output file has "start" and "end" added. If trim any results will have to modify the start and end comments.

image

And shows up after the "aggregation" step.

image

DST fix keeps only 1st record if more than one per date/time. Added QC check after the DST fix to ensure the last row has "end" in the logger deployment column.

leppott commented 3 years ago

Works for both QC and QC.file.

leppott commented 3 years ago

Some users missing the "end" value.

leppott commented 3 years ago

Tracked down to files with extra blank rows.

Users "trimmed" the file after the QC step and before the "Aggregate" step.

leppott commented 3 years ago

Still uses Date.Time to look for NA values. But the script will fail prior to this point if the Date Time field is NA. It fails when checking the date format.

leppott commented 3 years ago

Commit v2.0.5.9080 https://github.com/leppott/ContDataQC/commit/5ec8e97566cde970530cba5b82351d58b15b68e0