leppott / ContDataQC

Quality control checks on continuous data. Example data is from a HOBO data logger with 30 minute intervals.
https://leppott.github.io/ContDataQC/
MIT License
20 stars 10 forks source link

Aggregate to check for duplicates #112

Open leppott opened 3 years ago

leppott commented 3 years ago

Is your feature request related to a problem? Please describe. Users sometimes have files with overlapping time periods. Current version of Aggregate simply appends files resulting in duplicates.

Describe the solution you'd like Not sure see alternatives.

Describe alternatives you've considered

  1. Alert users so can modify input data. Maybe stop if datetimes overlap an give info in stop message.
  2. Use Unique to get rid of duplicates. But only if exact. If any differences then still have dup records. Would still need a check.

Additional context Users should have data stored only in one file but many users have the same date range in multiple files. Current "append" leads to extra time figuring out what went wrong and which data to retain.