search

ianjonsen / ArgosQC

Quality Control Process for the Integrated Marine Observing System's SMRU SRDL-CTD tag data
MIT License
1 stars 0 forks source link

Generalization work part 2 #2

Open ianjonsen opened 1 month ago

ianjonsen commented 1 month ago

The pkg needs to be generalized to handle data from Wildlife Computers Splash/Mk10/etc tags. As the data file structures greatly differ between WC & SMRU, careful planning is required here.

Options:

  1. a parallel set of functions written to handle WC server access & data.

  2. a driver layer be constructed to generally handle tag data from various servers & with various file structures & translate the data files to a common format recognizable to ArgosQC. eg. WC tag data is coerced to a SMRU-like data file structure & format on which generalized (per part 1) ArgosQC functions operate. the driver layer would then translate outputs back to the original WC format.

  3. other approaches??

jdpye commented 1 month ago

I would say the concept of creating a 'driver' layer is appropriate for future-proofing. The 'driver' in this case would be descriptors for each data source type defining where to find all the critical data columns across the various data sources, and there may still be some data formatting code per-type or per filetype required to get the heterogeneous data into the beginning of the QC process. I started doing this over in the RT satellite to OBIS publication repository but it'd be much more appropriate to do it at this phase and let the output of ArgosQC be suitable input to that publication step.