Meeting 8 November 2024

COMPASS-DOE / sensor-data-pipeline

Sensor data workflows and processing scripts

MIT License

4 stars 0 forks source link

With @bpbond @stephpenn1 @roylrich

Roy

A generalized approach to sensor data, applicable to other parts of his SERC work
New techs have good range of technical/programming skills
New NERRS call

Steph

New ESS-DIVE project - significant potential overlaps with above - generalizing pipeline, updating SR reporting format, versioning
Roy notes could use SMARTX, B4WARM, etc. datasets for testing; and he could devote some tech time for this
Deep soil warming synthesis with Margaret Torn, Peter Reich, Michael Schmidt
ESS-DIVE workshop next week
Paper almost ready for submission

Ben

Open PR #216 re request for injecting raw instrument data into processing pipeline
Roy points out that timestamps are a problem - instruments may not (like not) be on 'correct' loggernet time

v1.2

First publish TEMPEST data to ESS-DIVE, hopefully by AGU
Release v1-2 in January (?) comprising data through end of 2024
And then a midwinter upgrade, potentially with bigger changes, in preparation for field season
At some point, an orientation meeting for Liz or whomever Roy decides

	Approach	Pros	Cons
0	Don't support	Very easy	Puts everything on users; data unavailable for non-COMPASS folks
1	Just flag non-datalogger data	Easy for us	Users might miss documentation warnings about this
2	Adjust via technician-recorded instrument offset	Fairly easy for us; offset would be recorded in existing Exo and Troll inventory files	A pain for techs; potentially fragile
3	Document offset	Fairly easy for us; users can decide if they want to adjust based on reported instrument timestamps	Requires change in loggernet code; computationally non-trivial for users
4	Figure out offset algorithmically	If datalogger reports instrument timestamp, we can (probably) reconstruct offsets for orphaned data	Requires change in loggernet code; computationally non-trivial on the L1 processing side

Approach

Pros

Cons

Don't support

Very easy

Puts everything on users; data unavailable for non-COMPASS folks

Just flag non-datalogger data

Easy for us

Users might miss documentation warnings about this

Adjust via technician-recorded instrument offset

Fairly easy for us; offset would be recorded in existing Exo and Troll inventory files

A pain for techs; potentially fragile

Document offset

Fairly easy for us; users can decide if they want to adjust based on reported instrument timestamps

Requires change in loggernet code; computationally non-trivial for users

Figure out offset algorithmically

If datalogger reports instrument timestamp, we can (probably) reconstruct offsets for orphaned data

Requires change in loggernet code; computationally non-trivial on the L1 processing side

COMPASS-DOE / sensor-data-pipeline