search

COMPASS-DOE / sensor-data-pipeline

Sensor data workflows and processing scripts
MIT License
4 stars 0 forks source link

Comprehensive list of data cleanup for BBL and RR #111

Open bpbond opened 10 months ago

bpbond commented 10 months ago

Consolidates various issues. Assigning to @roylrich as these have all been fixed for 0.9.

Spurious first-day GCW TerosTable files #110

These two files should be deleted from the raw data archive; it looks like there was some initial confusion on the naming, and after this it's uniformly TerosTableC. I am excluding them from our data release.

COMPASS_GCW_W_431_TerosTableA_20221109000108.dat COMPASS_GCW_W_431_TerosTableB_20221109000111.dat

Spurious first-day SWH files with bad dates #88

Initial SWH data has bad dates (years 2024, 2032, etc). This is because loggers are placed for a while before the instrumentation is attached. As soon as they receive data, the time shifts to correct time in this file. The time is coordinated by server and so we did not have coms at transition until cell was installed later on.

Redox loggers #80

The TEMPEST redox loggers were temporary and I believe will be handled separately. These include CR1000_2 and RedoxMU3_83.

Spurious first-day OWC date problems

Same as SWH issue above, 1980/1981 dates in these files:

Compass_OWC_TR_322_SapflowB_20220824143828.dat Compass_OWC_TR_322_TerosTableB_20220824143828.dat Compass_OWC_W_321_TerosTableC_20220824143831.dat Compass_OWC_W_321_ClimaVue50_15min_20220824143830.dat Compass_OWC_W_321_ClimaVue50_24hr_20220824143831.dat

bpbond commented 10 months ago

TEMPEST 2022-07 corrupted dates

Some initial rows with year 2052 timestamps in

PNNL_11_sapflow_20220712000003.dat PNNL_11_Terosdata_20220712000004.dat

GCW 2022-11 bad dates

Some initial rows with year 1985 timestamps in

COMPASS_GCW_W_431_ClimaVue50_24hr_20221109000101.dat COMPASS_GCW_W_431_ClimaVue50_15min_20221109000057.dat COMPASS_GCW_W_431_TerosTableC_20221109000115.dat

GWI 2022-08 bad dates

Some initial rows with year 2004 timestamps in

Compass_GWI_TR_412_WaterLevel600B_20220824143823.dat Compass_GWI_TR_412_SapflowB_20220824143823.dat COMPASS_GCW_W_431_ClimaVue50_24hr_20221109000101.dat

TMP 2022-07 bad dates

Some rows with year 2052 timestamps in

PNNL_11_Terosdata_20220712000004.dat PNNL_11_sapflow_20220712000003.dat

bpbond commented 4 months ago

SWH 2024-06-01 bad dates

Compass_SWH_UP_423_CheckTable_20240601000508.dat bad dates

Jumps to the future 

"2024-05-15 15:00:00",42911,11.89,26.31,"LOW","OKAY",0,-1,0,-1,-1,0,-1,-1,0,0,0,0,0,0,"COMPASS_SWH_UP",423
"2024-08-29 02:15:00",0,11.95,26.27,"LOW","OKAY",0,-1,0,-1,0,0,-1,0,0,0,0,0,0,0,"COMPASS_SWH_UP",423

This is true for all of the SWH_UP423..._20240601 files