Open jr3cermak opened 1 year ago
I can now confirm that dbdreader can be used to reproduce dba_merge
. This code snippet will take the two files and merge the data via pandas to form a dataframe similar to that product to dba_merge
.
At this stage, the merged dataframe can be tossed into the parquet framework.
# Invoke dbdreader on sbd file
sbdf = "%s%s" % (os.path.join("binary", bn), ".sbd")
sbd = dbdreader.DBD(sbdf, cacheDir=cache_dir, skip_initial_line=False)
# Invoke dbdreader on tbd file
tbdf = "%s%s" % (os.path.join("binary", bn), ".tbd")
tbd = dbdreader.DBD(tbdf, cacheDir=cache_dir, skip_initial_line=False)
# Compare columns
dbd_columns = sbd.parameterNames + tbd.parameterNames
dbd_columns.sort()
slo_columns = list(gu.columns)
slo_columns.sort()
assert set(dbd_columns) == set(slo_columns), \
"Slocum merged and dbdreader columns do not match"
# Simulate: dba_merge
sbd_data = pd.DataFrame()
for col in sbd.parameterNames:
tm, data = sbd.get(col, return_nans=True, decimalLatLon=False)
sbd_data[col] = data
sbd_data = sbd_data.set_index('m_present_time')
sbd.close()
tbd_data = pd.DataFrame()
for col in tbd.parameterNames:
tm, data = tbd.get(col, return_nans=True, decimalLatLon=False)
tbd_data[col] = data
if col == 'sci_m_present_time':
# Create a duplicate column for merging
tbd_data['m_present_time'] = data
tbd_data = tbd_data.set_index('m_present_time')
tbd.close()
dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
dbd_data = dbd_data.reset_index()
A quick comparison of 9 separate files and values obtained via dba_merge
and dbdreader
:
$ python cmpData.py
===> ascii/unit_507_2022_078_0_0_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 4.010009767796419e-06 4.94964599617731e-05
m_battery 3.4179687524726887e-07 4.945220947227824e-05
m_battpos 2.682209010451686e-09 4.988937377659042e-07
m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
m_depth 0.0 0.0004963378906381877
m_depth_rate 0.0 4.78054046626486e-07
m_gps_lat 0.0 0.0
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 7.615256309660007e-10 4.785308838073377e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
m_lon 0.0 4.547473508864641e-11
m_pitch 9.338641165923534e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
m_water_depth 0.0 0.0004663085937579581
m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
sci_echodroid_aggindex 0.0 4.936542510880493e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.97009277324878e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 4.342842102156297e-06
sci_echodroid_sa 0.0 4.931030273525039e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.051462113930163e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1871337912694457e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_1_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
m_battery 3.0398559569988493e-05 3.076019287107101e-05
m_battpos 4.100151062003121e-07 4.100151062003121e-07
m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
m_depth 0.0 0.0
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
m_heading 9.155273437144729e-07 9.155273437144729e-07
m_lat 1.8189894035458565e-12 1.8189894035458565e-12
m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
m_lon 4.547473508864641e-11 4.547473508864641e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 7.152557373046875e-07 4.5299530029296875e-06
m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
m_roll 1.9259738928689707e-08 1.9259738928689707e-08
m_vacuum 4.727478026822496e-06 2.901611328098852e-05
m_water_depth 0.00015649414064000666 0.00015649414064000666
m_water_vx 2.952008247569804e-08 2.952008247569804e-08
m_water_vy 3.911998271849404e-08 3.911998271849404e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_2_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 4.674072265586915e-05
m_battery 4.711914062482947e-06 4.655303955125589e-05
m_battpos 2.6850545406867377e-09 4.959058761688784e-07
m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
m_depth 0.0 0.0004937744140676159
m_depth_rate 0.0 4.837455749584496e-07
m_gps_lat 0.0 0.0
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 9.047412874441108e-09 4.929962158506385e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
m_lon 3.637978807091713e-12 4.9112713895738125e-11
m_pitch 1.5230512619320014e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 2.639770535495245e-08 1.972961425700248e-05
m_water_depth 0.0 0.0004902343749790816
m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
sci_echodroid_aggindex 0.0 3.8234138488579816e-07
sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
sci_echodroid_eqarea 0.0 4.992523193436682e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.047749638507597e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1894226092934446e-07
sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
sci_water_temp 0.0 2.3803710913483656e-07
===> ascii/unit_507_2022_078_0_3_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 5.052185057863312e-06
m_battery 1.0815429687838218e-05 4.9467468262065495e-05
m_battpos 1.6136550906598046e-07 4.100151062003121e-07
m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
m_depth 0.0 1.7530155181633145e-08
m_depth_rate 0.0 0.0
m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
m_heading 3.634643555017192e-07 3.634643555017192e-07
m_lat 2.7284841053187847e-12 2.7284841053187847e-12
m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
m_lon 4.547473508864641e-11 4.547473508864641e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 4.76837158203125e-07 3.814697265625e-06
m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
m_roll 3.993892669362786e-08 3.993892669362786e-08
m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
m_water_depth 6.762695312545475e-05 6.762695312545475e-05
m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_4_sbd.dat
c_wpt_lat 0.0 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 4.9743652340339395e-05
m_battery 4.3533325193578776e-06 4.867706298838925e-05
m_battpos 2.24747955792845e-09 4.659767150749783e-07
m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
m_depth 0.0 0.0004984130859497782
m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 5.4016113359978135e-08 4.980773925744586e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
m_lon 1.8189894035458565e-12 4.9112713895738125e-11
m_pitch 7.615256309660007e-10 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
m_water_depth 0.0 0.0004943847656306843
m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
sci_echodroid_aggindex 0.0 4.7902107236375e-07
sci_echodroid_ctrmass 0.0 4.894409179456716e-05
sci_echodroid_eqarea 0.0 4.985046386707381e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.7564697268853706e-05
sci_flbbcd_bb_units 0.0 4.049237966631536e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_5_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 2.012023925956896e-05 2.012023925956896e-05
m_battery 3.368377685575297e-05 4.551544189368428e-05
m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
m_depth 0.0 1.7530155181633145e-08
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 0.0 0.0
m_heading 1.1898803711218875e-06 1.1898803711218875e-06
m_lat 9.094947017729282e-13 9.094947017729282e-13
m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
m_lon 3.8198777474462986e-11 3.8198777474462986e-11
m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
m_present_time 0.0 4.5299530029296875e-06
m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
m_roll 2.4125957495679806e-08 2.4125957495679806e-08
m_vacuum 4.447937012486136e-06 3.832702636685781e-05
m_water_depth 0.0003625488281215894 0.0003625488281215894
m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
m_water_vy 4.524588585047562e-08 4.524588585047562e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_6_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 6.713867186647349e-07 4.707641601697787e-05
m_battery 6.86645513781059e-08 4.9478149414028394e-05
m_battpos 6.442186237590186e-11 4.7547626494393036e-07
m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
m_depth 0.0 0.0005000000000023874
m_depth_rate 0.0 4.955315589943998e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 5.546569825654046e-08 4.954223632669397e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
m_lon 1.8189894035458565e-12 4.3655745685100555e-11
m_pitch 1.5230512619320014e-09 4.966812133799792e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
m_roll 2.153873440713383e-09 4.1464805602964283e-07
m_vacuum 2.639770535495245e-08 4.555969238317914e-05
m_water_depth 0.0 0.00042846679687613687
m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
m_water_vy 1.688668727767295e-08 4.524588585047562e-08
sci_echodroid_aggindex 0.0 4.939737320019955e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.858703613308535e-05
sci_echodroid_inertia 0.0 0.0004923706054711374
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.7564697268853706e-05
sci_flbbcd_bb_units 0.0 4.055169820814733e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_7_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 3.096618652520533e-05 3.096618652520533e-05
m_battery 2.7200317383346828e-05 4.9478149414028394e-05
m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
m_depth 0.0 0.0
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 0.0 0.0
m_heading 1.3704681398607477e-06 1.3704681398607477e-06
m_lat 1.8189894035458565e-12 1.8189894035458565e-12
m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
m_lon 4.001776687800884e-11 4.001776687800884e-11
m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
m_present_time 4.76837158203125e-07 4.76837158203125e-06
m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
m_roll 4.1464805602964283e-07 4.1464805602964283e-07
m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
m_water_depth 0.00015234375001682565 0.00015234375001682565
m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
m_water_vy 1.688668727767295e-08 1.688668727767295e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_8_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 6.713867151120212e-07 4.933776855509109e-05
m_battery 3.372192374229144e-07 4.8551940917462844e-05
m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
m_depth 0.0 0.0004981689453131821
m_depth_rate 0.0 4.955315589943998e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 2.3651121061618596e-09 4.9253082274880455e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
m_lon 1.8189894035458565e-12 4.9112713895738125e-11
m_pitch 2.155303935325037e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
m_roll 2.153873440713383e-09 4.1464805602964283e-07
m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
m_water_depth 0.0 0.0004304199218836402
m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
m_water_vy 7.65342712469308e-09 4.458808899182598e-08
sci_echodroid_aggindex 0.0 4.809398650995611e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.702453613347757e-05
sci_echodroid_inertia 0.0 0.0004923706054711374
sci_echodroid_propocc 0.0 3.5113906859951527e-06
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.053686261228789e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3803710913483656e-07
===> ascii/unit_507_2022_078_0_9_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 3.840332031401772e-05 3.840332031401772e-05
m_battery 5.773925781937805e-06 4.3586730956945985e-05
m_battpos 6.470108027123445e-08 6.470108027123445e-08
m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
m_depth 0.0 3.758859635008549e-09
m_depth_rate 0.0 0.0
m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
m_gps_lon 0.0 0.0
m_heading 2.923889160122428e-06 2.923889160122428e-06
m_lat 9.094947017729282e-13 9.094947017729282e-13
m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
m_lon 4.3655745685100555e-11 4.3655745685100555e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 9.5367431640625e-07 2.6226043701171875e-06
m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
m_roll 3.993892669362786e-08 3.993892669362786e-08
m_vacuum 2.748107910122144e-06 9.059143065925923e-06
m_water_depth 0.0002636718750181899 0.0002636718750181899
m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
m_water_vy 4.458808899182598e-08 4.458808899182598e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
sci_water_temp 0.0 0.0
Maximum deviation: 0.0005000000000023874
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py | less
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py
===> ascii/unit_507_2022_078_0_0_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 4.010009767796419e-06 4.94964599617731e-05
m_battery 3.4179687524726887e-07 4.945220947227824e-05
m_battpos 2.682209010451686e-09 4.988937377659042e-07
m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
m_depth 0.0 0.0004963378906381877
m_depth_rate 0.0 4.78054046626486e-07
m_gps_lat 0.0 0.0
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 7.615256309660007e-10 4.785308838073377e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
m_lon 0.0 4.547473508864641e-11
m_pitch 9.338641165923534e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
m_water_depth 0.0 0.0004663085937579581
m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
sci_echodroid_aggindex 0.0 4.936542510880493e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.97009277324878e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 4.342842102156297e-06
sci_echodroid_sa 0.0 4.931030273525039e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.051462113930163e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1871337912694457e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_1_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
m_battery 3.0398559569988493e-05 3.076019287107101e-05
m_battpos 4.100151062003121e-07 4.100151062003121e-07
m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
m_depth 0.0 0.0
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
m_heading 9.155273437144729e-07 9.155273437144729e-07
m_lat 1.8189894035458565e-12 1.8189894035458565e-12
m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
m_lon 4.547473508864641e-11 4.547473508864641e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 7.152557373046875e-07 4.5299530029296875e-06
m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
m_roll 1.9259738928689707e-08 1.9259738928689707e-08
m_vacuum 4.727478026822496e-06 2.901611328098852e-05
m_water_depth 0.00015649414064000666 0.00015649414064000666
m_water_vx 2.952008247569804e-08 2.952008247569804e-08
m_water_vy 3.911998271849404e-08 3.911998271849404e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_2_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 4.674072265586915e-05
m_battery 4.711914062482947e-06 4.655303955125589e-05
m_battpos 2.6850545406867377e-09 4.959058761688784e-07
m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
m_depth 0.0 0.0004937744140676159
m_depth_rate 0.0 4.837455749584496e-07
m_gps_lat 0.0 0.0
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 9.047412874441108e-09 4.929962158506385e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
m_lon 3.637978807091713e-12 4.9112713895738125e-11
m_pitch 1.5230512619320014e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 2.639770535495245e-08 1.972961425700248e-05
m_water_depth 0.0 0.0004902343749790816
m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
sci_echodroid_aggindex 0.0 3.8234138488579816e-07
sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
sci_echodroid_eqarea 0.0 4.992523193436682e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.047749638507597e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1894226092934446e-07
sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
sci_water_temp 0.0 2.3803710913483656e-07
===> ascii/unit_507_2022_078_0_3_sbd.dat
c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 5.052185057863312e-06
m_battery 1.0815429687838218e-05 4.9467468262065495e-05
m_battpos 1.6136550906598046e-07 4.100151062003121e-07
m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
m_depth 0.0 1.7530155181633145e-08
m_depth_rate 0.0 0.0
m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
m_heading 3.634643555017192e-07 3.634643555017192e-07
m_lat 2.7284841053187847e-12 2.7284841053187847e-12
m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
m_lon 4.547473508864641e-11 4.547473508864641e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 4.76837158203125e-07 3.814697265625e-06
m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
m_roll 3.993892669362786e-08 3.993892669362786e-08
m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
m_water_depth 6.762695312545475e-05 6.762695312545475e-05
m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_4_sbd.dat
c_wpt_lat 0.0 9.094947017729282e-13
c_wpt_lon 0.0 0.0
m_altitude 5.052185057863312e-06 4.9743652340339395e-05
m_battery 4.3533325193578776e-06 4.867706298838925e-05
m_battpos 2.24747955792845e-09 4.659767150749783e-07
m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
m_depth 0.0 0.0004984130859497782
m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 5.4016113359978135e-08 4.980773925744586e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
m_lon 1.8189894035458565e-12 4.9112713895738125e-11
m_pitch 7.615256309660007e-10 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
m_roll 2.153873440713383e-09 4.683246612735559e-08
m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
m_water_depth 0.0 0.0004943847656306843
m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
sci_echodroid_aggindex 0.0 4.7902107236375e-07
sci_echodroid_ctrmass 0.0 4.894409179456716e-05
sci_echodroid_eqarea 0.0 4.985046386707381e-05
sci_echodroid_inertia 0.0 0.00048474121092567657
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.7564697268853706e-05
sci_flbbcd_bb_units 0.0 4.049237966631536e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_5_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 2.012023925956896e-05 2.012023925956896e-05
m_battery 3.368377685575297e-05 4.551544189368428e-05
m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
m_depth 0.0 1.7530155181633145e-08
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 0.0 0.0
m_heading 1.1898803711218875e-06 1.1898803711218875e-06
m_lat 9.094947017729282e-13 9.094947017729282e-13
m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
m_lon 3.8198777474462986e-11 3.8198777474462986e-11
m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
m_present_time 0.0 4.5299530029296875e-06
m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
m_roll 2.4125957495679806e-08 2.4125957495679806e-08
m_vacuum 4.447937012486136e-06 3.832702636685781e-05
m_water_depth 0.0003625488281215894 0.0003625488281215894
m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
m_water_vy 4.524588585047562e-08 4.524588585047562e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_6_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 6.713867186647349e-07 4.707641601697787e-05
m_battery 6.86645513781059e-08 4.9478149414028394e-05
m_battpos 6.442186237590186e-11 4.7547626494393036e-07
m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
m_depth 0.0 0.0005000000000023874
m_depth_rate 0.0 4.955315589943998e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 5.546569825654046e-08 4.954223632669397e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
m_lon 1.8189894035458565e-12 4.3655745685100555e-11
m_pitch 1.5230512619320014e-09 4.966812133799792e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
m_roll 2.153873440713383e-09 4.1464805602964283e-07
m_vacuum 2.639770535495245e-08 4.555969238317914e-05
m_water_depth 0.0 0.00042846679687613687
m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
m_water_vy 1.688668727767295e-08 4.524588585047562e-08
sci_echodroid_aggindex 0.0 4.939737320019955e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.858703613308535e-05
sci_echodroid_inertia 0.0 0.0004923706054711374
sci_echodroid_propocc 0.0 9.159469605002357e-07
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.7564697268853706e-05
sci_flbbcd_bb_units 0.0 4.055169820814733e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3727417008956309e-07
===> ascii/unit_507_2022_078_0_7_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 3.096618652520533e-05 3.096618652520533e-05
m_battery 2.7200317383346828e-05 4.9478149414028394e-05
m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
m_depth 0.0 0.0
m_depth_rate 0.0 0.0
m_gps_lat 0.0 0.0
m_gps_lon 0.0 0.0
m_heading 1.3704681398607477e-06 1.3704681398607477e-06
m_lat 1.8189894035458565e-12 1.8189894035458565e-12
m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
m_lon 4.001776687800884e-11 4.001776687800884e-11
m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
m_present_time 4.76837158203125e-07 4.76837158203125e-06
m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
m_roll 4.1464805602964283e-07 4.1464805602964283e-07
m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
m_water_depth 0.00015234375001682565 0.00015234375001682565
m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
m_water_vy 1.688668727767295e-08 1.688668727767295e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
sci_water_temp 0.0 0.0
===> ascii/unit_507_2022_078_0_8_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 6.713867151120212e-07 4.933776855509109e-05
m_battery 3.372192374229144e-07 4.8551940917462844e-05
m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
m_depth 0.0 0.0004981689453131821
m_depth_rate 0.0 4.955315589943998e-07
m_gps_lat 0.0 9.094947017729282e-13
m_gps_lon 0.0 1.8189894035458565e-12
m_heading 2.3651121061618596e-09 4.9253082274880455e-06
m_lat 0.0 4.547473508864641e-12
m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
m_lon 1.8189894035458565e-12 4.9112713895738125e-11
m_pitch 2.155303935325037e-09 4.96858596821248e-07
m_present_time 0.0 5.0067901611328125e-06
m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
m_roll 2.153873440713383e-09 4.1464805602964283e-07
m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
m_water_depth 0.0 0.0004304199218836402
m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
m_water_vy 7.65342712469308e-09 4.458808899182598e-08
sci_echodroid_aggindex 0.0 4.809398650995611e-07
sci_echodroid_ctrmass 0.0 4.931030273525039e-05
sci_echodroid_eqarea 0.0 4.702453613347757e-05
sci_echodroid_inertia 0.0 0.0004923706054711374
sci_echodroid_propocc 0.0 3.5113906859951527e-06
sci_echodroid_sa 0.0 4.862060546884095e-05
sci_echodroid_sv 0.0 4.931030273525039e-05
sci_flbbcd_bb_units 0.0 4.053686261228789e-09
sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
sci_m_present_time 0.0 5.0067901611328125e-06
sci_rinkoii_do 0.0 3.784179682497779e-06
sci_water_cond 0.0 1.1917114273174434e-07
sci_water_pressure 0.0 9.460449206244448e-07
sci_water_temp 0.0 2.3803710913483656e-07
===> ascii/unit_507_2022_078_0_9_sbd.dat
c_wpt_lat 0.0 0.0
c_wpt_lon 0.0 0.0
m_altitude 3.840332031401772e-05 3.840332031401772e-05
m_battery 5.773925781937805e-06 4.3586730956945985e-05
m_battpos 6.470108027123445e-08 6.470108027123445e-08
m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
m_depth 0.0 3.758859635008549e-09
m_depth_rate 0.0 0.0
m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
m_gps_lon 0.0 0.0
m_heading 2.923889160122428e-06 2.923889160122428e-06
m_lat 9.094947017729282e-13 9.094947017729282e-13
m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
m_lon 4.3655745685100555e-11 4.3655745685100555e-11
m_pitch 1.534729003882518e-07 1.534729003882518e-07
m_present_time 9.5367431640625e-07 2.6226043701171875e-06
m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
m_roll 3.993892669362786e-08 3.993892669362786e-08
m_vacuum 2.748107910122144e-06 9.059143065925923e-06
m_water_depth 0.0002636718750181899 0.0002636718750181899
m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
m_water_vy 4.458808899182598e-08 4.458808899182598e-08
sci_echodroid_aggindex 0.0 0.0
sci_echodroid_ctrmass 0.0 0.0
sci_echodroid_eqarea 0.0 0.0
sci_echodroid_inertia 0.0 0.0
sci_echodroid_propocc 0.0 0.0
sci_echodroid_sa 0.0 0.0
sci_echodroid_sv 0.0 0.0
sci_flbbcd_bb_units 0.0 0.0
sci_flbbcd_cdom_units 0.0 0.0
sci_flbbcd_chlor_units 0.0 0.0
sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
sci_rinkoii_do 0.0 0.0
sci_water_cond 0.0 0.0
sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
sci_water_temp 0.0 0.0
Maximum deviation: 0.0005000000000023874
The deviations occur as dba_merge
does not pass through the full precision of the float values when the conversion to ascii happens. As I have mentioned in the past, we need to full precision from the dbdreader to extract the embedded echograms in the echometrics data.
The code will have to be refactored a bit to handle both pairs of files. The method should remain the same. Once that is written, the convertDbds.sh
can go away with the ascii conversion step.
Or convert the ascii reader using the above to parquet and then adapt the netcdf part to pull from parquet instead of ascii.
Yes, will make this a gradual transition. Added a hook to extra_kwargs
to enable utilization of dbdreader/parquet so both pathways will continue to exist for a little while.
deployment.json:
{
"glider": "unit_507",
"trajectory_date": "20220212T0000",
"filters": {
"tsint": 20,
"filter_z": 1,
"filter_time": 5,
"filter_points": 5,
"filter_distance": 1
},
"extra_kwargs": {
"enable_parquet": true,
"echograms": {
"enable_nc": true,
"enable_ascii": true,
"enable_image": true,
"plot_type": "profile",
"plot_cmap": "ek80",
"svb_thresholds": [-10,-20,-25,-30,-35,-45,-55],
"svb_limits": [-10.0, -55.0],
"echogram_range_bins": 20,
"echogram_range": -60.0,
"echogram_range_units": "meters"
}
},
"attributes": {
"acknowledgement": "This work was supported by funding from NOAA/IOOS/AOOS.",
"comment": "",
Enabling parquet also assumes dbdreader will also be used.
$ pytest -k TestEchoMetricsFive
====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0 -- /home/cermak/miniconda3/envs/gutils_py3_9/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/cermak/src/GUTILS, configfile: setup.cfg
plugins: anyio-3.6.2
collected 36 items / 35 deselected / 1 selected
gutils/tests/test_slocum.py::TestEchoMetricsFive::test_echogram 2023-06-17 14:34:08,422 - gutils.slocum - INFO - PARQUET enabled deployment
2023-06-17 14:34:08,423 - gutils.slocum - INFO - Creating echogram profile images with colormap ek80
2023-06-17 14:34:23,540 - gutils.slocum - INFO - Converted unit_507-2022-078-0-0.sbd,unit_507-2022-078-0-0.tbd to unit_507_2022_078_0_0_sbd.dat
Now to bolt on the decoding code...
... and push certain things over to the nc processing side.
Getting parquet tables to store datetime[ns, UTC] was giving me a problem initially. This gets us to the goal of parquet tables and nc/xarray and preserving the seconds since 1990-0-0 units.
Once the merge happens:
dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
dbd_data = dbd_data.reset_index()
drop sci_m_present_time
and rename m_present_time
to time
# This works for xarray and parquet
dbd_data['time'] = [datetime.datetime.fromtimestamp(tm, tz=datetime.timezone.utc) for tm in dbd_data['time'].values]
# Convert time to datetime and preserve fractional seconds
# NOTE: works for xarray; not for parquet
#dbd_data['time'] = pd.to_datetime(dbd_data['time'], utc=True, unit='s')
# Make time the index
dbd_data = dbd_data.set_index('time')
# Write files to storage
# write netcdf
fn_xr = "%s%s" % (bn, ".nc")
ds = dbd_data.to_xarray()
ds['time'] = ds['time'].astype('datetime64[ns]')
encodings = {}
for dvar in ds.data_vars:
encodings[dvar] = {"_FillValue": None}
encodings.update({
"time": {
"_FillValue": None,
"calendar": "standard",
"units": "seconds since 1990-01-01T00:00:00Z"
}
})
ds.to_netcdf(fn_xr, encoding = encodings)
# write parquet
fn_pq = "%s%s" % (bn, ".parquet")
dbd_data.to_parquet(fn_pq)
# Verify what we wrote can be read back in
# read netcdf
#nc_ver = xr.open_dataset(fn_xr)
# NOTE: initial checks showed that original values differ using xarray
# read parquet
#pq_ver = pq.read_table(fn_pq)
Instead of *.dat files, there will be .parquet files in the ascii directory for the nc side to utilize with the flag. Temporary until things get fully moved about.
As expected, picking up extra resolution by utilizing dbdreader and parquet for processing.
Original m_depth and m_altitude via dba_merge
:
m_altitude = _, _, _, _, _, 24.5446, _, _, _, _, _, 21.6618, _, _, _,
19.8181, _, _, _, _, _ ;
m_depth = _, _, 222.494995117188, _, _, 224.837997436523, _, _, _,
227.069000244141, _, _, _, 229.412002563477, _, _, _, _, _,
231.559997558594, _ ;
via dbdreader
and parquet
:
m_altitude = _, _, _, _, _, 24.5445671081543, _, _, _, _, _,
21.6617832183838, _, _, _, 19.8180713653564, _, _, _, _, _ ;
m_depth = _, _, 222.494979858398, _, _, 224.837829589844, _, _, _,
227.069107055664, _, _, _, 229.411956787109, _, _, _, _, _,
231.559555053711, _ ;
Still some work to do...
There are several steps involved in migration to parquet for intermediate processing of slocum glider data.
dbdreader
reproduces similar results to slocum binaries (https://github.com/smerckel/dbdreader/issues/18)convertDbds.sh
withdbdreader
/parquet
parquet
(just using tables for now)Is there a particular storage pattern or design desired for the
parquet
data structures? REF: https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-optionsHave to nail down a potential dbdreader issue first.