utilization of parquet for intermediate storage

jr3cermak commented 1 year ago

There are several steps involved in migration to parquet for intermediate processing of slocum glider data.

[x] ensure dbdreader reproduces similar results to slocum binaries (https://github.com/smerckel/dbdreader/issues/18)
[ ] replacement of convertDbds.sh with dbdreader/parquet
[x] desired storage pattern for parquet (just using tables for now)

Is there a particular storage pattern or design desired for the parquet data structures? REF: https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-options

Enforce version 2.4? 2.6?
Ensure the structure is queryable by time for speedy subsetting?
If enforcing 2.6, timestamp units become less an issue
Partitioning? Glider ID, Deployment ID, Process method (rt vs. delayed), QC'd (Level 0, 1, ...)
- local file system or plug into directly to duckdb (https://duckdb.org/docs/guides/python/filesystems.html) or allow both

Have to nail down a potential dbdreader issue first.

jr3cermak commented 1 year ago

I can now confirm that dbdreader can be used to reproduce dba_merge. This code snippet will take the two files and merge the data via pandas to form a dataframe similar to that product to dba_merge.

At this stage, the merged dataframe can be tossed into the parquet framework.

    # Invoke dbdreader on sbd file
    sbdf = "%s%s" % (os.path.join("binary", bn), ".sbd")
    sbd = dbdreader.DBD(sbdf, cacheDir=cache_dir, skip_initial_line=False)

    # Invoke dbdreader on tbd file
    tbdf = "%s%s" % (os.path.join("binary", bn), ".tbd")
    tbd = dbdreader.DBD(tbdf, cacheDir=cache_dir, skip_initial_line=False)

    # Compare columns
    dbd_columns = sbd.parameterNames + tbd.parameterNames
    dbd_columns.sort()

    slo_columns = list(gu.columns)
    slo_columns.sort()

    assert set(dbd_columns) == set(slo_columns), \
        "Slocum merged and dbdreader columns do not match"

    # Simulate: dba_merge

    sbd_data = pd.DataFrame()
    for col in sbd.parameterNames:
        tm, data = sbd.get(col, return_nans=True, decimalLatLon=False)
        sbd_data[col] = data

    sbd_data = sbd_data.set_index('m_present_time')
    sbd.close()

    tbd_data = pd.DataFrame()
    for col in tbd.parameterNames:
        tm, data = tbd.get(col, return_nans=True, decimalLatLon=False)
        tbd_data[col] = data
        if col == 'sci_m_present_time':
            # Create a duplicate column for merging
            tbd_data['m_present_time'] = data

    tbd_data = tbd_data.set_index('m_present_time')
    tbd.close()

    dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
    dbd_data = dbd_data.reset_index()

A quick comparison of 9 separate files and values obtained via dba_merge and dbdreader:

$ python cmpData.py 

===> ascii/unit_507_2022_078_0_0_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 4.010009767796419e-06 4.94964599617731e-05
   m_battery 3.4179687524726887e-07 4.945220947227824e-05
   m_battpos 2.682209010451686e-09 4.988937377659042e-07
   m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
   m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
   m_depth 0.0 0.0004963378906381877
   m_depth_rate 0.0 4.78054046626486e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 7.615256309660007e-10 4.785308838073377e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
   m_lon 0.0 4.547473508864641e-11
   m_pitch 9.338641165923534e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
   m_water_depth 0.0 0.0004663085937579581
   m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
   m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 4.936542510880493e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.97009277324878e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 4.342842102156297e-06
   sci_echodroid_sa 0.0 4.931030273525039e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.051462113930163e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1871337912694457e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_1_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
   m_battery 3.0398559569988493e-05 3.076019287107101e-05
   m_battpos 4.100151062003121e-07 4.100151062003121e-07
   m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
   m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 9.155273437144729e-07 9.155273437144729e-07
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
   m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 7.152557373046875e-07 4.5299530029296875e-06
   m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
   m_roll 1.9259738928689707e-08 1.9259738928689707e-08
   m_vacuum 4.727478026822496e-06 2.901611328098852e-05
   m_water_depth 0.00015649414064000666 0.00015649414064000666
   m_water_vx 2.952008247569804e-08 2.952008247569804e-08
   m_water_vy 3.911998271849404e-08 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_2_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.674072265586915e-05
   m_battery 4.711914062482947e-06 4.655303955125589e-05
   m_battpos 2.6850545406867377e-09 4.959058761688784e-07
   m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
   m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
   m_depth 0.0 0.0004937744140676159
   m_depth_rate 0.0 4.837455749584496e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 9.047412874441108e-09 4.929962158506385e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
   m_lon 3.637978807091713e-12 4.9112713895738125e-11
   m_pitch 1.5230512619320014e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 1.972961425700248e-05
   m_water_depth 0.0 0.0004902343749790816
   m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
   m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 3.8234138488579816e-07
   sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
   sci_echodroid_eqarea 0.0 4.992523193436682e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.047749638507597e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1894226092934446e-07
   sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07

===> ascii/unit_507_2022_078_0_3_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 5.052185057863312e-06
   m_battery 1.0815429687838218e-05 4.9467468262065495e-05
   m_battpos 1.6136550906598046e-07 4.100151062003121e-07
   m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 3.634643555017192e-07 3.634643555017192e-07
   m_lat 2.7284841053187847e-12 2.7284841053187847e-12
   m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 4.76837158203125e-07 3.814697265625e-06
   m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
   m_water_depth 6.762695312545475e-05 6.762695312545475e-05
   m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
   m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_4_sbd.dat
   c_wpt_lat 0.0 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.9743652340339395e-05
   m_battery 4.3533325193578776e-06 4.867706298838925e-05
   m_battpos 2.24747955792845e-09 4.659767150749783e-07
   m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
   m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
   m_depth 0.0 0.0004984130859497782
   m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.4016113359978135e-08 4.980773925744586e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 7.615256309660007e-10 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
   m_water_depth 0.0 0.0004943847656306843
   m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
   m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.7902107236375e-07
   sci_echodroid_ctrmass 0.0 4.894409179456716e-05
   sci_echodroid_eqarea 0.0 4.985046386707381e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.049237966631536e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_5_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 2.012023925956896e-05 2.012023925956896e-05
   m_battery 3.368377685575297e-05 4.551544189368428e-05
   m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
   m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
   m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.1898803711218875e-06 1.1898803711218875e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
   m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
   m_lon 3.8198777474462986e-11 3.8198777474462986e-11
   m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
   m_present_time 0.0 4.5299530029296875e-06
   m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_roll 2.4125957495679806e-08 2.4125957495679806e-08
   m_vacuum 4.447937012486136e-06 3.832702636685781e-05
   m_water_depth 0.0003625488281215894 0.0003625488281215894
   m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
   m_water_vy 4.524588585047562e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_6_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867186647349e-07 4.707641601697787e-05
   m_battery 6.86645513781059e-08 4.9478149414028394e-05
   m_battpos 6.442186237590186e-11 4.7547626494393036e-07
   m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
   m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
   m_depth 0.0 0.0005000000000023874
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.546569825654046e-08 4.954223632669397e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
   m_lon 1.8189894035458565e-12 4.3655745685100555e-11
   m_pitch 1.5230512619320014e-09 4.966812133799792e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 2.639770535495245e-08 4.555969238317914e-05
   m_water_depth 0.0 0.00042846679687613687
   m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
   m_water_vy 1.688668727767295e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.939737320019955e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.858703613308535e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.055169820814733e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_7_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.096618652520533e-05 3.096618652520533e-05
   m_battery 2.7200317383346828e-05 4.9478149414028394e-05
   m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
   m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.3704681398607477e-06 1.3704681398607477e-06
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
   m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
   m_lon 4.001776687800884e-11 4.001776687800884e-11
   m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
   m_present_time 4.76837158203125e-07 4.76837158203125e-06
   m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
   m_roll 4.1464805602964283e-07 4.1464805602964283e-07
   m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
   m_water_depth 0.00015234375001682565 0.00015234375001682565
   m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
   m_water_vy 1.688668727767295e-08 1.688668727767295e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_8_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867151120212e-07 4.933776855509109e-05
   m_battery 3.372192374229144e-07 4.8551940917462844e-05
   m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
   m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
   m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
   m_depth 0.0 0.0004981689453131821
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 2.3651121061618596e-09 4.9253082274880455e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 2.155303935325037e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
   m_water_depth 0.0 0.0004304199218836402
   m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
   m_water_vy 7.65342712469308e-09 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 4.809398650995611e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.702453613347757e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 3.5113906859951527e-06
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.053686261228789e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07

===> ascii/unit_507_2022_078_0_9_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_battery 5.773925781937805e-06 4.3586730956945985e-05
   m_battpos 6.470108027123445e-08 6.470108027123445e-08
   m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
   m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
   m_depth 0.0 3.758859635008549e-09
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 0.0 0.0
   m_heading 2.923889160122428e-06 2.923889160122428e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
   m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
   m_lon 4.3655745685100555e-11 4.3655745685100555e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 9.5367431640625e-07 2.6226043701171875e-06
   m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 2.748107910122144e-06 9.059143065925923e-06
   m_water_depth 0.0002636718750181899 0.0002636718750181899
   m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
   m_water_vy 4.458808899182598e-08 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
   sci_water_temp 0.0 0.0
Maximum deviation: 0.0005000000000023874
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py | less
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py 

===> ascii/unit_507_2022_078_0_0_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 4.010009767796419e-06 4.94964599617731e-05
   m_battery 3.4179687524726887e-07 4.945220947227824e-05
   m_battpos 2.682209010451686e-09 4.988937377659042e-07
   m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
   m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
   m_depth 0.0 0.0004963378906381877
   m_depth_rate 0.0 4.78054046626486e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 7.615256309660007e-10 4.785308838073377e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
   m_lon 0.0 4.547473508864641e-11
   m_pitch 9.338641165923534e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
   m_water_depth 0.0 0.0004663085937579581
   m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
   m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 4.936542510880493e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.97009277324878e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 4.342842102156297e-06
   sci_echodroid_sa 0.0 4.931030273525039e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.051462113930163e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1871337912694457e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_1_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
   m_battery 3.0398559569988493e-05 3.076019287107101e-05
   m_battpos 4.100151062003121e-07 4.100151062003121e-07
   m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
   m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 9.155273437144729e-07 9.155273437144729e-07
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
   m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 7.152557373046875e-07 4.5299530029296875e-06
   m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
   m_roll 1.9259738928689707e-08 1.9259738928689707e-08
   m_vacuum 4.727478026822496e-06 2.901611328098852e-05
   m_water_depth 0.00015649414064000666 0.00015649414064000666
   m_water_vx 2.952008247569804e-08 2.952008247569804e-08
   m_water_vy 3.911998271849404e-08 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_2_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.674072265586915e-05
   m_battery 4.711914062482947e-06 4.655303955125589e-05
   m_battpos 2.6850545406867377e-09 4.959058761688784e-07
   m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
   m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
   m_depth 0.0 0.0004937744140676159
   m_depth_rate 0.0 4.837455749584496e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 9.047412874441108e-09 4.929962158506385e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
   m_lon 3.637978807091713e-12 4.9112713895738125e-11
   m_pitch 1.5230512619320014e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 1.972961425700248e-05
   m_water_depth 0.0 0.0004902343749790816
   m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
   m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 3.8234138488579816e-07
   sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
   sci_echodroid_eqarea 0.0 4.992523193436682e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.047749638507597e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1894226092934446e-07
   sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07

===> ascii/unit_507_2022_078_0_3_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 5.052185057863312e-06
   m_battery 1.0815429687838218e-05 4.9467468262065495e-05
   m_battpos 1.6136550906598046e-07 4.100151062003121e-07
   m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 3.634643555017192e-07 3.634643555017192e-07
   m_lat 2.7284841053187847e-12 2.7284841053187847e-12
   m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 4.76837158203125e-07 3.814697265625e-06
   m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
   m_water_depth 6.762695312545475e-05 6.762695312545475e-05
   m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
   m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_4_sbd.dat
   c_wpt_lat 0.0 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.9743652340339395e-05
   m_battery 4.3533325193578776e-06 4.867706298838925e-05
   m_battpos 2.24747955792845e-09 4.659767150749783e-07
   m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
   m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
   m_depth 0.0 0.0004984130859497782
   m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.4016113359978135e-08 4.980773925744586e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 7.615256309660007e-10 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
   m_water_depth 0.0 0.0004943847656306843
   m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
   m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.7902107236375e-07
   sci_echodroid_ctrmass 0.0 4.894409179456716e-05
   sci_echodroid_eqarea 0.0 4.985046386707381e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.049237966631536e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_5_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 2.012023925956896e-05 2.012023925956896e-05
   m_battery 3.368377685575297e-05 4.551544189368428e-05
   m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
   m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
   m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.1898803711218875e-06 1.1898803711218875e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
   m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
   m_lon 3.8198777474462986e-11 3.8198777474462986e-11
   m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
   m_present_time 0.0 4.5299530029296875e-06
   m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_roll 2.4125957495679806e-08 2.4125957495679806e-08
   m_vacuum 4.447937012486136e-06 3.832702636685781e-05
   m_water_depth 0.0003625488281215894 0.0003625488281215894
   m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
   m_water_vy 4.524588585047562e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_6_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867186647349e-07 4.707641601697787e-05
   m_battery 6.86645513781059e-08 4.9478149414028394e-05
   m_battpos 6.442186237590186e-11 4.7547626494393036e-07
   m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
   m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
   m_depth 0.0 0.0005000000000023874
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.546569825654046e-08 4.954223632669397e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
   m_lon 1.8189894035458565e-12 4.3655745685100555e-11
   m_pitch 1.5230512619320014e-09 4.966812133799792e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 2.639770535495245e-08 4.555969238317914e-05
   m_water_depth 0.0 0.00042846679687613687
   m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
   m_water_vy 1.688668727767295e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.939737320019955e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.858703613308535e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.055169820814733e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07

===> ascii/unit_507_2022_078_0_7_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.096618652520533e-05 3.096618652520533e-05
   m_battery 2.7200317383346828e-05 4.9478149414028394e-05
   m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
   m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.3704681398607477e-06 1.3704681398607477e-06
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
   m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
   m_lon 4.001776687800884e-11 4.001776687800884e-11
   m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
   m_present_time 4.76837158203125e-07 4.76837158203125e-06
   m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
   m_roll 4.1464805602964283e-07 4.1464805602964283e-07
   m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
   m_water_depth 0.00015234375001682565 0.00015234375001682565
   m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
   m_water_vy 1.688668727767295e-08 1.688668727767295e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
   sci_water_temp 0.0 0.0

===> ascii/unit_507_2022_078_0_8_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867151120212e-07 4.933776855509109e-05
   m_battery 3.372192374229144e-07 4.8551940917462844e-05
   m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
   m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
   m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
   m_depth 0.0 0.0004981689453131821
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 2.3651121061618596e-09 4.9253082274880455e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 2.155303935325037e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
   m_water_depth 0.0 0.0004304199218836402
   m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
   m_water_vy 7.65342712469308e-09 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 4.809398650995611e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.702453613347757e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 3.5113906859951527e-06
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.053686261228789e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07

===> ascii/unit_507_2022_078_0_9_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_battery 5.773925781937805e-06 4.3586730956945985e-05
   m_battpos 6.470108027123445e-08 6.470108027123445e-08
   m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
   m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
   m_depth 0.0 3.758859635008549e-09
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 0.0 0.0
   m_heading 2.923889160122428e-06 2.923889160122428e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
   m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
   m_lon 4.3655745685100555e-11 4.3655745685100555e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 9.5367431640625e-07 2.6226043701171875e-06
   m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 2.748107910122144e-06 9.059143065925923e-06
   m_water_depth 0.0002636718750181899 0.0002636718750181899
   m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
   m_water_vy 4.458808899182598e-08 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
   sci_water_temp 0.0 0.0

Maximum deviation: 0.0005000000000023874

The deviations occur as dba_merge does not pass through the full precision of the float values when the conversion to ascii happens. As I have mentioned in the past, we need to full precision from the dbdreader to extract the embedded echograms in the echometrics data.

The code will have to be refactored a bit to handle both pairs of files. The method should remain the same. Once that is written, the convertDbds.sh can go away with the ascii conversion step.

jr3cermak commented 1 year ago

Or convert the ascii reader using the above to parquet and then adapt the netcdf part to pull from parquet instead of ascii.

jr3cermak commented 1 year ago

Yes, will make this a gradual transition. Added a hook to extra_kwargs to enable utilization of dbdreader/parquet so both pathways will continue to exist for a little while.

deployment.json:

{
    "glider": "unit_507",
    "trajectory_date": "20220212T0000",
    "filters": {
        "tsint": 20,
        "filter_z": 1,
        "filter_time": 5,
        "filter_points": 5,
        "filter_distance": 1
    },
    "extra_kwargs": {
        "enable_parquet": true,
        "echograms": {
            "enable_nc": true,
            "enable_ascii": true,
            "enable_image": true,
            "plot_type": "profile",
            "plot_cmap": "ek80",
            "svb_thresholds": [-10,-20,-25,-30,-35,-45,-55],
            "svb_limits": [-10.0, -55.0],
            "echogram_range_bins": 20,
            "echogram_range": -60.0,
            "echogram_range_units": "meters"
        }
    },
    "attributes": {
        "acknowledgement": "This work was supported by funding from NOAA/IOOS/AOOS.",
        "comment": "",

Enabling parquet also assumes dbdreader will also be used.

$ pytest -k TestEchoMetricsFive
====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0 -- /home/cermak/miniconda3/envs/gutils_py3_9/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/cermak/src/GUTILS, configfile: setup.cfg
plugins: anyio-3.6.2
collected 36 items / 35 deselected / 1 selected                                                                                                                                                  

gutils/tests/test_slocum.py::TestEchoMetricsFive::test_echogram 2023-06-17 14:34:08,422 - gutils.slocum - INFO - PARQUET enabled deployment
2023-06-17 14:34:08,423 - gutils.slocum - INFO - Creating echogram profile images with colormap ek80
2023-06-17 14:34:23,540 - gutils.slocum - INFO - Converted unit_507-2022-078-0-0.sbd,unit_507-2022-078-0-0.tbd to unit_507_2022_078_0_0_sbd.dat

Now to bolt on the decoding code...

... and push certain things over to the nc processing side.

Getting parquet tables to store datetime[ns, UTC] was giving me a problem initially. This gets us to the goal of parquet tables and nc/xarray and preserving the seconds since 1990-0-0 units.

Once the merge happens:

    dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
    dbd_data = dbd_data.reset_index()

drop sci_m_present_time and rename m_present_time to time

    # This works for xarray and parquet
    dbd_data['time'] = [datetime.datetime.fromtimestamp(tm, tz=datetime.timezone.utc) for tm in dbd_data['time'].values]

    # Convert time to datetime and preserve fractional seconds
    # NOTE: works for xarray; not for parquet
    #dbd_data['time'] = pd.to_datetime(dbd_data['time'], utc=True, unit='s')

    # Make time the index
    dbd_data = dbd_data.set_index('time')

    # Write files to storage

    # write netcdf
    fn_xr = "%s%s" % (bn, ".nc")
    ds = dbd_data.to_xarray()
    ds['time'] = ds['time'].astype('datetime64[ns]')
    encodings = {}
    for dvar in ds.data_vars:
        encodings[dvar] = {"_FillValue": None}
    encodings.update({
        "time": {
            "_FillValue": None,
            "calendar": "standard",
            "units": "seconds since 1990-01-01T00:00:00Z"
        }
    })
    ds.to_netcdf(fn_xr, encoding = encodings)

    # write parquet
    fn_pq = "%s%s" % (bn, ".parquet")
    dbd_data.to_parquet(fn_pq)

    # Verify what we wrote can be read back in

    # read netcdf
    #nc_ver = xr.open_dataset(fn_xr)
    # NOTE: initial checks showed that original values differ using xarray

    # read parquet
    #pq_ver = pq.read_table(fn_pq)

Instead of *.dat files, there will be .parquet files in the ascii directory for the nc side to utilize with the flag. Temporary until things get fully moved about.

jr3cermak commented 1 year ago

As expected, picking up extra resolution by utilizing dbdreader and parquet for processing.

Original m_depth and m_altitude via dba_merge:

m_altitude = _, _, _, _, _, 24.5446, _, _, _, _, _, 21.6618, _, _, _, 
    19.8181, _, _, _, _, _ ;

 m_depth = _, _, 222.494995117188, _, _, 224.837997436523, _, _, _, 
    227.069000244141, _, _, _, 229.412002563477, _, _, _, _, _, 
    231.559997558594, _ ;

via dbdreader and parquet:

m_altitude = _, _, _, _, _, 24.5445671081543, _, _, _, _, _, 
    21.6617832183838, _, _, _, 19.8180713653564, _, _, _, _, _ ;

m_depth = _, _, 222.494979858398, _, _, 224.837829589844, _, _, _, 
    227.069107055664, _, _, _, 229.411956787109, _, _, _, _, _, 
    231.559555053711, _ ;

Still some work to do...

SECOORA / GUTILS

utilization of parquet for intermediate storage #26