smerckel / dbdreader

A reader for binary data files created by Slocum ocean gliders (AUVs)
GNU General Public License v3.0
16 stars 14 forks source link

Support reading *bd files with swapped bytes (different endian'ness)? #6

Closed erinaceous closed 2 years ago

erinaceous commented 2 years ago

Hi!

We've got some Slocum G3s out in the wild, the timestamps and data in the .tbd and .sbd files we're getting back from them currently aren't reading right from dbdreader - I believe the first "known bytes binary cycle" is flipped for these files. When read in dbdreader it becomes clearer that, at least for the timestamps, the bytes should be swapped:

In [29]: dbd = dbdreader.DBD(filename='unit_887-2021-321-3-26.sbd', cacheDir='.')

In [30]: timestamps, data = dbd.get('m_battery')

In [31]: timestamps[0]
Out[31]: -2.8327471839612063e+35      # definitely NOT a unix timestamp

In [32]: timestamps[0].byteswap()
Out[32]: 1637300489.114                       # a unix timestamp

But unfortunately this trick only works for m_present_time I guess as it's always there as a float64..? The actual data does not become readable when byte-swapped:

In [43]: data[0]
Out[43]: -1.0945156144431104e+16

In [44]: data[0].byteswap()
Out[44]: 4.759192468537e-312

We're able to parse these using latest dbd2asc and the dinkum python library by CEOTR, which checks for the magic number 0x1234.

My C is a bit rusty so I might be reading it wrong, but it looks like get_by_read_per_byte currently skips over that known bytes cycle (which I calculated should be.. 16 bytes, so close to bin_offset+17?), and read_sensor_value assumes... big-endian (?) for floats and doubles?

If my C was better I'd submit a PR :-)

If you'd like some example data from one of our G3s, let me know and I'll email some!

Owain Jones NOC