Issue reading ADV data

[MasonBichanich]

Hi,

This seems like a silly issue, but its stumping me. I'm easily able to load the example ADV data (vector_data_imu01.VEC), but I can't load my own data. Below is the two lines of code, and below that are the errors thrown.

""" import dolfyn.io.api as api dat = api.read('test_file.VEC')

Reading file test_file.VEC ... Traceback (most recent call last): File "c:\mypath\ADV_Processing_MHKit.py", line 5, in dat = api.read_nortek('test_file.VEC') #vector_data_imu01 #test_file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mason\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\dolfyn\io\nortek.py", line 57, in read_nortek rdr.readfile() File "C:\Users\mason\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\dolfyn\io\nortek.py", line 388, in readfile retval = self.readnext() ^^^^^^^^^^^^^^^ File "C:\Users\mason\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\dolfyn\io\nortek.py", line 373, in readnext out = getattr(self, func_name)() # Should return None ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mason\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\dolfyn\io\nortek.py", line 488, in read_user_cfg cfg_u['coord_sys_axes'] = ['ENU', 'XYZ', 'beam'][tmp[14]]

IndexError: list index out of range

"""

[jmcvey3]

Hi @MasonBichanich, thanks for submitting this! I see you're using python 3.11 - see if you can create a python 3.9 environment and let me know if it works. In the meantime I'll try to get this package working on 3.10 and 3.11.

[jmcvey3]

The main reason 3.11 doesn't work is because we have netCDF4 pinned to 1.5.8 due to issues with more recent versions.

[MasonBichanich]

I tried it in a 3.9 environment and got the same error. I have no problems when I try to read the 'vector_data_imu01.vec' file in either a 3.9 or 3.11 environment.

[jmcvey3]

Gotcha. Can you share the deployment configuration file (.dep) and/or the test file?

[MasonBichanich]

Here is the file I'm trying to read. The data should be just white noise; the data was taken in air. I'm using this to develop post-processing for data that will be coming in shortly. It was taken with the exact same instrument settings, so the .dep will be the same. vec_and_dep.zip

[jmcvey3]

Got it, thanks. There's a corrupted ensemble at the end of the file, which you see for yourself by running dolfyn.read('test_file.VEC', debug=True) . This will still error out, but dolfyn will create a log file in your working directory. Open the log file in a text editor and scroll to the bottom to see this error.

You can avoid that particular error by reading in an exact number of ensembles listed at the bottom of the log file. Looks like ping #16668 was the last good measurement, so you can read up til then via ds = dolfyn.read('test_file.VEC', nens=16668).

I'd recommend getting another test file if you can, and make sure to disconnect the instrument after you finish recording and/or transfer the data file to make sure it's all there. Ping me again if this happens again, and I'll add some logic to return the non-corrupted data.

FYI I also noticed the date on the timestamps are Jan 1, 2002, which seemed strange to me.

[MasonBichanich]

That was it! I collected another data file and this time I made sure to properly terminate the test. FYI the weird timestamp was just because I hadn't set the instrument clock when I was recording online. Easy fix.

Thank you!!

[jmcvey3]

Great! Glad it was an simple fix.