rdi read file: KeyError: 'vel'

[apatlpo]

Trying to read shipboard ADCP data, I get the following error:

Reading file /Users/aponte/Current_Projects/drix/campaigns/drixmed22/data/ESSDRIX-EU-2022/ADCP/ADCP_Europe_20220921T083153_007_000000.ENS ...
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[4], line 2
      1 file = files[-1]
----> 2 dat = dlfn.read(file)
...
File ~/Code/dolfyn/dolfyn/io/base.py:122, in _create_dataset(data)
    120 print(list(data['data_vars']))
    121 print(data['data_vars'])
--> 122 beams = list(range(1, data['data_vars']['vel'].shape[0]+1))
    123 FoR['beam'] = xr.DataArray(beams, dims=['beam'], name='beam', attrs={
    124                            'units': '1', 'long_name': 'Beam Reference Frame'})
    125 FoR['dir'] = xr.DataArray(beams, dims=['dir'], name='dir', attrs={
    126                           'units': '1', 'long_name': 'Reference Frame'})

KeyError: 'vel'

Pocking around, I figured data['data_vars'] contains a vel_bt variable but no vel variable. What would be a good to try to fix this?

@naiglb

[apatlpo]

Any maintainer around ? @lkilcher @jmcvey3

[apatlpo]

I should had I am happy to dive in the code and propose a fix but would just need pointers as to the level at which this should be addressed.

[jmcvey3]

Hi @apatlpo, interesting, never seen this one before. What's the make model of the instrument?

The issue will lie in dolfyn.io.rdi.read_rdi. Reread the file and set debug_level=1 or 2, and that will help you see what id codes (vel, vel_bt, orientation, etc) are read or skipped in a corollary log file. The log file will be saved in the same place as your input file.

[apatlpo]

Great, thx for the suggestion !

I was able to produce the log file. Here are the variables read:

(pynsitu) ➜  drix_analysis git:(main) ✗ grep "INFO - Read" ADCP_Europe_20220921T083153_007_000000.log | sort -t: -u -k1,1
root - INFO - Read Bottom Track
root - INFO - Read Config
root - INFO - Read Fixed
root - INFO - Read Header
root - INFO - Read VMDAS
root - INFO - Read Var

So the variable 'vel' is indeed not read. I had an independent source of information from the producer of the data: the data was produced for a dedicated calibration procedure of a shipboard ADCP and they hence turned off velocity acquisition during this experiment to maximize the number of bottom track pings. Everything is overall consistent.

The question seems to be, should dolfyn be robust to such configuration? What do you think dolfyn developers?

As a sideline, I struggled a bit to find the log file and was suprised to see it ending up in the data repository with a name that turns out to be very close to other pre-existing log files (e.g. I had preexisting file ending with the suffix ".LOG"). I would have thought such log would end up wherever the python script is executed from and with an addition to the filename specifying the file originates from dolphyn ("..._dolfyn.log" suffix for instance). Relevant lines are for reference: https://github.com/lkilcher/dolfyn/blob/18fe9be676480eed0abe7fdfbb8a76d126da2841/dolfyn/io/rdi.py#L50

[jmcvey3]

Great, thanks for that information. I removed the dependence on the 'vel' variable that throws the error you show above in PR #118; if you can pull that PR or branch, see if that will run and not error.

Oof, that's an unfortunate oversight. I added ".dolfyn.log" to the end of the filename.

[apatlpo]

Hi, thx for your reactivity !

Unfortunately, it seems we are not quite there yet. I do not have the following error now:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[5], line 2
      1 file = files[-1]
----> 2 dat = dlfn.read(file, debug_level=2)

File ~/Code/dolfyn/dolfyn/io/api.py:103, in read(fname, userdata, nens, **kwargs)
     99     func_map = dict(RDI=read_rdi,
    100                     nortek=read_nortek,
    101                     signature=read_signature)
    102     func = func_map[file_type]
--> 103 return func(fname, userdata=userdata, nens=nens, **kwargs)

File ~/Code/dolfyn/dolfyn/io/rdi.py:92, in read_rdi(filename, userdata, nens, debug_level, vmdas_search, winriver, **kwargs)
     89     dat['data_vars'][ky] = tmlib.epoch2dt64(dat['data_vars'][ky])
     91 # Create xarray dataset from upper level dictionary
---> 92 ds = _create_dataset(dat)
     93 ds = _set_coords(ds, ref_frame=ds.coord_sys)
     95 # Create orientation matrices

File ~/Code/dolfyn/dolfyn/io/base.py:186, in _create_dataset(data)
    184 # Fill in dimensions and coordinates for each dataArray
    185 shp = data['data_vars'][key].shape
--> 186 vshp = data['data_vars']['vel'].shape
    187 l = len(shp)
    188 if l == 1:  # 1D variables

KeyError: 'vel'

[jmcvey3]

Hi, thx for your reactivity !

Unfortunately, it seems we are not quite there yet. I do not have the following error now:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[5], line 2
      1 file = files[-1]
----> 2 dat = dlfn.read(file, debug_level=2)

File ~/Code/dolfyn/dolfyn/io/api.py:103, in read(fname, userdata, nens, **kwargs)
     99     func_map = dict(RDI=read_rdi,
    100                     nortek=read_nortek,
    101                     signature=read_signature)
    102     func = func_map[file_type]
--> 103 return func(fname, userdata=userdata, nens=nens, **kwargs)

File ~/Code/dolfyn/dolfyn/io/rdi.py:92, in read_rdi(filename, userdata, nens, debug_level, vmdas_search, winriver, **kwargs)
     89     dat['data_vars'][ky] = tmlib.epoch2dt64(dat['data_vars'][ky])
     91 # Create xarray dataset from upper level dictionary
---> 92 ds = _create_dataset(dat)
     93 ds = _set_coords(ds, ref_frame=ds.coord_sys)
     95 # Create orientation matrices

File ~/Code/dolfyn/dolfyn/io/base.py:186, in _create_dataset(data)
    184 # Fill in dimensions and coordinates for each dataArray
    185 shp = data['data_vars'][key].shape
--> 186 vshp = data['data_vars']['vel'].shape
    187 l = len(shp)
    188 if l == 1:  # 1D variables

KeyError: 'vel'

Ah, my apologies, that line 186 should be removed completely. Just pushed another commit.

[apatlpo]

Works like a charm ! Looking forward to see #118 being merged. Many thanks !