Reading TRDI data (include wave option)

[ivicajan]

Hi, I am wondering if Dolfyn can handle the TRDI WH raw binary that is holding in additional wave data as well. In my case, we have ADCP deployed using the directional wave option and I would like to read ONLY standard ADCP variables (i.e. nothing related to waves). Just the standard way of reading data couldn't find any valid data. Then I modified io/rdi.py to extend the search for valid records: _search_num = 500000 Then I got something (12,000 records), with wrong time etc.. definitely not what I would expect (and based on RDI WinADCP simple app).

Thanks in advance! Ivica

[jmcvey3]

Hi @ivicajan, thanks for using dolfyn and bringing this up. Dolfyn can't read wave statistics from any ADCP at the moment, but it should still read all other variables. Can you check out https://github.com/MHKiT-Software/MHKiT-Python/pull/292? I made a number of bugfixes in this PR. This is a different package called mhkit, but you can use your current dolfyn code with from mhkit import dolfyn. Would recommend using a clean environment as well.

[ivicajan]

Hi James, thanks for your reply.

I switched to 292 and have installed MHKIT inside a separate conda environment to make sure all is clean. Unfortunately, it is not working and I have the same error as before:

(ADCP) ivica@omen:~/GIT/MHKiT-Python$ gh pr checkout 292 remote: Enumerating objects: 109, done. remote: Counting objects: 100% (109/109), done. remote: Compressing objects: 100% (53/53), done. remote: Total 109 (delta 83), reused 82 (delta 56), pack-reused 0 Receiving objects: 100% (109/109), 157.02 KiB | 1.65 MiB/s, done. Resolving deltas: 100% (83/83), completed with 14 local objects. From https://github.com/MHKiT-Software/MHKiT-Python

• [new ref] refs/pull/292/head -> rdi_sl_fix Switched to branch 'rdi_sl_fix'

(ADCP) ivica@omen:~/GIT/MHKiT-Python$ git status On branch rdi_sl_fix nothing to commit, working tree clean

(ADCP) ivica@omen:~/GIT/MHKiT-Python$ python3 Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from mhkit import dolfyn from mhkit.dolfyn.adp import api ds = dolfyn.read("/mnt/qnap/AUSTRALIA/CB/SCHISM/test/new/ADCP/RAW/CS - Jan 2022/ADCP Workhorse #11265_CS1/WH600000.000")

Reading file /mnt/qnap/AUSTRALIA/CB/SCHISM/test/new/ADCP/RAW/CS - Jan 2022/ADCP Workhorse #11265_CS1/WH600000.000 ... Traceback (most recent call last): File "", line 1, in File "/home/ivica/GIT/MHKiT-Python/mhkit/dolfyn/io/api.py", line 114, in read return func(fname, userdata=userdata, nens=nens, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ivica/GIT/MHKiT-Python/mhkit/dolfyn/io/rdi.py", line 70, in read_rdi datNB, datBB = rdr.load_data(nens=nens) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ivica/GIT/MHKiT-Python/mhkit/dolfyn/io/rdi.py", line 375, in load_data if not self.read_buffer(): ^^^^^^^^^^^^^^^^^^ File "/home/ivica/GIT/MHKiT-Python/mhkit/dolfyn/io/rdi.py", line 491, in read_buffer if not self.search_buffer(): ^^^^^^^^^^^^^^^^^^^^ File "/home/ivica/GIT/MHKiT-Python/mhkit/dolfyn/io/rdi.py", line 582, in search_buffer raise Exception( Exception: Searched 30000 entries... Bad data encountered. -> [121, 127]

If you have any other ideas let me know. P.S. I can put the file on my Google Drive and send you the link (it is 104M)

[ivicajan]

Just tried to increase the search length value in rdi.py (the original was 30000) to 10000000 and seems it is helping a bit. Now it reads part of the data (10166 records) but not all of it as in RDI WinADCP (records = 12139). If I increase the number for searching for valid headers even more nothing happens (i.e. I can't get all records).

Cheers I

[jmcvey3]

Ah yes, this is a rather infamous bug (that we thought we fixed). For whatever reason this data is stored in a slightly different format than what RDI documents.

Essentially what's happening here is that dolfyn is skipping over headers starting with the 127 or 121 key til it finds a "valid" header. This is why extending the search records helps it eventually find something.

If you can share the file over drive that would be fantastic

[ivicajan]

Hi James, here is a link to the WH RDI file:

https://drive.google.com/file/d/196ts1HuUpbYu0YN6tzILrzhoYTcew28p/view?usp=drive_link

let me know when you download it or if any problems. Cheers, Ivica

On Thu, 7 Mar 2024 at 01:07, James McVey @.***> wrote:

Ah yes, this is a rather infamous bug (that we thought we fixed). For whatever reason this data is stored in a slightly different format than what RDI documents.

Essentially what's happening here is that dolfyn is skipping over headers starting with the 127 or 121 key til it finds a "valid" header. This is why extending the search records helps it eventually find something.

If you can share the file over drive that would be fantastic

— Reply to this email directly, view it on GitHub https://github.com/lkilcher/dolfyn/issues/128#issuecomment-1981370463, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEN4TMDILKQATQZ5YWP5AG3YW5ELTAVCNFSM6AAAAABDZTNE2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBRGM3TANBWGM . You are receiving this because you were mentioned.Message ID: @.***>

[jmcvey3]

Hi James, here is a link to the WH RDI file: https://drive.google.com/file/d/196ts1HuUpbYu0YN6tzILrzhoYTcew28p/view?usp=drive_link let me know when you download it or if any problems. Cheers, Ivica … On Thu, 7 Mar 2024 at 01:07, James McVey @.> wrote: Ah yes, this is a rather infamous bug (that we thought we fixed). For whatever reason this data is stored in a slightly different format than what RDI documents. Essentially what's happening here is that dolfyn is skipping over headers starting with the 127 or 121 key til it finds a "valid" header. This is why extending the search records helps it eventually find something. If you can share the file over drive that would be fantastic — Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEN4TMDILKQATQZ5YWP5AG3YW5ELTAVCNFSM6AAAAABDZTNE2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBRGM3TANBWGM . You are receiving this because you were mentioned.Message ID: @.>

Just requested permission to access.

[jmcvey3]

Hi Ivica, thanks for sharing the file. I was able to read the file in the above PR and get a similar output to WinADCP. Were you expecting wave measurements as well? This file is from a Workhorse, which can't take wave measurements.

[ivicajan]

HI James, thanks for fixing this. Will try next week if it works. Will it be merged into the master/development branch or should I use PR? There should be wave data inside the file but I am not expecting dolfyn to read that chunk. Long time ago there was a small app, even possibly DOS, that split from the big RDI binary file into 2 files, one with waves and other (much smaller) with only "standard WH" variables (as without wave mode). Have you compared the number of ensembles from WinADCP and Python? If they are the same then that's it.

Thanks! Ivica

On Sat, 9 Mar 2024 at 05:33, James McVey @.***> wrote:

Hi Ivica, thanks for sharing the file. I was able to read the file in the above PR and get a similar output to WinADCP, though, there didn't appear to be any wave data saved in it. Was that something you were expecting?

— Reply to this email directly, view it on GitHub https://github.com/lkilcher/dolfyn/issues/128#issuecomment-1986450255, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEN4TMDA3THCPI66MKAQW2LYXIVA3AVCNFSM6AAAAABDZTNE2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWGQ2TAMRVGU . You are receiving this because you were mentioned.Message ID: @.***>

[jmcvey3]

HI James, thanks for fixing this. Will try next week if it works. Will it be merged into the master/development branch or should I use PR? There should be wave data inside the file but I am not expecting dolfyn to read that chunk. Long time ago there was a small app, even possibly DOS, that split from the big RDI binary file into 2 files, one with waves and other (much smaller) with only "standard WH" variables (as without wave mode). Have you compared the number of ensembles from WinADCP and Python? If they are the same then that's it. Thanks! Ivica

Hi Ivica, yes you can use the PR for now. It'll get merged into develop and eventually published from master. Gotcha, and yes, the number of ensembles is similar. Looks like dolfyn reads in a few more empty ensembles that WinADCP trims off.

Best, James