Wave height and peak period

[ouhssam]

Is it possible to use OCE to calculate the wave height, peak period, direction from ADCP Teledyne workstation 1200khz. Thanks you for your help. Best, Mustapha

[dankelley]

The idea of oce is to read data files and provide the results to the user, to be processed in a way that makes sense for a given application, guided by the scientific literature.

The first question is whether your Workhorse device was set up with the 'waves' package (see snapshot from a workhorse manual, at the end of this comment). And then, was the configuration set up to make those measurements, with what settings, etc.

If the dataset contains wave data, I think oce should be able to read them. If not, please email me part of the file so I can look at it ... "kelley AT dal DOT ca" and I can take a look. Do not post your data to this site because it is public, and I assume you want to work with your data alone, in hopes of publishing results.

@richardsc might have further comments, as someone with experience in these matters.

---

[richardsc]

I agree with what @dankelley said -- oce is intended to help you read the data you have, but we are not experts in all the possible data analysis techniques so generally don't provide algorithms that are best left to the specific scientific application.

On the question of whether oce can read Workhorse 1200 data that has been upgraded to allow waves fields, I'm not 100% sure of the answer as I have never worked with such data. If I had to guess I'd say that oce probably doesn't read the relevant fields for a wave-measuring instrument (but we'd need to see an example file to know for sure).

[dankelley]

I just did grep -i wave oce/R/adp.rdi.R and there are no hits, so oce does not read wave data. Perhaps that could be added, if we had a sample file. But we would need documentation on the format...

I just looked in the "workhorse ... commands and output data format" of June 2018, and the word "wave" only shows up there for the commands to cause wave sampling to be done. Possibly I missed it, though. And possibly other Teledyne manuals cover this.

Without clear document and a sample file, we cannot make progress on this.

[dankelley]

@ouhssam -- thanks for sending that file privately.

I see that read.adp.rdi() does not decode the wave data in this file. I'm not surprised, actually, because code for that has not been written.

We are at an impasse here, because I have no documentation on this particular file format. (See above comments.)

A side note: the first two bytes of this file are not 0x7F, which is the expected start for the file.

Below (click Details) I am pasting the output of a read.adp.rdi(...,debug=1) call. (I have redacted your filename for privacy.) The comment

NOTE: bad ensemble-start encountered 87274 times

is important. That might be wave information.

We cannot proceed further unless we have documentation from Teledyne/RDI on the file format for wave data. I have 35 PDF documents from RDI about the instruments and file formats. Since they are PDF, they are difficult to search without opening them all. But I have checked the ones that I think are most relevant (used as a reference for the oce code) do not have anything about the waves format.

I may see @richardsc next week, and we might have time to discuss this matter then.

``` fileSize=149871642 read.adp.rdi("(REDACTED)", from=(missing), to=(missing), by=(missing)...) { have Binary Fixed Attitude Header: FALSE isSentinel=FALSE near adp.rdi.R line 829 calling ldc_rdi_in_file() w/ numeric values of from etc skipping 96120 bytes at start of the file, seeking 7F7F byte pair Warning: bad ensemble-start at index 97830 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 99536 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 101242 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 102948 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 104654 (at most 5 warnings will be issued) NOTE: bad ensemble-start encountered 87274 times skipping 96120 bytes at start of the file, seeking 7F7F byte pair Warning: bad ensemble-start at index 97830 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 99536 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 101242 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 102948 (at most 5 warnings will be issued) Warning: bad ensemble-start at index 104654 (at most 5 warnings will be issued) NOTE: bad ensemble-start encountered 87274 times ensemble detection in old (C) code took 5.350000s (user), 0.142000s (system), 5.511000s (total) ensemble detection in new (C++) code took 5.221000s (user), 0.128000s (system), 5.365000s (total) old (C) and new (C++) methods agree on ensemble-detection results completed ensemble detection with numeric from, by, to values next is str(ldc). I think we can shrink this after extracting things List of 6 $ ensembleStart : int [1:3063] 1 855 1709 2563 3417 4271 5125 5979 6833 7687 ... $ ensembleStart64 : num [1:3063] 1 855 1709 2563 3417 ... $ time : int [1:3063] 1712149200 1712150100 1712151000 1712151900 1712152800 1712153700 1712154600 1712155500 1712156400 1712157300 ... $ sec100 : int [1:3063] 0 0 0 0 0 0 0 0 0 0 ... $ buf : raw [1:2615802] 7f 7f 54 03 ... $ ensemble_in_file: int [1:3063] 96121 96975 193949 194803 291777 292631 389605 390459 487433 488287 ... NULL setting from=1, by=1, to=3063 check whether file is large FIXME: delete profileStart before trimming[1:3063]: 78, 932, ..., 2614172, 2615026 profilesInFile=3063 filename: "/Users/kelley/Downloads/DP1K_000.000" profilesToRead:3063 numberOfBeams:4 numberOfCells:35 codes[,1]=0x7f0x000x800x000x000x000x00 codes[,2]=0x7f0x000x000x010x020x030x04 NOTE: only the following codes are recognized. See e.g. Table 45 of the Teledyn-RDI document entitled `Ocean Surveyor Technical Manual_Dec20.pdf`, P/N 95A-6012-00 (December 2020) 0x00 0x01 velocity 0x00 0x02 correlation 0x00 0x03 echo intensity 0x00 0x04 percent good 0x00 0x06 bottom track 0x00 0x0a Sentinel vertical beam velocity 0x00 0x0b Sentinel vertical beam correlation 0x00 0x0c Sentinel vertical beam amplitude 0x00 0x0d Sentinel vertical beam percent good 0x00 0x20 VMDASS 0x00 0x30 Variable Fixed Attitude header 0x00 0x32 Sentinel transformation matrix 0x00 0x0a Sentinel data 0x00 0x0b Sentinel correlation 0x00 0x0c Sentinel amplitude 0x00 0x0d Sentinel percent good 0x01 0x0f ? something to do with V series and 4-beam set up 'v' (velocity) storage for3063profiles,35cells, and4beams set up 'q' (correlation) storage for3063profiles,35cells, and4beams set up 'a' (echo intensity) storage for3063profiles,35cells, and4beams set up 'g' (percent good) storage for3063profiles,35cells, and4beams length(profileStart):3063 header$numberOfDataTypes: 6 profilesToRead=3063 Recognized but unhandled ID codes: $xxGGA [1] 0 $xxVTA [1] 0 $xxGSA [1] 0 length(time)=3063 length(time)=3063 after possible shortening to match length(profileStart) length(profileStart)=3063 after checking for bad profiles) heading[1:3063]: 334.35, 346.85, ..., 1.34, 1.00 profileStart2[1:6126]: 78, 79, ..., 2615026, 2615027 pitch[1:3063]: 0.79, 1.30, ..., -1.29, -1.27 roll[1:3063]: 0.17, 0.54, ..., -0.61, -0.61 will adjust the pitch as explained on page 14 of 'adcp coordinate transformation.pdf' pitch, before correction[1:3063]: 0.79, 1.30, ..., -1.29, -1.27 pitch, after correction[1:3063]: 0.7900, 1.3001, ..., -1.2901, -1.2701 temperature[1:3063]: 24.09, 23.37, ..., 22.07, 22.20 pressure[1:3063]: 12.356, 12.358, ..., -0.257, -0.234 creating data slot for a non-SentinelV file } # read.adp.rdi() ADP Summary ----------- * Filename: "/Users/kelley/Downloads/(REDACTED)" * Instrument: adcp * Manufacturer: teledyne rdi * Serial number: 68611 * Firmware: 50.41 * Cell Size: 0.50 m * Beam Angle: 20 deg * Location: unknown latitude, unknown longitude * Frequency: 1200 kHz * Ensemble Numbers: 1, 2, ..., 3062, 3063 * Transformation matrix:: 1.462 -1.462 0.000 0.000 0.000 0.000 -1.462 1.462 0.266 0.266 0.266 0.266 1.034 1.034 -1.034 -1.034 * Time: 2024-04-03 13:00:00 to 2024-05-05 10:30:00 (3063 samples, mean increment 15 min) * Data Overview Min. Mean Max. Dim. NAs v [m/s] -5.16 -0.02938 9.023 3063x35x4 55400 q NA NA NA 3063x35x4 0 a NA NA NA 3063x35x4 0 g NA NA NA 3063x35x4 0 distance [m] 0.92 9.42 17.92 35 0 pressure [dbar] -0.424 12.077 12.865 3063 0 temperature [°C, ITS-90] 18.73 25.114 38.22 3063 0 salinity [PSS-78] 40 40 40 3063 0 depth [m] 0 11.91 12.7 3063 0 soundSpeed [m/s] 1524 1540.2 1566 3063 0 heading [°] 0.42 338.55 359.78 3063 0 pitch [°] -34.491 1.3262 8.2904 3063 0 roll [°] -32.32 0.66281 5.34 3063 0 headingStd [°] 0 0 0 3063 0 pitchStd [°] 0 0 0 3063 0 rollStd [°] 0 0 0 3063 0 pressureStd 9 16.665 4942 3063 0 xmitCurrent 57 63.69 73 3063 0 xmitVoltage 147 158.49 177 3063 0 ambientTemp 59 80.214 92 3063 0 pressurePlus 76 91.811 93 3063 0 pressureMinus 61 62.317 78 3063 0 attitudeTemp 62 78.464 90 3063 0 attitude [°] 130 130 130 3063 0 contaminationSensor 159 159.22 160 3063 0 * Processing Log - 2024-06-02 12:49:55.600 UTC: `read.adp.rdi(file = f, debug = 1)` [1] "v" "q" "a" [4] "g" "distance" "time" [7] "pressure" "temperature" "salinity" [10] "depth" "soundSpeed" "heading" [13] "pitch" "roll" "headingStd" [16] "pitchStd" "rollStd" "pressureStd" [19] "xmitCurrent" "xmitVoltage" "ambientTemp" [22] "pressurePlus" "pressureMinus" "attitudeTemp" [25] "attitude" "contaminationSensor" ```

[dankelley]

I did some web searching and found a 13 year-old document at https://www.comm-tec.com/Docs/Manuali/RDI/WavesMon%20Quick%20Start%20Guide.pdf called WavesMon v3.08 User’s Guide that has the following on page 59. But it's 13 years old, and I'm not sure it would make sense to start writing new code based on such an old document. (Coding such things is not trivial.)

[dankelley]

@ouhssam it looks as though matlab has code to read waves files. You might want to check that out. See https://pubs.usgs.gov/of/2005/1211/

[dankelley]

As an experiment, I tried running the following code on the data file, but the data disagree with the RDI documentation (at https://www.comm-tec.com/Docs/Manuali/RDI/WavesMon%20Quick%20Start%20Guide.pdf, called WavesMon v3.08 User’s Guide) on byte 9 in the file. Either the documentation is wrong or this is a new format, in addition to the approximately 10 formats listed in the documentation. I suspect the latter, given that the documentation is old.

> library(oce)
> f <- "(REDACTED)"
> buf <- readBin(f, "raw", n = 1e4)
> i <- 1
> if (buf[i] == 0x7f && buf[i + 1] == 0x79) {
+     cat("Got 0x7f 0x79 (for waves type) see wavemon manual section 4.5 table 16\n")
+     i <- i + 2
+     checksumOffset <- readBin(buf[i + 0:1], "integer", n = 1, size = 2)
+     cat("  ", vectorShow(checksumOffset))
+     i <- i + 2
+     spare <- buf[i]
+     cat("  ", vectorShow(spare))
+     i <- i + 1
+     numberOfDataTypes <- readBin(buf[i], "integer", n = 1, size = 1)
+     cat("  ", vectorShow(numberOfDataTypes))
+     i <- i + 1
+     offset <- readBin(buf[i + 0:1], "integer", n = 1, size = 2)
+     cat("  ", vectorShow(offset))
+     buf[offset+2]
+     i <- offset + 1 # or maybe i + 2
+     buf[15] # maybe nbins
+     # NEXT FAILS. is manual wrong, or am I reading it incorrectly?
+     if (buf[i] == 0x01 && buf[i + 1] == 0x03) {
+         cat("First Leader Type at index ", i, "\n")
+     } else {
+         cat("   ERROR: need 0x01 0x03 (first leader) but have 0x", buf[i], " 0x",
+             buf[i + 1], "\n",
+             sep = ""
+         )
+     }
+ }
Got 0x7f 0x79 (for waves type) see wavemon manual section 4.5 table 16
   checksumOffset: 86
   spare: 00
   numberOfDataTypes: 1
   offset: 8
   ERROR: need 0x01 0x03 (first leader)  but have 0x03 0x01

[dankelley]

I've discussed this with co-developer @richardsc and the decision is not to attempt to alter oce to support this file type at this time.

We explored a similar file about a year ago (see sandbox/dk/rdi in this repo) but concluded then that it is more practical for users to try using matlab code or RDI applications for this work.

There are a couple of issues involved here.

1. We do not have access to recent documentation (see previous comments) and so we have no way to know how to handle modern files. My guess, based on my previous comment, is that either the manual I have contains a typo or there is a new format, in addition to the 10 or so formats discussed within the document I have.
2. Developing new code to handle even one of the wave formats will be tricky, involving C++ as well as R code. We need C++ for speed in doing low-level computations involving checksums; R is likely hundreds of times slower than C++. But the package has to be rebuilt entirely for each C++ change, which means something like a 5-minute delay for every test. And that C++ code is quite extensive, running to several hundreds of lines of code, so altering it will be tricky.

We may return to this if a local student needs help with a file. Being local means that we can work closely with the student, to check that whatever results we get in oce match results from matlab. Without such checks, it is quite difficult to support a new binary filetype, working from manuals that are often vague (e.g. they do not state the format used for the bytes in the file -- sometimes integer, sometimes unsigned integer, sometimes floating point). There can be a fair bit of guesswork involved in coding from vague manuals. And, of course, if manuals have errors (see the previous comment for what may be an example of a manual) then we will have a difficult time discovering how to work past those errors, without alternative software to use for comparisons.

@ouhssam in oce, we ask reporters to close issues. I recommend you read this -- and go through the emails that you sent to us privately -- and consider closing this issue.

Sorry we could not help. Dan.

[dankelley]

I think it's not unlikely that the byte-order reversal in the above is a result of an assumption of endianness in the RDI documentation. Nevertheless, I conclude that handling this format would be an excessive amount of work.

[ouhssam]

Thanks a lot. I will have a look on the matlab code.

[dankelley]

PS. yesterday, I asked Teledyn/RDInstruments some questions about the format, because the wavesmon document had quite a few missing elements. The reply indicated that they will not be clarifying the document, because the intension is that users employ the wavesmon software to get text-based files.

The matlab code deal with those text-based files and, since that code is quite old and presumably well-used, I hope you'll have good luck. At the moment, we do not plan to add similar code to oce, preferring instead to focus on common topics for which other solutions are lacking.

Thanks.