Sontek ADV altimetry data not under @data

[koRneel82]

Dear, I wonder if you could provide this parameter when using the oce function (read.adv.sontek.adr)? Best regards

[dankelley]

Can you email a sample file to me at dan.kelley@dal.ca, so I can test? I will likely only need the first 100K or so, if you know how to chop files using unix commands. (I will also share it with @richardsc but neither of us will put it online, so your data will be safely private.)

[dankelley]

Thanks. I have a copy of the file, and can give it to @richardsc when I see him tomorrow. Therefore, you can delete the link, if you wish to (for privacy).

Further on the privacy issue, is it OK if we discuss in comments here some of the results (likely just for the first 2 or 3 samples -- that should be enough). Discussion on github is best because then we can go back and check.

One more thing, relating to privacy. Would it be OK if we extracted the first 2 or 3 samples and used them as a test file that other users and developers could see? We like to have tests built into oce, so that we can catch any new errors that might occur. Again, we would only use a few profiles (partly because large files are not suitable for a test suite).

[koRneel82]

OK for using samples

[dankelley]

@koRneel82 -- I removed your comment that contained a link to a private file, since I've downloaded it already.

A further question: are you able to build oce from the github source? That's where you'll see the new feature (if I can see how to do it). I will likely look at this within the next few days. My first step will be to write a new function called advFileTrim(), which will read an ADV file and write a shortened version. Once I have that, I will extract perhaps the first 3 or so samples from your dataset and proceed to look in the code (and the Sontek docs I have available) to see where to add something to read this field.

Thanks.

[dankelley]

@koRneel82 are you OK with my putting here a comment that states the statistics of your data? I ask because I don't think oce is reading the data correctly, and want to show you why. In case it's not OK, I ask that you do

library(oce)
d <- read.oce("YOUR FILENAME HERE")
summary(d)

and look at the Data Overview section. Notice that the min and max values of heading, pitch, roll and temperature are not physical. (They are more like 2^15, as we expect for a limiting value for a 2-byte integer.) Actually, the mean values of those quantities also seem wrong. Possibly the code is not reading these things right.

PS. I will likely see @richardsc tomorrow, and he knows a lot about this sort of instrument.

[dankelley]

@koRneel82 Also, the velocities look wrong to me. TO see that, read the data (as above) and then do

t<-d[["time"]]
v<-d[["v"]]
par(mfrow=c(3, 1))
look <- seq(1, 500)
oce.plot.ts(t[look],v[look,1])
look <- 500 + look
oce.plot.ts(t[look],v[look,1])
look <- 400000 + look
oce.plot.ts(t[look],v[look,1])

and you'll see that the pattern seems non-physical, both near the start and somewhat near the end of the time-series.

Either the data are odd or (more likely) oce is not reading them correctly.

[dankelley]

@koRneel82 is there any chance you can use some other software to extract the first 3 or so velocities, pressures, etc. and email them privately to me at dan.kelley@dal.ca? I ask because oce seems to be reading the data incorrectly.

It seems that it is reading the metadata correctly (instrument numbers, coordinate system, etc). It also gets times that make sense. However, we get very strange results for the actual data (pressures, temperatures, and velocities) so I think something is wrong. I don't have a theory for this yet, but a clue is that the histograms for temperature, pressure, heading, pitch and roll all look the same (ignoring powers of 10, i.e. scale factors). A test code is below, in the Details section.

The code is based on examination of the Sontek docs from 2001 (citation below; see PDF page 99, print page 87). Oce works for some old files we have at Dalhousie, but when I try to read your data, I get very strange results. I don't know whether this is because of the deficiency in our code or a change in data format, but in either case, if we had some "ground truth" from other software, we could likely find a solution a lot easier.

SonTek/YSI Incorporated. “Argonaut Acoustic Doppler Current Meter Operation Manual Firmware Version 7.9.” SonTek/YSI, May 1, 2001. https://eng.ucmerced.edu/snsjho/files/San_Joaquin/Sensors_and_Loggers/SonTek/SonTek_Argonaut/ArgonautXR.pdf.

**Test code (change the filename)** ```R library(oce) f <- "PUT YOUR FILENAME HERE" d <- read.oce(f) summary(d) # note odd numbers (min/max for 16-bit) t<-d[["time"]] v<-d[["v"]] if (!interactive()) png("adr01_ts.png", unit="in", width=7, height=7, res=200) par(mfrow=c(3, 1)) look <- seq(1, 500) oce.plot.ts(t[look],v[look,1]) look <- 500 + look oce.plot.ts(t[look],v[look,1]) look <- 400000 + look oce.plot.ts(t[look],v[look,1]) if (!interactive()) dev.off() if (!interactive()) png("adr01_hist_%d.png", unit="in", width=7, height=7, res=200) par(mar=c(3,3,1,1), mgp=c(2,0.7,0)) hist(d[["heading"]], breaks=360) hist(d[["pitch"]], breaks=360) hist(d[["roll"]], breaks=360) hist(d[["pressure"]], breaks=360) hist(d[["temperature"]], breaks=360) if (!interactive()) dev.off() ```

[dankelley]

@koRneel82 The manual I have (the one I refer to in the previous comment) does not contain the string "altimet", and also my reading of page 99 and the few pages thereafter has no mention of this. Therefore, I basically don't have any way to guess where the altimeter data will be (before velocity, after, etc). Also there is likely something in a header that tells us whether the dataset has altimeter turned on, but oce is not reading this because the documentation does not mention this possibility.

The only way forward would be to find documentation, perhaps like that mentioned in the previous comment, but for the newer instrument. I cannot blame Sontek/YSI for this, of course. It's been over 20 years, after all, and they obviously will need to change their data structure if they add new sensors.

I am enclosing the documentation I have at the moment, upon which oce is based. If I can get new documentation, likely also with a second-opinion on some data (see previous comment), I can likely change oce to read both the old type and the new type. I cannot promise that, though.

PS. a place to look for manuals might be https://www.ysi.com/customer-support/resource-library but I didn't have much success there because the titles seem to relate to product numbers, and I don't know the proper value for that, in this case.

SonTekYSI Incorporated - 2001 - Argonaut Acoustic Doppler Current Meter Operation .pdf

[dankelley]

PS I'm attaching the manual I used in writing the oce code. Note that this disagrees with the manual I included in the previous comment. I have no way of knowing which is relevant (even for old files).

SonTekYSI Incorporated - 2001 - ADVFieldHydra Operation Manual.pdf

[dankelley]

This issue has been closed, since discussion has stalled and there is insufficient information to proceed with alterations to oce. The original reporter should feel free to reopen it, though, or to open new issues that might be related. Thanks.

PS. This is a standardized reply.

[richardsc]

This isn't explicitly related to the Sontek instrument discussed here, but I noticed last week while working with a Nortek ADV in the lab that the "Vector" software similarly displays a "distance to boundary" but AFAIK it isn't actually saved to the binary data file.

[koRneel82]

Update from my side: I downloaded the SonTek software ViewHydra, and I was able to export all variables (incl. boundary distances):

16 2017 2 6 11 45 52 25.00 7500 183 1467.7 18.82 3.28 0.00 -21.53 -4.54 -1.92 147 152 156 91 96 96 131.0 3.4 0.3 4.52 9.68 0.00010051.417600 5017.661867 6.15 6.42 2.05 3 3 3 4 2 2 0.1 0.1 0.0 0.02 0.05 0.000 565.119753 574.212934 4.4647 2.93126 0.000 30.7280

distance between ADV and boundary is 18.82 cm distance between sampling volume and boundary is 3.28 cm

cheers