search

dankelley / oce

R package for oceanographic processing
http://dankelley.github.io/oce/
GNU General Public License v3.0
142 stars 42 forks source link

subset doesn't work with sentinel V vertical beam data #1837

Closed jessecusack closed 3 years ago

jessecusack commented 3 years ago

Short summary of problem

The subset function doesn't apply to the vertical beam data from sentinel V ADCP such as vv, vq, va etc.

How urgent is this?

Not very...

Reproducible Example

adp0 <- subset(adp, pressure > 100)

# Compare
dim(adp0[['v']])
# with
dim(adp0[['vv']])
dankelley commented 3 years ago

A question for either @jessecusack or @richardsc: can you supply a sample file or, if you are @richardsc, can you assign yourself to this issue and take care of it?

I know we have likely corresponded before about sample data, but I do not keep old emails, and I tend to delete files once issues are closed, for data privacy. Pairing datasets with issues is the first step to solving those issues.

The code change is likely to be simple, and I'm very happy to do it, but it's very hard to work without a sample dataset. We are talking about hundreds (maybe thousands) of lines of code that might be activated by a given dataset. Finding the 3 or so lines of relevant code is obviously a lot easier with a sample file, since then I can insert tracing code to discover the flow of control as the code is run.

Many thanks!

dankelley commented 3 years ago

I think this may work now in the "develop" branch, commit d08fce190afc2121ff400784e84dce219637711f

Below is a reprex, created with reprex::reprex() using the built-in dataset, which does not have a vertical beam. Perhaps @richardsc can take a peak at the code diffs (and @jessecusack can see them too, by clicking on the link at the end of the previous paragraph) to see if what I've done makes sense. Note that I have also altered some nearby existing code, to use the same scheme (which uses fewer lines and should be easier to maintain). Because of this alteration, the new fifth-beam code is also used for the other beams, and that makes me think my solution might be correct ... but only an actual sentinel-V test will confirm that.

``` r library(oce) #> Loading required package: gsw #> Loading required package: testthat #> Loading required package: sf #> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 data(adp) P <- 39 # cutoff pressure for subset deep <- subset(adp, pressure > P) par(mfrow=c(2, 2)) plot(adp, which="19", type="o") plot(adp, which="pressure", type="o") abline(h=P, col=2, lwd=2) plot(deep, which="19", type="o") plot(deep, which="pressure", type="o") abline(h=P, col=2, lwd=2) ``` ![](https://i.imgur.com/QH03JPm.png) Created on 2021-06-12 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)
jessecusack commented 3 years ago

@dankelley I installed the develop branch and did a quick subset test. It now appears to work as expected! Great!

I can ask if the dataset I'm working with could be used as a sample (I am not the owner), however it is currently an active research project so unlikely to be made public soon. It is also rather large (600 MB).

dankelley commented 3 years ago

Oh, thanks -- I won't need a dataset for this issue, if you think the fix has worked.

And if you think that it has worked, please close the issue -- that's the convention we have in oce.

Thanks for the quick response!

jessecusack commented 3 years ago

I did a few more checks with subset that justify reopening this issue.

Subsetting by distance e.g.

subset(adp, distance < 100)

does not subset the vertical beam data. However, vertical beam data seem to be linked with another distance variable vdistance. For my dataset vdistance and distance are not identical (but are very close to identical, one is offset from the other by about 3 cm).

Subsetting by vdistance does not throw an error, but instead just subsets by distance... perhaps something to do with how the code interprets the logic string? I'm not sure, but it seems like a bug.

I think several different behaviours might be reasonable. One would be for subsetting by distance to also apply to the vertical beam data, but using instead the vdistance variable. The other would be to require a second use of subset, applied to vdistance which only changes the vertical data e.g. vv etc. I don't know which is better.

dankelley commented 3 years ago

All of these subsets are individually coded, so I'm not surprised that we have continuing problems. I see why there is a problem with "distance", because the code we have for recognizing a subset by distance (see https://github.com/dankelley/oce/blob/1da5964623d48c9b10b64218e4b815b85a0061a3/R/adp.R#L1058) is searching for the string "distance", and that also captures "vdistance". I can fix that by checking for "vdistance" first, in a block above this one.

Any chance you could post a comment in which you display the "distance" and "vdistance" data? Just do e.g.

d <- read.oce("your data file")
d[["distance"]]
d[["vdistance"]]

in a reprex. (Yes, really, please use reprex::reprex() since it makes things a whole lot easier, even if I don't have a data file. Otherwise, when a bug is reported, the code authors have to also write code to make the error happen.)

dankelley commented 3 years ago

This is going to be a bit slow, without local test data. I hope you can be patient. Please build from github, and try running the distance subset. It should print a bunch of stuff out.

Please examine code screenshot below (which is the code you'll be using after your rebuild from github). If control gets to this spot, then we'll know that this is where I can fix your problem. If not, I'll need to add more print statements throughout the code to try to discover how the problem flow is working. (This R file is 4000 lines long. The file that reads your data is 2000 lines long. There are lots of ways control can meander through such code terrain and the only way I can discover the flow remotely is by printing.)

Notice that line 1078 prints a line you can easily recognize. I'm keen to see the lines above that. This will tell me (a) that I am catching the condition and (b) the vdistance values. I also need to see the distance values (see previous comment). From these things I ought to be able to get subset by distance working. It should likely only take a few more tests like this, so, at perhaps half a day per test, we might get a solution by week's end.

Screen Shot 2021-06-14 at 6 43 28 PM
richardsc commented 3 years ago

I've got kids in the shower and am also trying to clean the kitchen, but just remembered I have some SentinelV data. Hopefully this helps/works for what you need. I'll try and build/run/debug later this evening if I can.

sentinel_adcp_example.zip

dankelley commented 3 years ago

Got it -- thanks, Clark. I'll look this morning.

If I try for subset(adp, distance), that is without making the user supply a vdistance variant, I'll need to figure out how to alter expressions, for the tricky business of evaluating the stated subset partly in the user's context, so e.g. if they have a variable d0 defined, they can do subset(adp,distance<d0), and also in the context of the adp object. This is the work done in lines like

 keep <- eval(expr=substitute(expr=subset, env=environment()), envir=x@data, enclos=parent.frame(2))

which are a bit tricky, e.g. they changed something in R a couple of years ago that forced me to change all those calls. (There are 35 such lines, at least 1 for each data class.)

I do think that a one-step process will be simpler for the user, but if this turns out to be a hard coding task, I'll fall back to a two-step one, where the user is forced to subset on distance and vdistance separately.

dankelley commented 3 years ago

Hm. Here is what I get for Clark's file. Note that read.oce() is not reading vdistance for that file.

@jessecusack can you supply a reprex that indicates to me why you know that vdistance is present in the object as you read it? (Just read it, and use names() twice, like I've done below.). I want to verify that before I can decide what to do next. It may be that the file Clark sent is from a slightly different format, so read.oce() is not reading vdistance. Or maybe, @jessecusack, you used read.adp()? Or, maybe you used read.adp.rdi()? As you can see, providing a reprex prevents a lot of questions later. If I can get a file that is suitable, this should not be a hard thing to fix, I think. But I need such a file, otherwise it's a matter of my making a guess, waiting a day or two for a test, then making a new test, etc.

Note that we don't need (or want) a full file. Just 1M or so should be enough to get several profiles. Likely they are in the air, and therefore not even useful values. Unix and unix-like machines have nice ways to chop out part of a file.

Meantime, I will alter the code from what I did yesterday, because it will break on lots of files, the way it is now. (I was guessing that if vv was defined, then vdistance would be defined. Not so.)

setwd("~/git/oce-issues/18xx/1837")
library(oce)
#> Loading required package: gsw
#> Loading required package: testthat
#> Loading required package: sf
#> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1
if (file.exists("~/git/oce/R/adp.R"))
    source("~/git/oce/R/adp.R")
if (!file.exists("sentinel_adcp_example.pd0"))
    stop("cannot test this, since 'sentinel_adcp_example.pd0' is not present")
d <- read.oce("sentinel_adcp_example.pd0")
#> Got to end of data while trying to read an RDI file (cindex=998294; last7f7f=998290)
#> Warning in read.adp.rdi(file, processingLog = processingLog, ...): skipping the first ensemble (a temporary solution that eases reading of SentinelV files)
#> Warning in read.adp.rdi(file, processingLog = processingLog, ...): A list of unhandled segment codes follows. Several Teledyne RDI manuals
#>   describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014.
#>   Ocean Surveyor/Ocean Observer Technical Manual.
#>   P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf)
#>     Code 0x01 0x0f occurred 444 times
#>     Code 0x00 0x70 occurred 444 times
#>     Code 0x01 0x70 occurred 444 times
#>     Code 0x02 0x70 occurred 444 times
#>     Code 0x00 0x32 occurred 444 times
#>     Code 0x04 0x70 occurred 444 times
#> Warning in read.adp.rdi(file, processingLog = processingLog, ...): length(time)=444 exceeds length(profileStart)=443 so trimming time
#> length(time)=444 exceeds length(profileStart)=443 so trimming time
cat(vectorShow(d[["distance"]])) # 95 elements, highest value 95.92
#> d[["distance"]] [1:95]: 1.92, 2.92, ..., 94.92, 95.92

# Q: where is 'vdistance'?
sort(names(d@data))
#>  [1] "a"                   "ambientTemp"         "attitude"           
#>  [4] "attitudeTemp"        "contaminationSensor" "depth"              
#>  [7] "distance"            "g"                   "heading"            
#> [10] "headingStd"          "pitch"               "pitchStd"           
#> [13] "pressure"            "pressureMinus"       "pressurePlus"       
#> [16] "pressureStd"         "q"                   "roll"               
#> [19] "rollStd"             "salinity"            "soundSpeed"         
#> [22] "temperature"         "time"                "v"                  
#> [25] "va"                  "vdistance"           "vg"                 
#> [28] "vq"                  "vv"                  "xmitCurrent"        
#> [31] "xmitVoltage"
sort(names(d@metadata))
#>  [1] "beamAngle"            "beamConfig"           "beamPattern"         
#>  [4] "bin1Distance"         "binMappingUsed"       "bytesPerEnsemble"    
#>  [7] "cellSize"             "codes"                "coordTransform"      
#> [10] "cpuBoardSerialNumber" "dataOffset"           "depthMean"           
#> [13] "ensembleInFile"       "ensembleNumber"       "errorVelocityMaximum"
#> [16] "falseTargetThresh"    "filename"             "firmwareVersion"     
#> [19] "firmwareVersionMajor" "firmwareVersionMinor" "flags"               
#> [22] "frequency"            "haveActualData"       "headingAlignment"    
#> [25] "headingBias"          "instrumentSubtype"    "instrumentType"      
#> [28] "latitude"             "longitude"            "lowCorrThresh"       
#> [31] "manufacturer"         "numberOfBeams"        "numberOfCells"       
#> [34] "numberOfCodeReps"     "numberOfDataTypes"    "numberOfSamples"     
#> [37] "oceBeamUnspreaded"    "oceCoordinate"        "orientation"         
#> [40] "originalCoordinate"   "percentGdMinimum"     "pingsPerEnsemble"    
#> [43] "profilingMode"        "sensorsAvailable"     "sensorSource"        
#> [46] "serialNumber"         "systemBandwidth"      "systemConfiguration" 
#> [49] "threeBeamUsed"        "tiltUsed"             "transducerDepth"     
#> [52] "transformationMatrix" "transmitLagDistance"  "units"               
#> [55] "vBeamHeader"          "velocityMaximum"      "velocityResolution"  
#> [58] "wpRefLayerAverage"    "xmitPulseLength"

# Next fails, as there is no 'vdistance'
ds <- subset(d, distance < 50)
#> should handle va, vg, vq and vv now.  But we need to debug this first...
#> next is vdistance:
#> Error in print(vdistance): object 'vdistance' not found

Created on 2021-06-15 by the reprex package (v2.0.0)

dankelley commented 3 years ago

Oh, I take that back. I was looking in the wrong place for vdistance. NOTE: on Clark's test file, vdistance is identical to distance, except for a 4.00cm difference. And that's on a range that goes up to 95.92. Therefore, if I cannot figure out the trick to use vdistance, I would not be too bothered about using distance.

dankelley commented 3 years ago

I think I have this working in "develop" commit 36fcedcfa94c5218f8f8cd3f6ea643bd4227a7bb; see the test code by clicking 'Details' below. I have not looked in great detail at the output, but at least the summary() reveals that the dimensions are correct, and the code is relatively self-evident (at least to those who understand eval(), substitute(), environments, etc.).

PS. Following my usual habit, I've retained some commented-out code that I'll delete later. (I like to keep such things to save time, if the modifications don't prove successful.)

``` r # Test for issue 1837 (https://github.com/dankelley/oce/issues/1837) # Must copy CR's sample file here for this to work. setwd("~/git/oce-issues/18xx/1837") library(oce) #> Loading required package: gsw #> Loading required package: testthat #> Loading required package: sf #> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 if (!file.exists("sentinel_adcp_example.pd0")) stop("must put 'sentinel_adcp_example.pd0' here for this test") d <- read.oce("sentinel_adcp_example.pd0") #> Got to end of data while trying to read an RDI file (cindex=998294; last7f7f=998290) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): skipping the first ensemble (a temporary solution that eases reading of SentinelV files) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): A list of unhandled segment codes follows. Several Teledyne RDI manuals #> describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014. #> Ocean Surveyor/Ocean Observer Technical Manual. #> P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf) #> Code 0x01 0x0f occurred 444 times #> Code 0x00 0x70 occurred 444 times #> Code 0x01 0x70 occurred 444 times #> Code 0x02 0x70 occurred 444 times #> Code 0x00 0x32 occurred 444 times #> Code 0x04 0x70 occurred 444 times #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): length(time)=444 exceeds length(profileStart)=443 so trimming time #> length(time)=444 exceeds length(profileStart)=443 so trimming time summary(d) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/Dropbox/oce-working-notes/rdi/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of profiles: 443 #> * Number of cells: 95 #> * Number of beams: 4 #> * Cell size: 1 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.500 0.500 0.500 5.726 0.500 165.500 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.010573 2.923 443x95x4 95 - #> q 0 19.884 170 443x95x4 0 - #> a 33 39.261 123 443x95x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.019293 3.191 443x95 35 - #> vq 1 20.747 126 443x95 0 - #> va 35 39.853 149 443x95 0 - #> vg NA NA NA 0 0 - #> vdistance 1.88 48.88 95.88 95 0 - #> distance [m] 1.92 48.92 95.92 95 0 - #> pressure [dbar] 0.034 0.047115 0.055 443 0 - #> temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - #> salinity [PSS-78] 35 35 35 443 0 - #> depth [m] 0 0.017607 0.1 443 0 - #> soundSpeed [m/s] 1499 1499.5 1500 443 0 - #> heading [°] 327.35 331.39 335.61 443 0 - #> pitch [°] 89.277 89.305 89.337 443 0 - #> roll [°] 89.28 89.308 89.34 443 0 - #> headingStd [°] 0 0 0 443 0 - #> pitchStd [°] 0 0 0 443 0 - #> rollStd [°] 0 0 0 443 0 - #> pressureStd 0 0 0 443 0 - #> xmitCurrent 0 0 0 443 0 - #> xmitVoltage 189 189.8 190 443 0 - #> ambientTemp 0 0 0 443 0 - #> pressurePlus 0 0 0 443 0 - #> pressureMinus 0 0 0 443 0 - #> attitudeTemp 0 0 0 443 0 - #> attitude [°] 0 0 0 443 0 - #> contaminationSensor 0 0 0 443 0 - #> #> * Processing Log #> #> - 2021-06-15 11:44:17 UTC: `read.oce("sentinel_adcp_example.pd0")` ds <- subset(d, distance < 50) summary(ds) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/Dropbox/oce-working-notes/rdi/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of profiles: 443 #> * Number of cells: 49 #> * Number of beams: 4 #> * Cell size: 1 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.500 0.500 0.500 5.726 0.500 165.500 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 49, centered at 1.920 m to 49.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.023832 2.923 443x49x4 51 - #> q 0 21.237 170 443x49x4 0 - #> a 33 40.394 123 443x49x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.024954 3.191 443x49 17 - #> vq 1 22.412 126 443x49 0 - #> va 35 41.405 149 443x49 0 - #> vdistance 1.88 25.88 49.88 49 0 - #> distance [m] 1.92 25.92 49.92 49 0 - #> pressure [dbar] 0.034 0.047115 0.055 443 0 - #> temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - #> salinity [PSS-78] 35 35 35 443 0 - #> depth [m] 0 0.017607 0.1 443 0 - #> soundSpeed [m/s] 1499 1499.5 1500 443 0 - #> heading [°] 327.35 331.39 335.61 443 0 - #> pitch [°] 89.277 89.305 89.337 443 0 - #> roll [°] 89.28 89.308 89.34 443 0 - #> headingStd [°] 0 0 0 443 0 - #> pitchStd [°] 0 0 0 443 0 - #> rollStd [°] 0 0 0 443 0 - #> pressureStd 0 0 0 443 0 - #> xmitCurrent 0 0 0 443 0 - #> xmitVoltage 189 189.8 190 443 0 - #> ambientTemp 0 0 0 443 0 - #> pressurePlus 0 0 0 443 0 - #> pressureMinus 0 0 0 443 0 - #> attitudeTemp 0 0 0 443 0 - #> attitude [°] 0 0 0 443 0 - #> contaminationSensor 0 0 0 443 0 - #> #> * Processing Log #> #> - 2021-06-15 11:44:17 UTC: `read.oce("sentinel_adcp_example.pd0")` #> - 2021-06-15 11:44:17 UTC: `subset.adp(x, subset=distance < 50)` ``` Created on 2021-06-15 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)
jessecusack commented 3 years ago

@dankelley I tested your latest changes. It looks like subsetting by distance and pressure is now working correctly. The one remaining thing is all the other metadata variables like xmitCurrent etc. are left unaltered by the subset, is this intended behaviour?

Reprex below.

``` r # Load libraries library(oce) #> Loading required package: gsw #> Loading required package: testthat #> Loading required package: sf #> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 # library(ncdf4) # library(yaml) # source("ncwrite.R") # Set parameters # Parameters file <- "~/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0" pmin <- 125. # cut off pressure [dbar] dmax <- 150. # cut off distance [m] lat <- 56.8370392 lon <- -132.3574123 dec <- 19.32 # magnetic declination ori <- "upward" # orientation # Load data and set orientation file <- path.expand(file) adp <- read.adp(file, latitude = lat, longitude = lon) #> Increasing ensembles,times,sec100s storage to 150000 elements ... #> Increasing ensembles,times,sec100s storage to 225000 elements ... #> Increasing ensembles,times,sec100s storage to 337500 elements ... #> Increasing ensembles,times,sec100s storage to 506250 elements ... #> Increasing ensembles,times,sec100s storage to 759375 elements ... #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): skipping the first ensemble (a temporary solution that eases reading of SentinelV files) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): A list of unhandled segment codes follows. Several Teledyne RDI manuals #> describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014. #> Ocean Surveyor/Ocean Observer Technical Manual. #> P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf) #> Code 0x01 0x0f occurred 517369 times #> Code 0x00 0x70 occurred 517369 times #> Code 0x01 0x70 occurred 517369 times #> Code 0x02 0x70 occurred 517369 times #> Code 0x00 0x32 occurred 517369 times #> Code 0x04 0x70 occurred 517369 times #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): the instrument orientation is not constant. The user is advised to determine #> the orientation during the relevant measurement phase, and to set this into the #> object with e.g. #> adp<-oceSetMetadata(adp,'orientation','upward') #> in case conversion to ENU is to be done later. #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): length(time)=517369 exceeds length(profileStart)=517368 so trimming time #> length(time)=517369 exceeds length(profileStart)=517368 so trimming time adp <- oceSetMetadata(adp, 'orientation', ori) summary(adp) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 17312 #> * Firmware version: 47.20 #> * Source filename: ``/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0`` #> * Location: 56.83704 N, -132.35741 E #> * Number of profiles: 517368 #> * Number of cells: 40 #> * Number of beams: 4 #> * Cell size: 4 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 2.5 2.5 2.5 2.5 2.5 7.5 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:517368]: 2, 3, ..., 451825, 451826 #> * Cells: 40, centered at 5.930 m to 161.930 m, spaced by 4.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 25 #> Orientation: upward #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1642 -1.1722 0.0066 -0.0015 #> -0.0084 -0.0003 -1.1638 1.1776 #> 0.2780 0.2749 0.2793 0.2739 #> 0.8317 0.8224 -0.8347 -0.8187 #> * Time ranges from 2018-09-01 23:34:59 to 2018-09-16 22:52:16 with 517368 samples and mean increment 2.500039 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.138 0.027348 3.159 517368x40x4 3207 - #> q 0 99.172 139 517368x40x4 0 - #> a 34 99.516 255 517368x40x4 0 - #> g NA NA NA 0 0 - #> vv -3.488 0.032963 3.481 517368x40 1796 - #> vq 0 92.755 140 517368x40 0 - #> va 40 94.741 255 517368x40 0 - #> vg NA NA NA 0 0 - #> vdistance 5.89 83.89 161.89 40 0 - #> distance [m] 5.93 83.93 161.93 40 0 - #> pressure [dbar] 0 124.52 137.65 517368 0 - #> temperature [°C, ITS-90] 4.16 7.7986 21.45 517368 0 - #> salinity [PSS-78] 35 35 35 517368 0 - #> depth [m] 0 124 137.1 517368 0 - #> soundSpeed [m/s] 1467 1483.9 1525 517368 0 - #> heading [°] 0 58.544 359.99 517368 0 - #> pitch [°] -88.91 1.9317 85.009 517368 0 - #> roll [°] -87.19 3.8484 89.1 517368 0 - #> headingStd [°] 0 0 0 517368 0 - #> pitchStd [°] 0 0 0 517368 0 - #> rollStd [°] 0 0 0 517368 0 - #> pressureStd 0 0 0 517368 0 - #> xmitCurrent 0 0 0 517368 0 - #> xmitVoltage 152 163.21 186 517368 0 - #> ambientTemp 0 0 0 517368 0 - #> pressurePlus 0 0 0 517368 0 - #> pressureMinus 0 0 0 517368 0 - #> attitudeTemp 0 0 0 517368 0 - #> attitude [°] 0 0 0 517368 0 - #> contaminationSensor 0 0 0 517368 0 - #> #> * Processing Log #> #> - 2021-06-15 18:59:29 UTC: `read.oce("/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0", ...)` #> - 2021-06-15 18:59:29 UTC: `oceSetMetadata(object = adp, name = "orientation", value = ori)` # Subset data using pressure adp <- subset(adp, pressure > pmin) summary(adp) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 17312 #> * Firmware version: 47.20 #> * Source filename: ``/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0`` #> * Location: 56.83704 N, -132.35741 E #> * Number of profiles: 479496 #> * Number of cells: 40 #> * Number of beams: 4 #> * Cell size: 4 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 2.5 2.5 2.5 2.5 2.5 5.0 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:517368]: 2, 3, ..., 451825, 451826 #> * Cells: 40, centered at 5.930 m to 161.930 m, spaced by 4.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 25 #> Orientation: upward #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1642 -1.1722 0.0066 -0.0015 #> -0.0084 -0.0003 -1.1638 1.1776 #> 0.2780 0.2749 0.2793 0.2739 #> 0.8317 0.8224 -0.8347 -0.8187 #> * Time ranges from 2018-09-02 01:16:36 to 2018-09-15 22:15:41 with 479496 samples and mean increment 2.500016 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.092 0.029633 3.096 479496x40x4 445 - #> q 0 106.47 139 479496x40x4 0 - #> a 35 104.35 238 479496x40x4 0 - #> g NA NA NA 0 0 - #> vv -3.291 0.035272 3.458 479496x40 286 - #> vq 0 99.555 140 479496x40 0 - #> va 43 98.832 235 479496x40 0 - #> vdistance 5.89 83.89 161.89 40 0 - #> distance [m] 5.93 83.93 161.93 40 0 - #> pressure [dbar] 125.2 134.34 137.65 479496 0 - #> temperature [°C, ITS-90] 7.62 7.7506 8.13 479496 0 - #> salinity [PSS-78] 35 35 35 479496 0 - #> depth [m] 124.7 133.78 137.1 479496 0 - #> soundSpeed [m/s] 1467 1483.9 1525 517368 0 - #> heading [°] 0 49.479 359.99 479496 0 - #> pitch [°] -8.0189 -0.16318 4.761 479496 0 - #> roll [°] -14.15 3.6996 12.87 479496 0 - #> headingStd [°] 0 0 0 517368 0 - #> pitchStd [°] 0 0 0 517368 0 - #> rollStd [°] 0 0 0 517368 0 - #> pressureStd 0 0 0 517368 0 - #> xmitCurrent 0 0 0 517368 0 - #> xmitVoltage 152 163.21 186 517368 0 - #> ambientTemp 0 0 0 517368 0 - #> pressurePlus 0 0 0 517368 0 - #> pressureMinus 0 0 0 517368 0 - #> attitudeTemp 0 0 0 517368 0 - #> attitude [°] 0 0 0 517368 0 - #> contaminationSensor 0 0 0 517368 0 - #> #> * Processing Log #> #> - 2021-06-15 18:59:29 UTC: `read.oce("/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0", ...)` #> - 2021-06-15 18:59:29 UTC: `oceSetMetadata(object = adp, name = "orientation", value = ori)` #> - 2021-06-15 18:59:46 UTC: `subset.adp(x, subset=pressure > pmin)` # Subset data using distance adp <- subset(adp, distance < dmax) summary(adp) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 17312 #> * Firmware version: 47.20 #> * Source filename: ``/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0`` #> * Location: 56.83704 N, -132.35741 E #> * Number of profiles: 479496 #> * Number of cells: 37 #> * Number of beams: 4 #> * Cell size: 4 m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 2.5 2.5 2.5 2.5 2.5 5.0 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:517368]: 2, 3, ..., 451825, 451826 #> * Cells: 37, centered at 5.930 m to 149.930 m, spaced by 4.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 #> Slantwise Angle: 25 #> Orientation: upward #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1642 -1.1722 0.0066 -0.0015 #> -0.0084 -0.0003 -1.1638 1.1776 #> 0.2780 0.2749 0.2793 0.2739 #> 0.8317 0.8224 -0.8347 -0.8187 #> * Time ranges from 2018-09-02 01:16:36 to 2018-09-15 22:15:41 with 479496 samples and mean increment 2.500016 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.063 0.034992 3.096 479496x37x4 137 - #> q 0 109.01 139 479496x37x4 0 - #> a 36 106.95 238 479496x37x4 0 - #> g NA NA NA 0 0 - #> vv -3.291 0.041211 3.458 479496x37 139 - #> vq 0 102.46 140 479496x37 0 - #> va 44 101.12 235 479496x37 0 - #> vdistance 5.89 77.89 149.89 37 0 - #> distance [m] 5.93 77.93 149.93 37 0 - #> pressure [dbar] 125.2 134.34 137.65 479496 0 - #> temperature [°C, ITS-90] 7.62 7.7506 8.13 479496 0 - #> salinity [PSS-78] 35 35 35 479496 0 - #> depth [m] 124.7 133.78 137.1 479496 0 - #> soundSpeed [m/s] 1467 1483.9 1525 517368 0 - #> heading [°] 0 49.479 359.99 479496 0 - #> pitch [°] -8.0189 -0.16318 4.761 479496 0 - #> roll [°] -14.15 3.6996 12.87 479496 0 - #> headingStd [°] 0 0 0 517368 0 - #> pitchStd [°] 0 0 0 517368 0 - #> rollStd [°] 0 0 0 517368 0 - #> pressureStd 0 0 0 517368 0 - #> xmitCurrent 0 0 0 517368 0 - #> xmitVoltage 152 163.21 186 517368 0 - #> ambientTemp 0 0 0 517368 0 - #> pressurePlus 0 0 0 517368 0 - #> pressureMinus 0 0 0 517368 0 - #> attitudeTemp 0 0 0 517368 0 - #> attitude [°] 0 0 0 517368 0 - #> contaminationSensor 0 0 0 517368 0 - #> #> * Processing Log #> #> - 2021-06-15 18:59:29 UTC: `read.oce("/Users/jmcusack/Dropbox/LeConte/Data/ocean/september2018/raw/moorings/ABLE_Sentinel/ADCP/LeConte S-ABLE Sept2018 20180901T233454.pd0", ...)` #> - 2021-06-15 18:59:29 UTC: `oceSetMetadata(object = adp, name = "orientation", value = ori)` #> - 2021-06-15 18:59:46 UTC: `subset.adp(x, subset=pressure > pmin)` #> - 2021-06-15 19:00:01 UTC: `subset.adp(x, subset=distance < dmax)` ``` Created on 2021-06-15 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)
dankelley commented 3 years ago

Thanks for the reprex, and the question. I'm a bit too tired (as a morning person in an eastern timezone) to go into the code again. I am not actually too sure what a lot of the other things are, to be frank.

Your comment has two issues, I think, one for subset by pressure and another for subset by distance. These are very different things, of course, and when I look at this tomorrow I'll try to see what's going on in each.
In the meantime, my notes are as follows ... restricted to the subset-by-pressure problem.

I don't think this will take long to fix. I'll be working with the test file that Clark provided, so I can have a think-edit-check cycle that is measured in seconds, not hours.

**subset-by-pressure case** I see e.g. as below. This tells me that soundspeed is not being trimmed, nor heading, etc. I won't bother trying to write this all out in a sentence. All I want to do at the moment is to give myself a reminder of the part of your reprex that I will focus on. ``` #> pressure [dbar] 125.2 134.34 137.65 479496 0 - #> temperature [°C, ITS-90] 7.62 7.7506 8.13 479496 0 - #> salinity [PSS-78] 35 35 35 479496 0 - #> depth [m] 124.7 133.78 137.1 479496 0 - #> soundSpeed [m/s] 1467 1483.9 1525 517368 0 - #> heading [°] 0 49.479 359.99 479496 0 - #> pitch [°] -8.0189 -0.16318 4.761 479496 0 - #> roll [°] -14.15 3.6996 12.87 479496 0 - #> headingStd [°] 0 0 0 517368 0 - #> pitchStd [°] 0 0 0 517368 0 - #> rollStd [°] 0 0 0 517368 0 - #> pressureStd 0 0 0 517368 0 - #> xmitCurrent 0 0 0 517368 0 - #> xmitVoltage 152 163.21 186 517368 0 - #> ambientTemp 0 0 0 517368 0 - #> pressurePlus 0 0 0 517368 0 - #> pressureMinus 0 0 0 517368 0 - #> attitudeTemp 0 0 0 517368 0 - #> attitude [°] 0 0 0 517368 0 - #> contaminationSensor 0 0 0 517368 0 ```
dankelley commented 3 years ago

I see the problem: I am working through data items that were known to us when we wrote the original code, many years ago. The snapshot illustrates. I'll change it so that I go through all the items in the data slot. I may need to think a bit about this, because e.g. suppose we have N profiles. For a subset by pressure, I create a logical called keep of length N. But we can only apply that to things that are keyed to profile. I think most things are, but I'm not completely sure of that. I think what I'll do is e.g.

This is for pressure subsets. A similar thing will be needed for a time subset (and there are likely other subsets that are similar).

Once I get them done, I'll look at the distance subsets.

Screen Shot 2021-06-15 at 4 21 45 PM
dankelley commented 3 years ago

I have tackled the subset-by-pressure case in commit 94ebb08dcb8d94778ebf270f7c51ac902f17807a of the "develop" branch. A test code is 1837a.R at https://github.com/dankelley/oce-issues/tree/main/18xx/1837, and it's results are shown in the Details below. (Readers should git-clone the oce-issues repository, so they can reproduce this test.)

I will look at subset-by-distance next. (Here, as so often, I really wish GH had threaded comments, but I ask that readers use thumbs-up or thumbs-down here, to indicate whether they think this worked. If you make a new issue referring to this, please include a link to this comment, so at least readers can see what you're referring to.)

NOTE: the new code is heavily instrumented with oceDebug() calls, so using e.g. subset(d,pressure<median(d[["pressure"]], debug=1) will produce information on each data item, as it is examined.

In the Details, notice that wherever 443 appears in the summary for 'd', 188 appears in the summary for 'dsp'. If that were not true, we'd know that the subset() was failing. (By logic, the reverse need not be true, but examination of the code by clicking the link at the top of this issue comment should help readers to judge the correctness of the update.)

The output from 1837a.R follows. (You may reproduce this by typing 'make' in this directory.) ``` R version 4.1.0 (2021-05-18) -- "Camp Pontanezen" Copyright (C) 2021 The R Foundation for Statistical Computing Platform: x86_64-apple-darwin17.0 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > # FILE: oce-issues/18xx/1837/1837a.R > # > # PURPOSE: test subset-by-pressure or issue "subset doesn't work with sentinel > # V vertical beam data" (https://github.com/dankelley/oce/issues/1837), using a > # dataset provided Clark Richards. > # > # CHECK: In the summaries, is there uniform reduction of length, first index, > # etc., as appropriate to each variable type? > > library(oce) Loading required package: gsw Loading required package: testthat Loading required package: sf Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 > > d <- read.oce("sentinel_adcp_example.pd0") Got to end of data while trying to read an RDI file (cindex=998294; last7f7f=998290) length(time)=444 exceeds length(profileStart)=443 so trimming time Warning messages: 1: In read.adp.rdi(file, processingLog = processingLog, ...) : skipping the first ensemble (a temporary solution that eases reading of SentinelV files) 2: In read.adp.rdi(file, processingLog = processingLog, ...) : A list of unhandled segment codes follows. Several Teledyne RDI manuals describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014. Ocean Surveyor/Ocean Observer Technical Manual. P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf) Code 0x01 0x0f occurred 444 times Code 0x00 0x70 occurred 444 times Code 0x01 0x70 occurred 444 times Code 0x02 0x70 occurred 444 times Code 0x00 0x32 occurred 444 times Code 0x04 0x70 occurred 444 times 3: In read.adp.rdi(file, processingLog = processingLog, ...) : length(time)=444 exceeds length(profileStart)=443 so trimming time > summary(d) ADP Summary ----------- * Instrument: adcp * Serial number: 125 * Firmware version: 47.19 * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` * Location: unknown latitude, unknown longitude * Number of profiles: 443 * Number of cells: 95 * Number of beams: 4 * Cell size: 1 m * Summary of times between profiles: Min. 1st Qu. Median Mean 3rd Qu. Max. 0.500 0.500 0.500 5.726 0.500 165.500 * Frequency: 250 kHz * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m * Coordinate system: beam [originally], beam [presently] * Beams:: Number: 4 Slantwise Angle: 25 Orientation: downward:443 Unspreaded: FALSE * Transformation matrix:: 1.1844 -1.1763 0.0047 -0.0099 -0.0043 0.0103 -1.1878 1.1783 0.2743 0.2774 0.2742 0.2778 0.8311 0.8405 -0.8299 -0.8409 * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s * Data Overview Min. Mean Max. Dim. NAs OriginalName v [m/s] -3.145 -0.010573 2.923 443x95x4 95 - q 0 19.884 170 443x95x4 0 - a 33 39.261 123 443x95x4 0 - g NA NA NA 0 0 - vv -3.09 -0.019293 3.191 443x95 35 - vq 1 20.747 126 443x95 0 - va 35 39.853 149 443x95 0 - vg NA NA NA 0 0 - vdistance 1.88 48.88 95.88 95 0 - distance [m] 1.92 48.92 95.92 95 0 - pressure [dbar] 0.034 0.047115 0.055 443 0 - temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - salinity [PSS-78] 35 35 35 443 0 - depth [m] 0 0.017607 0.1 443 0 - soundSpeed [m/s] 1499 1499.5 1500 443 0 - heading [°] 327.35 331.39 335.61 443 0 - pitch [°] 89.277 89.305 89.337 443 0 - roll [°] 89.28 89.308 89.34 443 0 - headingStd [°] 0 0 0 443 0 - pitchStd [°] 0 0 0 443 0 - rollStd [°] 0 0 0 443 0 - pressureStd 0 0 0 443 0 - xmitCurrent 0 0 0 443 0 - xmitVoltage 189 189.8 190 443 0 - ambientTemp 0 0 0 443 0 - pressurePlus 0 0 0 443 0 - pressureMinus 0 0 0 443 0 - attitudeTemp 0 0 0 443 0 - attitude [°] 0 0 0 443 0 - contaminationSensor 0 0 0 443 0 - * Processing Log - 2021-06-16 10:46:39.900 UTC: `read.oce("sentinel_adcp_example.pd0")` > > dsp <- subset(d, pressure < median(d[["pressure"]])) > summary(dsp) ADP Summary ----------- * Instrument: adcp * Serial number: 125 * Firmware version: 47.19 * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` * Location: unknown latitude, unknown longitude * Number of profiles: 188 * Number of cells: 95 * Number of beams: 4 * Cell size: 1 m * Summary of times between profiles: Min. 1st Qu. Median Mean 3rd Qu. Max. 0.50 0.50 1.00 13.53 1.50 180.50 * Frequency: 250 kHz * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m * Coordinate system: beam [originally], beam [presently] * Beams:: Number: 4 Slantwise Angle: 25 Orientation: downward:443 Unspreaded: FALSE * Transformation matrix:: 1.1844 -1.1763 0.0047 -0.0099 -0.0043 0.0103 -1.1878 1.1783 0.2743 0.2774 0.2742 0.2778 0.8311 0.8405 -0.8299 -0.8409 * Time ranges from 2018-04-21 00:00:01.5 to 2018-04-21 00:42:11 with 188 samples and mean increment 13.52674 s * Data Overview Min. Mean Max. Dim. NAs OriginalName v [m/s] -3.145 -0.012452 2.842 188x95x4 41 - q 0 19.887 169 188x95x4 0 - a 33 39.29 123 188x95x4 0 - g NA NA NA 0 0 - vv -3.09 -0.013172 2.528 188x95 14 - vq 1 20.725 126 188x95 0 - va 35 39.87 149 188x95 0 - vg NA NA NA 0 0 - vdistance 1.88 48.88 95.88 95 0 - distance [m] 1.92 48.92 95.92 95 0 - pressure [dbar] 0.034 0.043888 0.046 188 0 - temperature [°C, ITS-90] 12.49 12.796 12.98 188 0 - salinity [PSS-78] 35 35 35 188 0 - depth [m] 0 0 0 188 0 - soundSpeed [m/s] 1499 1499.6 1500 188 0 - heading [°] 327.35 331.33 335.21 188 0 - pitch [°] 89.277 89.303 89.337 188 0 - roll [°] 89.28 89.305 89.34 188 0 - headingStd [°] 0 0 0 188 0 - pitchStd [°] 0 0 0 188 0 - rollStd [°] 0 0 0 188 0 - pressureStd 0 0 0 188 0 - xmitCurrent 0 0 0 188 0 - xmitVoltage 189 189.91 190 188 0 - ambientTemp 0 0 0 188 0 - pressurePlus 0 0 0 188 0 - pressureMinus 0 0 0 188 0 - attitudeTemp 0 0 0 188 0 - attitude [°] 0 0 0 188 0 - contaminationSensor 0 0 0 188 0 - * Processing Log - 2021-06-16 10:46:39.900 UTC: `read.oce("sentinel_adcp_example.pd0")` - 2021-06-16 10:46:39.950 UTC: `subset.adp(x, subset=pressure < median(d[["pressure"]]))` > > ```
dankelley commented 3 years ago

Oh, hang on. Now that I'm looking at subset-by-distance, I see that these data can have e.g. @metadata$flags$v, so I'll need to assure that this works throughout. (Lots of this code is over a decade old, and so the data structure is not in my head!)

PS. I have asked @richardsc for permission to put his sample file into the oce repo (but not the CRAN tarball) so that I can hardwire tests on things like this. (A code test is self-evident, unlike one of dozens of comments in one of hundreds of closed issues...)

dankelley commented 3 years ago

Oh, hang on^2. I see that my test dataset does not have any @metadata$flags content, so I won't be able to test with that. Therefore, I am not going to bother with this ... non test, nulla codice

dankelley commented 3 years ago

I think things work now for subset-by-pressure (see Details)

``` r setwd("~/git/oce-issues/18xx/1837") # PURPOSE: test subset-by-pressure or issue "subset doesn't work with sentinel # V vertical beam data" (https://github.com/dankelley/oce/issues/1837), using a # dataset provided Clark Richards. # # CHECK: In the summaries, is there uniform reduction of length, first index, # etc., as appropriate to each variable type? library(oce) #> Loading required package: gsw #> Loading required package: testthat #> Loading required package: sf #> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 d <- read.oce("sentinel_adcp_example.pd0") #> Got to end of data while trying to read an RDI file (cindex=998294; last7f7f=998290) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): skipping the first ensemble (a temporary solution that eases reading of SentinelV files) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): A list of unhandled segment codes follows. Several Teledyne RDI manuals #> describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014. #> Ocean Surveyor/Ocean Observer Technical Manual. #> P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf) #> Code 0x01 0x0f occurred 444 times #> Code 0x00 0x70 occurred 444 times #> Code 0x01 0x70 occurred 444 times #> Code 0x02 0x70 occurred 444 times #> Code 0x00 0x32 occurred 444 times #> Code 0x04 0x70 occurred 444 times #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): length(time)=444 exceeds length(profileStart)=443 so trimming time #> length(time)=444 exceeds length(profileStart)=443 so trimming time summary(d) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of beams: 4 #> * Number of profiles: 443 #> * Number of cells: 95 #> * Number of beams: 4 #> * Cell size: 1m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.500 0.500 0.500 5.726 0.500 165.500 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 slanted, plus 1 central #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.010573 2.923 443x95x4 95 - #> q 0 19.884 170 443x95x4 0 - #> a 33 39.261 123 443x95x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.019293 3.191 443x95 35 - #> vq 1 20.747 126 443x95 0 - #> va 35 39.853 149 443x95 0 - #> vg NA NA NA 0 0 - #> vdistance 1.88 48.88 95.88 95 0 - #> distance [m] 1.92 48.92 95.92 95 0 - #> pressure [dbar] 0.034 0.047115 0.055 443 0 - #> temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - #> salinity [PSS-78] 35 35 35 443 0 - #> depth [m] 0 0.017607 0.1 443 0 - #> soundSpeed [m/s] 1499 1499.5 1500 443 0 - #> heading [°] 327.35 331.39 335.61 443 0 - #> pitch [°] 89.277 89.305 89.337 443 0 - #> roll [°] 89.28 89.308 89.34 443 0 - #> headingStd [°] 0 0 0 443 0 - #> pitchStd [°] 0 0 0 443 0 - #> rollStd [°] 0 0 0 443 0 - #> pressureStd 0 0 0 443 0 - #> xmitCurrent 0 0 0 443 0 - #> xmitVoltage 189 189.8 190 443 0 - #> ambientTemp 0 0 0 443 0 - #> pressurePlus 0 0 0 443 0 - #> pressureMinus 0 0 0 443 0 - #> attitudeTemp 0 0 0 443 0 - #> attitude [°] 0 0 0 443 0 - #> contaminationSensor 0 0 0 443 0 - #> #> * Processing Log #> #> - 2021-06-16 17:34:27 UTC: `read.oce("sentinel_adcp_example.pd0")` keep <- d[["pressure"]] < median(d[["pressure"]]) dsp <- subset(d, pressure < median(d[["pressure"]])) summary(dsp) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of beams: 4 #> * Number of profiles: 188 #> * Number of cells: 95 #> * Number of beams: 4 #> * Cell size: 1m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.50 0.50 1.00 13.53 1.50 180.50 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 slanted, plus 1 central #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 00:00:01 to 2018-04-21 00:42:11 with 188 samples and mean increment 13.52674 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.012452 2.842 188x95x4 41 - #> q 0 19.887 169 188x95x4 0 - #> a 33 39.29 123 188x95x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.013172 2.528 188x95 14 - #> vq 1 20.725 126 188x95 0 - #> va 35 39.87 149 188x95 0 - #> vg NA NA NA 0 0 - #> vdistance 1.88 48.88 95.88 95 0 - #> distance [m] 1.92 48.92 95.92 95 0 - #> pressure [dbar] 0.034 0.043888 0.046 188 0 - #> temperature [°C, ITS-90] 12.49 12.796 12.98 188 0 - #> salinity [PSS-78] 35 35 35 188 0 - #> depth [m] 0 0 0 188 0 - #> soundSpeed [m/s] 1499 1499.6 1500 188 0 - #> heading [°] 327.35 331.33 335.21 188 0 - #> pitch [°] 89.277 89.303 89.337 188 0 - #> roll [°] 89.28 89.305 89.34 188 0 - #> headingStd [°] 0 0 0 188 0 - #> pitchStd [°] 0 0 0 188 0 - #> rollStd [°] 0 0 0 188 0 - #> pressureStd 0 0 0 188 0 - #> xmitCurrent 0 0 0 188 0 - #> xmitVoltage 189 189.91 190 188 0 - #> ambientTemp 0 0 0 188 0 - #> pressurePlus 0 0 0 188 0 - #> pressureMinus 0 0 0 188 0 - #> attitudeTemp 0 0 0 188 0 - #> attitude [°] 0 0 0 188 0 - #> contaminationSensor 0 0 0 188 0 - #> #> * Processing Log #> #> - 2021-06-16 17:34:27 UTC: `read.oce("sentinel_adcp_example.pd0")` #> - 2021-06-16 17:34:27 UTC: `subset.adp(x, subset=pressure < median(d[["pressure"]]))` library(testthat) expect_equal(sum(keep), dim(dsp[["v"]])[1]) expect_equal(sum(keep), length(dsp[["time"]])) for (slant in c("v", "a")) { # "g" is NULL expect_equal(sum(keep), dim(dsp[[slant]])[1]) } # Details (d) str(d@metadata,2) #> List of 59 #> $ units :List of 14 #> ..$ v :List of 2 #> ..$ distance :List of 2 #> ..$ pressure :List of 2 #> ..$ salinity :List of 2 #> ..$ temperature:List of 2 #> ..$ soundSpeed :List of 2 #> ..$ heading :List of 2 #> ..$ pitch :List of 2 #> ..$ roll :List of 2 #> ..$ headingStd :List of 2 #> ..$ pitchStd :List of 2 #> ..$ rollStd :List of 2 #> ..$ attitude :List of 2 #> ..$ depth :List of 2 #> $ flags : list() #> $ oceCoordinate : chr "beam" #> $ orientation : chr [1:443] "downward" "downward" "downward" "downward" ... #> $ instrumentType : chr "adcp" #> $ instrumentSubtype : chr "sentinelV" #> $ firmwareVersionMajor: int 47 #> $ firmwareVersionMinor: int 19 #> $ firmwareVersion : chr "47.19" #> $ bytesPerEnsemble : int 2424 #> $ systemConfiguration : chr "01001010-01010111" #> $ frequency : num 250 #> $ beamAngle : num 25 #> $ beamPattern : chr "convex" #> $ beamConfig : chr "5 beam janus" #> $ numberOfDataTypes : int 14 #> $ dataOffset : int [1:14] 34 94 160 922 1304 1686 1726 1918 2015 2112 ... #> $ codes : raw [1:16, 1:2] 7f 00 80 00 ... #> $ numberOfBeams : int 4 #> $ numberOfCells : int 95 #> $ pingsPerEnsemble : int 1 #> $ cellSize : num 1 #> $ transducerDepth : int 0 #> $ profilingMode : int 1 #> $ lowCorrThresh : int 0 #> $ numberOfCodeReps : int 3 #> $ percentGdMinimum : int 0 #> $ errorVelocityMaximum: int 0 #> $ coordTransform : chr "00000000" #> $ originalCoordinate : chr "beam" #> $ tiltUsed : logi FALSE #> $ threeBeamUsed : logi FALSE #> $ binMappingUsed : logi FALSE #> $ headingAlignment : num 0 #> $ headingBias : num 0 #> $ sensorSource : chr "01111101" #> $ sensorsAvailable : chr "01111101" #> $ bin1Distance : num 1.92 #> $ xmitPulseLength : num 1.37 #> $ wpRefLayerAverage : int 0 #> $ falseTargetThresh : int -1 #> $ transmitLagDistance : int 47 #> $ cpuBoardSerialNumber: int [1:8] 48 50 97 48 52 51 53 49 #> $ systemBandwidth : int 0 #> $ serialNumber : int 125 #> $ haveActualData : logi TRUE #> $ vBeamHeader :List of 22 #> ..$ numberOfVCells : int 95 #> ..$ verticalPings : int 1 #> ..$ depthCellSize : num 1 #> ..$ firstCellRange : num 1.88 #> ..$ verticalMode : int 1 #> ..$ verticalTransmit : num 1.3 #> ..$ verticalLagLength : num 0.47 #> ..$ transmitCodeElements : int 102 #> ..$ verticalRssiThreshold : int 0 #> ..$ verticalShallowBin : int 0 #> ..$ verticalStartBin : int 0 #> ..$ verticalShallowRssiBin: int 0 #> ..$ maxCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ pingOffsetTime : int 250 #> ..$ surfSpare1 : int 0 #> ..$ depthScreen : int 0 #> ..$ percentGoodThreshold : int 0 #> ..$ verticalDOproofing : int 0 #> ..$ vSeriesConfig :List of 6 #> ..$ vSeriesPingSetup :List of 11 #> $ ensembleNumber : num [1:443] 2 3 4 5 6 7 8 9 10 11 ... #> $ manufacturer : chr "teledyne rdi" #> $ filename : chr "/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0" #> $ longitude : logi NA #> $ latitude : logi NA #> $ ensembleInFile : int [1:444] 1 2427 4675 6923 9171 11419 13667 15915 18163 20411 ... #> $ velocityResolution : num 0.001 #> $ velocityMaximum : num 32.8 #> $ numberOfSamples : int 443 #> $ oceBeamUnspreaded : logi FALSE #> $ depthMean : num 0.0176 #> $ transformationMatrix: num [1:4, 1:4] 1.1844 -0.0043 0.2743 0.8311 -1.1763 ... str(d@data,2) #> List of 31 #> $ v : num [1:443, 1:95, 1:4] -0.7 -0.666 -0.7 -0.7 -0.7 -0.728 -0.707 -0.713 -0.682 -0.713 ... #> $ q : raw [1:443, 1:95, 1:4] 23 20 20 21 ... #> $ a : raw [1:443, 1:95, 1:4] 6d 6d 6d 6d ... #> $ g : NULL #> $ vv : num [1:443, 1:95] -1.1 -1.1 -1.1 -1.09 -1.09 ... #> $ vq : raw [1:443, 1:95] 79 78 77 78 ... #> $ va : raw [1:443, 1:95] 94 94 94 94 ... #> $ vg : NULL #> $ vdistance : num [1:95] 1.88 2.88 3.88 4.88 5.88 ... #> $ distance : num [1:95] 1.92 2.92 3.92 4.92 5.92 ... #> $ time : POSIXct[1:443], format: "2018-04-21 00:00:00" "2018-04-21 00:00:00" ... #> $ pressure : num [1:443] 0.048 0.048 0.049 0.043 0.048 0.045 0.045 0.047 0.045 0.052 ... #> $ temperature : num [1:443] 12.5 12.5 12.5 12.5 12.5 ... #> $ salinity : int [1:443] 35 35 35 35 35 35 35 35 35 35 ... #> $ depth : num [1:443] 0 0 0 0 0 0 0 0 0 0.1 ... #> $ soundSpeed : int [1:443] 1499 1499 1499 1499 1499 1499 1499 1499 1499 1499 ... #> $ heading : num [1:443] 332 331 334 332 331 ... #> $ pitch : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ roll : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ headingStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pitchStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ rollStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureStd : int [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitCurrent : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitVoltage : num [1:443] 189 189 189 189 189 189 189 189 189 189 ... #> $ ambientTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressurePlus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureMinus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitudeTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitude : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ contaminationSensor: num [1:443] 0 0 0 0 0 0 0 0 0 0 ... # Details (dsp) str(dsp@metadata,2) #> List of 59 #> $ units :List of 14 #> ..$ v :List of 2 #> ..$ distance :List of 2 #> ..$ pressure :List of 2 #> ..$ salinity :List of 2 #> ..$ temperature:List of 2 #> ..$ soundSpeed :List of 2 #> ..$ heading :List of 2 #> ..$ pitch :List of 2 #> ..$ roll :List of 2 #> ..$ headingStd :List of 2 #> ..$ pitchStd :List of 2 #> ..$ rollStd :List of 2 #> ..$ attitude :List of 2 #> ..$ depth :List of 2 #> $ flags : list() #> $ oceCoordinate : chr "beam" #> $ orientation : chr [1:443] "downward" "downward" "downward" "downward" ... #> $ instrumentType : chr "adcp" #> $ instrumentSubtype : chr "sentinelV" #> $ firmwareVersionMajor: int 47 #> $ firmwareVersionMinor: int 19 #> $ firmwareVersion : chr "47.19" #> $ bytesPerEnsemble : int 2424 #> $ systemConfiguration : chr "01001010-01010111" #> $ frequency : num 250 #> $ beamAngle : num 25 #> $ beamPattern : chr "convex" #> $ beamConfig : chr "5 beam janus" #> $ numberOfDataTypes : int 14 #> $ dataOffset : int [1:14] 34 94 160 922 1304 1686 1726 1918 2015 2112 ... #> $ codes : raw [1:16, 1:2] 7f 00 80 00 ... #> $ numberOfBeams : int 4 #> $ numberOfCells : int 95 #> $ pingsPerEnsemble : int 1 #> $ cellSize : num 1 #> $ transducerDepth : int 0 #> $ profilingMode : int 1 #> $ lowCorrThresh : int 0 #> $ numberOfCodeReps : int 3 #> $ percentGdMinimum : int 0 #> $ errorVelocityMaximum: int 0 #> $ coordTransform : chr "00000000" #> $ originalCoordinate : chr "beam" #> $ tiltUsed : logi FALSE #> $ threeBeamUsed : logi FALSE #> $ binMappingUsed : logi FALSE #> $ headingAlignment : num 0 #> $ headingBias : num 0 #> $ sensorSource : chr "01111101" #> $ sensorsAvailable : chr "01111101" #> $ bin1Distance : num 1.92 #> $ xmitPulseLength : num 1.37 #> $ wpRefLayerAverage : int 0 #> $ falseTargetThresh : int -1 #> $ transmitLagDistance : int 47 #> $ cpuBoardSerialNumber: int [1:8] 48 50 97 48 52 51 53 49 #> $ systemBandwidth : int 0 #> $ serialNumber : int 125 #> $ haveActualData : logi TRUE #> $ vBeamHeader :List of 22 #> ..$ numberOfVCells : int 95 #> ..$ verticalPings : int 1 #> ..$ depthCellSize : num 1 #> ..$ firstCellRange : num 1.88 #> ..$ verticalMode : int 1 #> ..$ verticalTransmit : num 1.3 #> ..$ verticalLagLength : num 0.47 #> ..$ transmitCodeElements : int 102 #> ..$ verticalRssiThreshold : int 0 #> ..$ verticalShallowBin : int 0 #> ..$ verticalStartBin : int 0 #> ..$ verticalShallowRssiBin: int 0 #> ..$ maxCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ pingOffsetTime : int 250 #> ..$ surfSpare1 : int 0 #> ..$ depthScreen : int 0 #> ..$ percentGoodThreshold : int 0 #> ..$ verticalDOproofing : int 0 #> ..$ vSeriesConfig :List of 6 #> ..$ vSeriesPingSetup :List of 11 #> $ ensembleNumber : num [1:443] 2 3 4 5 6 7 8 9 10 11 ... #> $ manufacturer : chr "teledyne rdi" #> $ filename : chr "/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0" #> $ longitude : logi NA #> $ latitude : logi NA #> $ ensembleInFile : int [1:444] 1 2427 4675 6923 9171 11419 13667 15915 18163 20411 ... #> $ velocityResolution : num 0.001 #> $ velocityMaximum : num 32.8 #> $ numberOfSamples : int 188 #> $ oceBeamUnspreaded : logi FALSE #> $ depthMean : num 0.0176 #> $ transformationMatrix: num [1:4, 1:4] 1.1844 -0.0043 0.2743 0.8311 -1.1763 ... str(dsp@data,2) #> List of 31 #> $ v : num [1:188, 1:95, 1:4] -0.7 -0.728 -0.707 -0.682 -0.694 -0.638 -0.724 -0.704 -0.663 -0.659 ... #> $ q : raw [1:188, 1:95, 1:4] 21 1e 21 1f ... #> $ a : raw [1:188, 1:95, 1:4] 6d 6d 6d 6d ... #> $ g : NULL #> $ vv : num [1:188, 1:95] -1.09 -1.08 -1.09 -1.08 -1.07 ... #> $ vq : raw [1:188, 1:95] 78 79 78 78 ... #> $ va : raw [1:188, 1:95] 94 94 94 94 ... #> $ vg : NULL #> $ vdistance : num [1:95] 1.88 2.88 3.88 4.88 5.88 ... #> $ distance : num [1:95] 1.92 2.92 3.92 4.92 5.92 ... #> $ time : POSIXct[1:188], format: "2018-04-21 00:00:01" "2018-04-21 00:00:02" ... #> $ pressure : num [1:188] 0.043 0.045 0.045 0.045 0.043 0.044 0.046 0.042 0.043 0.046 ... #> $ temperature : num [1:188] 12.5 12.5 12.5 12.5 12.5 ... #> $ salinity : int [1:188] 35 35 35 35 35 35 35 35 35 35 ... #> $ depth : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ soundSpeed : int [1:188] 1499 1499 1499 1499 1499 1499 1499 1499 1499 1499 ... #> $ heading : num [1:188] 332 333 330 333 330 ... #> $ pitch : num [1:188] 89.3 89.3 89.3 89.3 89.3 ... #> $ roll : num [1:188] 89.3 89.3 89.3 89.3 89.3 ... #> $ headingStd : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ pitchStd : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ rollStd : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureStd : int [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitCurrent : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitVoltage : num [1:188] 189 189 189 189 189 189 189 189 189 189 ... #> $ ambientTemp : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressurePlus : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureMinus : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitudeTemp : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitude : num [1:188] 0 0 0 0 0 0 0 0 0 0 ... #> $ contaminationSensor: num [1:188] 0 0 0 0 0 0 0 0 0 0 ... ``` Created on 2021-06-16 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)
dankelley commented 3 years ago

I think things work now for subset-by-distance (see Details)

``` r setwd("~/git/oce-issues/18xx/1837") # PURPOSE: test subset-by-distance or issue "subset doesn't work with sentinel # V vertical beam data" (https://github.com/dankelley/oce/issues/1837), using a # dataset provided Clark Richards. # # CHECK: In the summaries, is there uniform reduction of second index in # arrays, as appropriate to each variable type? library(oce) #> Loading required package: gsw #> Loading required package: testthat #> Loading required package: sf #> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1 d <- read.oce("sentinel_adcp_example.pd0") #> Got to end of data while trying to read an RDI file (cindex=998294; last7f7f=998290) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): skipping the first ensemble (a temporary solution that eases reading of SentinelV files) #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): A list of unhandled segment codes follows. Several Teledyne RDI manuals #> describe such codes; see e.g. Table 33 of Teledyne RD Instruments, 2014. #> Ocean Surveyor/Ocean Observer Technical Manual. #> P/N 95A-6012-00 April 2014 (OS_TM_Apr14.pdf) #> Code 0x01 0x0f occurred 444 times #> Code 0x00 0x70 occurred 444 times #> Code 0x01 0x70 occurred 444 times #> Code 0x02 0x70 occurred 444 times #> Code 0x00 0x32 occurred 444 times #> Code 0x04 0x70 occurred 444 times #> Warning in read.adp.rdi(file, processingLog = processingLog, ...): length(time)=444 exceeds length(profileStart)=443 so trimming time #> length(time)=444 exceeds length(profileStart)=443 so trimming time summary(d) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of beams: 4 #> * Number of profiles: 443 #> * Number of cells: 95 #> * Number of beams: 4 #> * Cell size: 1m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.500 0.500 0.500 5.726 0.500 165.500 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 95, centered at 1.920 m to 95.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 slanted, plus 1 central #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.010573 2.923 443x95x4 95 - #> q 0 19.884 170 443x95x4 0 - #> a 33 39.261 123 443x95x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.019293 3.191 443x95 35 - #> vq 1 20.747 126 443x95 0 - #> va 35 39.853 149 443x95 0 - #> vg NA NA NA 0 0 - #> vdistance 1.88 48.88 95.88 95 0 - #> distance [m] 1.92 48.92 95.92 95 0 - #> pressure [dbar] 0.034 0.047115 0.055 443 0 - #> temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - #> salinity [PSS-78] 35 35 35 443 0 - #> depth [m] 0 0.017607 0.1 443 0 - #> soundSpeed [m/s] 1499 1499.5 1500 443 0 - #> heading [°] 327.35 331.39 335.61 443 0 - #> pitch [°] 89.277 89.305 89.337 443 0 - #> roll [°] 89.28 89.308 89.34 443 0 - #> headingStd [°] 0 0 0 443 0 - #> pitchStd [°] 0 0 0 443 0 - #> rollStd [°] 0 0 0 443 0 - #> pressureStd 0 0 0 443 0 - #> xmitCurrent 0 0 0 443 0 - #> xmitVoltage 189 189.8 190 443 0 - #> ambientTemp 0 0 0 443 0 - #> pressurePlus 0 0 0 443 0 - #> pressureMinus 0 0 0 443 0 - #> attitudeTemp 0 0 0 443 0 - #> attitude [°] 0 0 0 443 0 - #> contaminationSensor 0 0 0 443 0 - #> #> * Processing Log #> #> - 2021-06-16 17:36:57 UTC: `read.oce("sentinel_adcp_example.pd0")` keepSlant <- sum(d[["distance"]] < median(d[["distance"]])) keepVertical <- sum(d[["vdistance"]] < median(d[["distance"]])) dsd <- subset(d, distance < median(d[["distance"]])) summary(dsd) #> ADP Summary #> ----------- #> #> * Instrument: adcp #> * Serial number: 125 #> * Firmware version: 47.19 #> * Source filename: ``/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0`` #> * Location: unknown latitude, unknown longitude #> * Number of beams: 4 #> * Number of profiles: 443 #> * Number of cells: 47 #> * Number of beams: 4 #> * Cell size: 1m #> * Summary of times between profiles: #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.500 0.500 0.500 5.726 0.500 165.500 #> * Frequency: 250 kHz #> * Ensemble Numbers: object@metadata$ensembleNumber [1:443]: 2, 3, ..., 443, 444 #> * Cells: 47, centered at 1.920 m to 47.920 m, spaced by 1.000 m #> * Coordinate system: beam [originally], beam [presently] #> * Beams:: #> Number: 4 slanted, plus 1 central #> Slantwise Angle: 25 #> Orientation: downward:443 #> Unspreaded: FALSE #> * Transformation matrix:: #> 1.1844 -1.1763 0.0047 -0.0099 #> -0.0043 0.0103 -1.1878 1.1783 #> 0.2743 0.2774 0.2742 0.2778 #> 0.8311 0.8405 -0.8299 -0.8409 #> * Time ranges from 2018-04-21 to 2018-04-21 00:42:11 with 443 samples and mean increment 5.726244 s #> * Data Overview #> #> Min. Mean Max. Dim. NAs OriginalName #> v [m/s] -3.145 -0.025047 2.923 443x47x4 48 - #> q 1 21.355 170 443x47x4 0 - #> a 33 40.494 123 443x47x4 0 - #> g NA NA NA 0 0 - #> vv -3.09 -0.023315 3.191 443x48 16 - #> vq 1 22.477 126 443x48 0 - #> va 35 41.471 149 443x48 0 - #> vdistance 1.88 25.38 48.88 48 0 - #> distance [m] 1.92 24.92 47.92 47 0 - #> pressure [dbar] 0.034 0.047115 0.055 443 0 - #> temperature [°C, ITS-90] 12.49 12.734 12.98 443 0 - #> salinity [PSS-78] 35 35 35 443 0 - #> depth [m] 0 0.017607 0.1 443 0 - #> soundSpeed [m/s] 1499 1499.5 1500 443 0 - #> heading [°] 327.35 331.39 335.61 443 0 - #> pitch [°] 89.277 89.305 89.337 443 0 - #> roll [°] 89.28 89.308 89.34 443 0 - #> headingStd [°] 0 0 0 443 0 - #> pitchStd [°] 0 0 0 443 0 - #> rollStd [°] 0 0 0 443 0 - #> pressureStd 0 0 0 443 0 - #> xmitCurrent 0 0 0 443 0 - #> xmitVoltage 189 189.8 190 443 0 - #> ambientTemp 0 0 0 443 0 - #> pressurePlus 0 0 0 443 0 - #> pressureMinus 0 0 0 443 0 - #> attitudeTemp 0 0 0 443 0 - #> attitude [°] 0 0 0 443 0 - #> contaminationSensor 0 0 0 443 0 - #> #> * Processing Log #> #> - 2021-06-16 17:36:57 UTC: `read.oce("sentinel_adcp_example.pd0")` #> - 2021-06-16 17:36:57 UTC: `subset.adp(x, subset=distance < median(d[["distance"]]))` # Tests (will incorporate into local test suite) library(testthat) expect_equal(keepSlant, length(dsd[["distance"]])) expect_equal(keepVertical , length(dsd[["vdistance"]])) for (vert in c("va", "vq", "vv")) { # no "vg" in this dataset expect_equal(keepVertical, dim(dsd[[vert]])[2]) } for (slant in c("v", "a")) { # "g" is NULL expect_equal(keepSlant, dim(dsd[[slant]])[2]) } # Details (d) str(d@metadata,2) #> List of 59 #> $ units :List of 14 #> ..$ v :List of 2 #> ..$ distance :List of 2 #> ..$ pressure :List of 2 #> ..$ salinity :List of 2 #> ..$ temperature:List of 2 #> ..$ soundSpeed :List of 2 #> ..$ heading :List of 2 #> ..$ pitch :List of 2 #> ..$ roll :List of 2 #> ..$ headingStd :List of 2 #> ..$ pitchStd :List of 2 #> ..$ rollStd :List of 2 #> ..$ attitude :List of 2 #> ..$ depth :List of 2 #> $ flags : list() #> $ oceCoordinate : chr "beam" #> $ orientation : chr [1:443] "downward" "downward" "downward" "downward" ... #> $ instrumentType : chr "adcp" #> $ instrumentSubtype : chr "sentinelV" #> $ firmwareVersionMajor: int 47 #> $ firmwareVersionMinor: int 19 #> $ firmwareVersion : chr "47.19" #> $ bytesPerEnsemble : int 2424 #> $ systemConfiguration : chr "01001010-01010111" #> $ frequency : num 250 #> $ beamAngle : num 25 #> $ beamPattern : chr "convex" #> $ beamConfig : chr "5 beam janus" #> $ numberOfDataTypes : int 14 #> $ dataOffset : int [1:14] 34 94 160 922 1304 1686 1726 1918 2015 2112 ... #> $ codes : raw [1:16, 1:2] 7f 00 80 00 ... #> $ numberOfBeams : int 4 #> $ numberOfCells : int 95 #> $ pingsPerEnsemble : int 1 #> $ cellSize : num 1 #> $ transducerDepth : int 0 #> $ profilingMode : int 1 #> $ lowCorrThresh : int 0 #> $ numberOfCodeReps : int 3 #> $ percentGdMinimum : int 0 #> $ errorVelocityMaximum: int 0 #> $ coordTransform : chr "00000000" #> $ originalCoordinate : chr "beam" #> $ tiltUsed : logi FALSE #> $ threeBeamUsed : logi FALSE #> $ binMappingUsed : logi FALSE #> $ headingAlignment : num 0 #> $ headingBias : num 0 #> $ sensorSource : chr "01111101" #> $ sensorsAvailable : chr "01111101" #> $ bin1Distance : num 1.92 #> $ xmitPulseLength : num 1.37 #> $ wpRefLayerAverage : int 0 #> $ falseTargetThresh : int -1 #> $ transmitLagDistance : int 47 #> $ cpuBoardSerialNumber: int [1:8] 48 50 97 48 52 51 53 49 #> $ systemBandwidth : int 0 #> $ serialNumber : int 125 #> $ haveActualData : logi TRUE #> $ vBeamHeader :List of 22 #> ..$ numberOfVCells : int 95 #> ..$ verticalPings : int 1 #> ..$ depthCellSize : num 1 #> ..$ firstCellRange : num 1.88 #> ..$ verticalMode : int 1 #> ..$ verticalTransmit : num 1.3 #> ..$ verticalLagLength : num 0.47 #> ..$ transmitCodeElements : int 102 #> ..$ verticalRssiThreshold : int 0 #> ..$ verticalShallowBin : int 0 #> ..$ verticalStartBin : int 0 #> ..$ verticalShallowRssiBin: int 0 #> ..$ maxCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ pingOffsetTime : int 250 #> ..$ surfSpare1 : int 0 #> ..$ depthScreen : int 0 #> ..$ percentGoodThreshold : int 0 #> ..$ verticalDOproofing : int 0 #> ..$ vSeriesConfig :List of 6 #> ..$ vSeriesPingSetup :List of 11 #> $ ensembleNumber : num [1:443] 2 3 4 5 6 7 8 9 10 11 ... #> $ manufacturer : chr "teledyne rdi" #> $ filename : chr "/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0" #> $ longitude : logi NA #> $ latitude : logi NA #> $ ensembleInFile : int [1:444] 1 2427 4675 6923 9171 11419 13667 15915 18163 20411 ... #> $ velocityResolution : num 0.001 #> $ velocityMaximum : num 32.8 #> $ numberOfSamples : int 443 #> $ oceBeamUnspreaded : logi FALSE #> $ depthMean : num 0.0176 #> $ transformationMatrix: num [1:4, 1:4] 1.1844 -0.0043 0.2743 0.8311 -1.1763 ... str(d@data,2) #> List of 31 #> $ v : num [1:443, 1:95, 1:4] -0.7 -0.666 -0.7 -0.7 -0.7 -0.728 -0.707 -0.713 -0.682 -0.713 ... #> $ q : raw [1:443, 1:95, 1:4] 23 20 20 21 ... #> $ a : raw [1:443, 1:95, 1:4] 6d 6d 6d 6d ... #> $ g : NULL #> $ vv : num [1:443, 1:95] -1.1 -1.1 -1.1 -1.09 -1.09 ... #> $ vq : raw [1:443, 1:95] 79 78 77 78 ... #> $ va : raw [1:443, 1:95] 94 94 94 94 ... #> $ vg : NULL #> $ vdistance : num [1:95] 1.88 2.88 3.88 4.88 5.88 ... #> $ distance : num [1:95] 1.92 2.92 3.92 4.92 5.92 ... #> $ time : POSIXct[1:443], format: "2018-04-21 00:00:00" "2018-04-21 00:00:00" ... #> $ pressure : num [1:443] 0.048 0.048 0.049 0.043 0.048 0.045 0.045 0.047 0.045 0.052 ... #> $ temperature : num [1:443] 12.5 12.5 12.5 12.5 12.5 ... #> $ salinity : int [1:443] 35 35 35 35 35 35 35 35 35 35 ... #> $ depth : num [1:443] 0 0 0 0 0 0 0 0 0 0.1 ... #> $ soundSpeed : int [1:443] 1499 1499 1499 1499 1499 1499 1499 1499 1499 1499 ... #> $ heading : num [1:443] 332 331 334 332 331 ... #> $ pitch : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ roll : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ headingStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pitchStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ rollStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureStd : int [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitCurrent : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitVoltage : num [1:443] 189 189 189 189 189 189 189 189 189 189 ... #> $ ambientTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressurePlus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureMinus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitudeTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitude : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ contaminationSensor: num [1:443] 0 0 0 0 0 0 0 0 0 0 ... # Details (dsp) str(dsd@metadata,2) #> List of 59 #> $ units :List of 14 #> ..$ v :List of 2 #> ..$ distance :List of 2 #> ..$ pressure :List of 2 #> ..$ salinity :List of 2 #> ..$ temperature:List of 2 #> ..$ soundSpeed :List of 2 #> ..$ heading :List of 2 #> ..$ pitch :List of 2 #> ..$ roll :List of 2 #> ..$ headingStd :List of 2 #> ..$ pitchStd :List of 2 #> ..$ rollStd :List of 2 #> ..$ attitude :List of 2 #> ..$ depth :List of 2 #> $ flags : list() #> $ oceCoordinate : chr "beam" #> $ orientation : chr [1:443] "downward" "downward" "downward" "downward" ... #> $ instrumentType : chr "adcp" #> $ instrumentSubtype : chr "sentinelV" #> $ firmwareVersionMajor: int 47 #> $ firmwareVersionMinor: int 19 #> $ firmwareVersion : chr "47.19" #> $ bytesPerEnsemble : int 2424 #> $ systemConfiguration : chr "01001010-01010111" #> $ frequency : num 250 #> $ beamAngle : num 25 #> $ beamPattern : chr "convex" #> $ beamConfig : chr "5 beam janus" #> $ numberOfDataTypes : int 14 #> $ dataOffset : int [1:14] 34 94 160 922 1304 1686 1726 1918 2015 2112 ... #> $ codes : raw [1:16, 1:2] 7f 00 80 00 ... #> $ numberOfBeams : int 4 #> $ numberOfCells : int 47 #> $ pingsPerEnsemble : int 1 #> $ cellSize : num 1 #> $ transducerDepth : int 0 #> $ profilingMode : int 1 #> $ lowCorrThresh : int 0 #> $ numberOfCodeReps : int 3 #> $ percentGdMinimum : int 0 #> $ errorVelocityMaximum: int 0 #> $ coordTransform : chr "00000000" #> $ originalCoordinate : chr "beam" #> $ tiltUsed : logi FALSE #> $ threeBeamUsed : logi FALSE #> $ binMappingUsed : logi FALSE #> $ headingAlignment : num 0 #> $ headingBias : num 0 #> $ sensorSource : chr "01111101" #> $ sensorsAvailable : chr "01111101" #> $ bin1Distance : num 1.92 #> $ xmitPulseLength : num 1.37 #> $ wpRefLayerAverage : int 0 #> $ falseTargetThresh : int -1 #> $ transmitLagDistance : int 47 #> $ cpuBoardSerialNumber: int [1:8] 48 50 97 48 52 51 53 49 #> $ systemBandwidth : int 0 #> $ serialNumber : int 125 #> $ haveActualData : logi TRUE #> $ vBeamHeader :List of 22 #> ..$ numberOfVCells : int 95 #> ..$ verticalPings : int 1 #> ..$ depthCellSize : num 1 #> ..$ firstCellRange : num 1.88 #> ..$ verticalMode : int 1 #> ..$ verticalTransmit : num 1.3 #> ..$ verticalLagLength : num 0.47 #> ..$ transmitCodeElements : int 102 #> ..$ verticalRssiThreshold : int 0 #> ..$ verticalShallowBin : int 0 #> ..$ verticalStartBin : int 0 #> ..$ verticalShallowRssiBin: int 0 #> ..$ maxCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ minCoreThreshold : int 0 #> ..$ pingOffsetTime : int 250 #> ..$ surfSpare1 : int 0 #> ..$ depthScreen : int 0 #> ..$ percentGoodThreshold : int 0 #> ..$ verticalDOproofing : int 0 #> ..$ vSeriesConfig :List of 6 #> ..$ vSeriesPingSetup :List of 11 #> $ ensembleNumber : num [1:443] 2 3 4 5 6 7 8 9 10 11 ... #> $ manufacturer : chr "teledyne rdi" #> $ filename : chr "/Users/kelley/git/oce-issues/18xx/1837/sentinel_adcp_example.pd0" #> $ longitude : logi NA #> $ latitude : logi NA #> $ ensembleInFile : int [1:444] 1 2427 4675 6923 9171 11419 13667 15915 18163 20411 ... #> $ velocityResolution : num 0.001 #> $ velocityMaximum : num 32.8 #> $ numberOfSamples : int 443 #> $ oceBeamUnspreaded : logi FALSE #> $ depthMean : num 0.0176 #> $ transformationMatrix: num [1:4, 1:4] 1.1844 -0.0043 0.2743 0.8311 -1.1763 ... str(dsd@data,2) #> List of 30 #> $ v : num [1:443, 1:47, 1:4] -0.7 -0.666 -0.7 -0.7 -0.7 -0.728 -0.707 -0.713 -0.682 -0.713 ... #> $ q : raw [1:443, 1:47, 1:4] 23 20 20 21 ... #> $ a : raw [1:443, 1:47, 1:4] 6d 6d 6d 6d ... #> $ g : NULL #> $ vv : num [1:443, 1:48] -1.1 -1.1 -1.1 -1.09 -1.09 ... #> $ vq : raw [1:443, 1:48] 79 78 77 78 ... #> $ va : raw [1:443, 1:48] 94 94 94 94 ... #> $ vdistance : num [1:48] 1.88 2.88 3.88 4.88 5.88 ... #> $ distance : num [1:47] 1.92 2.92 3.92 4.92 5.92 ... #> $ time : POSIXct[1:443], format: "2018-04-21 00:00:00" "2018-04-21 00:00:00" ... #> $ pressure : num [1:443] 0.048 0.048 0.049 0.043 0.048 0.045 0.045 0.047 0.045 0.052 ... #> $ temperature : num [1:443] 12.5 12.5 12.5 12.5 12.5 ... #> $ salinity : int [1:443] 35 35 35 35 35 35 35 35 35 35 ... #> $ depth : num [1:443] 0 0 0 0 0 0 0 0 0 0.1 ... #> $ soundSpeed : int [1:443] 1499 1499 1499 1499 1499 1499 1499 1499 1499 1499 ... #> $ heading : num [1:443] 332 331 334 332 331 ... #> $ pitch : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ roll : num [1:443] 89.3 89.3 89.3 89.3 89.3 ... #> $ headingStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pitchStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ rollStd : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureStd : int [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitCurrent : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ xmitVoltage : num [1:443] 189 189 189 189 189 189 189 189 189 189 ... #> $ ambientTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressurePlus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ pressureMinus : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitudeTemp : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ attitude : num [1:443] 0 0 0 0 0 0 0 0 0 0 ... #> $ contaminationSensor: num [1:443] 0 0 0 0 0 0 0 0 0 0 ... ``` Created on 2021-06-16 by the [reprex package](https://reprex.tidyverse.org) (v2.0.0)
dankelley commented 3 years ago

The test suite now incorporates the tests that I showed in the previous two issue comments. These tests will not be done in CRAN, but this should not matter, because they will be activated whenever a developer builds and tests oce from the repo source.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because no comments have been made in two weeks. Unless comments are made in the next two weeks (or the 'pinned' label is applied), this issue will be closed automatically in a week. Please note that this is not a reflection on the quality or importance of the issue, but merely a reminder of the march of time. A simple comment is all it takes to keep this issue from being closed automatically, two weeks from now.

dankelley commented 3 years ago

Term is approaching and I'd like to clean things up. From a quick look at the comments, it seems that this issue has been addressed.

Perhaps, @jessecusack or @richardsc would consider either closing the issue or adding comments indicating what remains to be done?

jessecusack commented 3 years ago

Subsetting now works for 5th beam data and all the other variables in a sentinal V dataset, so I'm happy to close the issue! Thanks @dankelley for all the time and effort you put in!