Error using calibrateDepth function

[RuthDunn]

I have recently tried to use the calibrateDepth function on some time-depth recorder data collected on auks but have had this error returned for some of my files:

Error message: Record is truncated at the beginning and at the end 1 phase detected 189 dives detected Error in if (all(Ad1.maybe >= 0)) { : Missing value where TRUE/FALSE needed

As far as I can tell, there are no errors in the files that I am trying to process. If you are able to advise me on what might be going wrong here, that would be great!

[spluque]

Hi Ruth,

On Tue, 19 Mar 2019 17:46:56 +0000 (UTC), "Ruth E. Dunn" notifications@github.com wrote:

I have recently tried to use the calibrateDepth function on some time-depth recorder data collected on auks but have had this error returned for some of my files:

Error message: Record is truncated at the beginning and at the end 1 phase detected 189 dives detected Error in if (all(Ad1.maybe >= 0)) { : Missing value where TRUE/FALSE needed

As far as I can tell, there are no errors in the files that I am trying to process. If you are able to advise me on what might be going wrong here, that would be great!

What files are you using? Where's your script? What version R, diveMove, etc.?

-- Seb

[RuthDunn]

Hi Sebastian,

Thanks for your response. I am using R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch" and diveMove 1.4.5..

Here is my script:

timedepth <-
  read.delim2("test_guilli.txt", header = TRUE, sep = ",")

# Make sure times and depths are in correct format:
timedepth$Time <-
  as.POSIXct(strptime(timedepth$Time, format = "%Y-%m-%d %H:%M:%S"),
             tz = "GMT")
timedepth$Depth <- as.numeric(as.character(timedepth$Depth))

# Change "dtime" to reflect the 32 s sampling rate
# Include temperature data too
divesTDR<-createTDR(timedepth$Time,
                    depth = timedepth$Depth,
                    dtime = 32,
                    concurrentData=data.frame(timedepth$Temp),
                    file="bird197_day5", speed=F)

# ZOC:
dcalib<-calibrateDepth(divesTDR, zoc.method="filter",
                       k=c(3, 5760),
                       probs=c(0.5, 0.02),
                       dive.thr=1)

This is the message that I receive here:

Record is truncated at the beginning and at the end
1 phases detected
159 dives detected
Error in if (all(Ad1.maybe >= 0)) { : 
  missing value where TRUE/FALSE needed

Attached is an example data file too. test_guilli.txt

[rhiaustin]

Hi Sebastian,

I am having exactly the same issue as Ruth. The code has been working for 80% of my dive files, but not the other 20%, and I have spent days trying to figure out what the issue is with no luck. There seems to be no obvious difference between the files that work and those that do not. I can send you a sample file as well if that helps?

It would be great to get the bottom of this so that I can continue to use the package in my work. I have used diveMove multiple times on multiple datasets in the past and have not come across this issue until now.

Thanks for your time!

[spluque]

I cannot reproduce Ruth's example as I don't know where the as.data.table comes from, and her file shows an irregular time series, which diveMove isn't designed to handle.

Can you show a reproducible example and sample data file?

[rhiaustin]

Hi Sebastian,

Many thanks for your reply. I attach a sample file that is currently giving me the error message, as well as the code that I am using. As mentioned previously, I have used diveMove many times in the past and never had this issue before, and it is only happening with 5 of my TDR files (the rest, which have been downloaded and processed in exactly the same way are working fine).

library(diveMove)

#Load data
dir = "/Users/rm/Desktop" #change directory as needed
setwd(dir)
tg = "test_data_RA.txt"
df =  read.delim(tg)
str(df)
head(df)
names(df)

#create a string of the file path
filepath = list.files(path=dir, pattern=tg, full.names=TRUE, recursive=FALSE) 

#convert datetime to POSIXct object
tdr.time = as.POSIXct(strptime(df$Datetime, format="%d/%m/%Y %H:%M:%S"), tz="GMT") 
head(tdr.time)

#depth is a column in the txt file that is the same as the pressure column but with a conversion for depth
tdr.depth = df$Depth

#create TDR object
tdrXno = createTDR(time=tdr.time, depth=tdr.depth, speed=FALSE, 
                   dtime=1, file=filepath)

#specify depth range
db = c(-5,5) 
#specify dive threshold
dt = 0.25

#apply zero offset and identify dives
dcalib = calibrateDepth(tdrXno, dive.thr=dt, zoc.method="filter",
                        k=c(3, 600), probs=c(0.5, 0.05), na.rm=TRUE,
                        knot.factor=50, depth.bounds=db)

# The error message that this returns with my problem files is as follows:
# >Record is truncated at the beginning and at the end
# >1 phases detected
# >2865 dives detected
# >Error in if (all(Ad1.maybe >= 0)) { : 
# >  missing value where TRUE/FALSE needed

Would love to get to the bottom of this so that I can use these data with your package. It has been so useful in the past.

Many thanks for your help.

Regards, Rhiannon

test_data_RA.txt

[spluque]

The file you sent doesn't match the script, so can't get it to read properly. Can you update your script and make sure that it does what you want to read the sample data file?

[rhiaustin]

Hi Sebastian,

Apologies. Code and txt file amended now - hopefully this will work (data were originally stored in a csv file but giithub won't accept them).

Many thanks, Rhiannon

[RuthDunn]

Hi Sebastian,

I have updated my example code and data too based on your comments. Looking forward to hearing from you.

Thanks,

Ruth

[spluque]

OK, in both cases, you can save yourselves a lot of grief at the beginning by using readTDR. For Ruth's data (renamed to csv):

bird197 <- readTDR("test_guilli.csv", dateCol=1, timeCol=NULL,
                   dtformat="%Y-%m-%d %H:%M:%S", concurrentCols=3,
                   subsamp=32)

and this data set doesn't show any issues with calibrateDepth. Make sure you're using latest R and diveMove.

I can reproduce the other case (Rhiannon's). These data show several problems discussed in Luque & Fried (2011) combined with very shallow dives, so the 0.25 dive threshold is way too hopeful, given the performance of your zero offset correction:

d.filter <- diveMove:::.depthFilter(depth=getDepth(tdrXno),
                                    k=c(3, 600), probs=c(0.5, 0.05),
                                    depth.bounds=db, na.rm=TRUE)
plotZOC(tdrXno, d.filter)

Notice all the gaps between the two filters. There's no problems if you increase the dive threshold to >= 0.5. Otherwise, you need to do a better job with the filters for zero offset correction.

At any rate, the error message could be more helpful, so I'll work on that for the next version.

[rhiaustin]

Hi Sebastian. Thanks for this. Yes, a higher dive threshold does work. Strange, I had tested a number of thresholds initially, but obviously not on these files with the same combination of parameters I gave you in the example. Expanded error messages would definitely be great. Many thanks for your time.

[androiddimatteo]

Hi Sebastian,

I am also have an issue with calibrateDepth. We have 1sec interval pressure sensor data for 8 loggerhead turtles. For half the tags the below code has no issues. For the other half I get the following error:

Record is truncated at the beginning and at the end 1 phases detected Error in diveid[dive != 0] <- rep(seq(along = table(diveid)[-1]), table(diveid)[-1]) : replacement has length zero

I am using R version 4.0.2 and diveMove 1.5.2. The code I am using is:

path <- "C://Users//saxon//Documents//Navy//analysis//NUWC_Cape_Canaveral//OpenTag_20200722//diveMove//csvs//g_dep.csv" g_tdr <- readTDR(path, dateCol = 2, timeCol = NULL, depthCol = 4, dtformat = "%Y-%m-%dT0%H:%M:%OS", subsamp = 1) g_calib <- calibrateDepth(g_tdr, dive.thr = 2, zoc.method = "offset", offset = 0, dive.model = "smooth.spline")

You can alter the path as needed. I attached the files as .txt but they are actually .csv. The file extension just needs to be renamed. The 'a_dep' file is an example of a tag that is working, the 'g_dep' file is one that is not working.

For the tags it is working for it this package is just what I need so I hope there is an easy fix. I tried to find a consistent reason for the error but could not. I thought it may have been an issue with uneven time intervals (there are some reporting intervals of more than 1sec) but the code worked for some tags that had these gaps. Even when I filled the gaps, repeating the previously sampled value, I received the same error.

Many thanks for the package and any advice you can provide.

Andrew a_dep.txt g_dep.txt

[spluque]

@androiddimatteo Have you plotted the data for g_dep.txt? The offset is certainly not zero as you're assuming in your code.

[androiddimatteo]

Hi Sebastian,

Thanks for the fast response. Looks like this was my inexperience dealing with raw dive data (used to getting summarized outputs from Argos tags). I am now using the filter method for offset following recommendations in your 2011 paper and this is producing much better results.

Apologies for the naïve question!

Andrew

[jennyhoward9]

Hi Sebastian,

I'm having similar issues to the ones posted here, getting the error "Record is truncated at the beginning and at the end" after running calibrateDepth. I've tried it with both dive.model="smooth.spline" and dive.model="unimodal". I still am able to use diveStats to get summary information about the dives so is this an error/warning I need to be worried about? I am using the latest R (4.1.0) and the latest diveMove package (1.6.0). The code I am using is below and I attached an example txt file that is throwing the error (but they all seem to be). I would appreciate any advice you can offer!

Thanks!

Jenny

library(diveMove)

Set my WD

setwd("C:/Users/howajl14/Dropbox/Howajl14/DIVEY/data/")

tdrXcsv <- read.csv("FG1 20175 31 oct-7 nov 2010 age 7_01.txt", header = TRUE)

sfp <- "FG1 20175 31 oct-7 nov 2010 age 7_01.csv"

srcfn <- basename(sfp)

tdrX <- createTDR(time=time.posixct, depth=tdrXcsv$Depth, file=srcfn)

specify depth range

db = c(-0.75,0.75)

specify dive threshold

dt = 0.5

Specify k

3 seems like a good first filter

k = k=c(3, 60)

specify probs

0.01 is too small for second filter

0.05 seems to do best for second quantile!

P = c(0.5, 0.05)

Use this to visualize the smoothing

d.filter <- diveMove:::.depthFilter(depth = getDepth(tdrX), k = k, probs = P, depth.bounds = db, na.rm = TRUE) plotZOC(tdrX, d.filter)

This will do the calibration so you can create a summary of your dives

You can set your dive.model to "smooth.spline" or "unimodal" to identify dive phases, but for dives with less than five observations, smoothing spline regression is used regardless

dcalib <- calibrateDepth(tdrX, dive.thr = 0.5, zoc.method = "filter", k = k, probs = P, dive.model="smooth.spline", depth.bounds = db, na.rm = TRUE)

tdrXSumm1 <- diveStats(dcalib)

FG1 20175 31 oct-7 nov 2010 age 7_01.txt

[spluque]

@jennyhoward9 I'm not sure which error you're referring to. The message about the record being truncated at the beginning/end is just for your information to let you know that there's no "dry" time at those ends, as expected if you deployed and recovered the instrument when "dry". It is unrelated to the errors reported in this issue.

[jennyhoward9]

So sorry, my bad! I interpreted that message as an error and was concerned I was missing values of dives. Thank you for clarifying!

[JessHCarroll]

Hi, just jumping on here (sorry if it's an obvious answer)! I had the same problem as Rhiannon, and as advised I increased the dive threshold (from 3 to 4m). Thanks so much for the explanation, it was really helpful. This is now allowing me to process the dives previously producing the error (huray!) But!! Now some of the other dives are throwing up this error;

"Error in validObject(.Object) : 
  invalid class “diveModel” object: invalid object for slot "ascent.crit" in class "diveModel": got class "logical", should be or extend class "numeric""

Could you please explain what this means and if this can be fixed? I'm not really sure where to start with this! I've tried increments between 3-4m and it's not happy.

Thanks so much!!!

tdrdata <- createTDR(time = ttdr$date, depth = ttdr$depth,
                     dtime = 300,  # sampling interval (in seconds)
                     file = "ttdr.csv")  # path to the file

dcalib <-calibrateDepth(tdrdata,
                        wet.thr = 3610,  # (seconds) At-sea phases shorter than this threshold will be considered as trivial wet. Delete periods of wet activity that are too short to be compared with other wet periods.
                        dive.thr = 3,    # (meters) threshold depth below which an underwater phase should be considered a dive.
                        zoc.method ="filter",  # see Luque and Fried (2011)
                        k = c(12, 240),  # (60 and 1200 minutes) Vector of moving window width integers to be applied sequentially.
                        probs = c(0.5, 0.05),   # Vector of quantiles to extract at each step indicated by k (so it must be as long as k)
                        depth.bounds = c(-5, 15),  # minimum and maximum depth to bound the search for the surface.
                        na.rm = TRUE)

[spluque]

@jharv3y can you please attach a sample data file along with your code? If the error is not thrown by calibrateDepth, please open a new issue. Thanks

[JessHCarroll]

Hi @spluque thanks so much!!

Yes it's an error from calibrateDepth, like I say I'm sure it's something obvious :) Attached is code and sample data.

Code:

rm(list=ls())
require(dplyr)
require(lubridate)
require(diveMove)

data<- read.csv("sampledata.csv")

data$date <- paste(data$Day, data$Time)

data$date <- parse_date_time(data$date, "dmY HMS", tz="UTC")

write.csv(data, file = "ttdr.csv")

tdrdata <- createTDR(time = data$date, depth = data$Depth,
                     dtime = 300,  # sampling interval (in seconds)
                     file = "ttdr.csv")  # path to the file

dcalib <-calibrateDepth(tdrdata,
                        wet.thr = 3610,  # (seconds) At-sea phases shorter than this threshold will be considered as trivial wet.Delete periods of wet activity that are too short to be compared with other wet periods.
                        dive.thr = 4,    # (meters) threshold depth below which an underwater phase should be considered a dive.
                        zoc.method ="filter",  # see Luque and Fried (2011)
                        k = c(12, 240),  # (60 and 1200 minutes) Vector of moving window width integers to be applied sequentially.
                        probs = c(0.5, 0.05),   # Vector of quantiles to extract at each step indicated by k (so it must be as long as k)
                        depth.bounds = c(-5, 15),  # minimum and maximum depth to bound the search for the surface.
                        na.rm = TRUE)

Thanks a million, really appreciate it!!

Jess sampledata.csv

[spluque]

@jharv3y I can't reproduce your error with your sample data. You do have a one year gap in your timestamps so they are not regular, but that is a different problem.

You really don't need other packages just to read the data:

ifile <- "sampledata.csv"
tdrX <- readTDR(ifile, dateCol=6, timeCol=7, depthCol=11, concurrentCols=NULL,
                dtformat="%d-%b-%y %H:%M:%S", subsamp=300)
plotTDR(tdrX)

and make sure to plot your data to identify problems like that gap.

[JessHCarroll]

@spluque thanks for taking a look! Thanks for highlighting that issue, in the full code I remove the 'false' readouts and do some other preprocessing.

Apologies, I did check with the sample data and it was throwing up the same error, but I must have messed something up. Could you see if the full data file is throwing up the error please? I am still getting the same issue. Could it be something with my R? series_data.csv

rm(list=ls())
require(diveMove)

# TTDR data, WCT 5 minute bins so:
tfreq <- 5 * 60  # time interval from TTDR data, in seconds

data<- read.csv("series_data.csv")
data$date <- paste(data$Day, data$Time)

data$date <- parse_date_time(data$date, "dmY HMS", tz="UTC")

date_deploy<-c("2016-06-22 UTC")

data <- filter(data, date >= date_deploy)

write.csv(data, file = "ttdr.csv")

tdrdata <- createTDR(time = data$date, depth = data$Depth,
                     dtime = 300,  # sampling interval (in seconds)
                     file = "ttdr.csv")  # path to the file
plotTDR(tdrdata)

dcalib <-calibrateDepth(tdrdata,
                        wet.thr = 3610,  # (seconds) At-sea phases shorter than this threshold will be considered as trivial wet.Delete periods of wet activity that are too short to be compared with other wet periods.
                        dive.thr = 4,    # (meters) threshold depth below which an underwater phase should be considered a dive.
                        zoc.method ="filter",  # see Luque and Fried (2011)
                        k = c(12, 240),  # (60 and 1200 minutes) Vector of moving window width integers to be applied sequentially.
                        probs = c(0.5, 0.05),   # Vector of quantiles to extract at each step indicated by k (so it must be as long as k)
                        depth.bounds = c(-5, 15),  # minimum and maximum depth to bound the search for the surface.
                        na.rm = TRUE)

_Record is truncated at the beginning and at the end
347 phases detected
5068 dives detected
Error in validObject(.Object) : 
  invalid class “diveModel” object: invalid object for slot "ascent.crit" in class "diveModel": got class "logical", should be or extend class "numeric"_

[spluque]

@jharv3y This error is thrown because the dive model fails due to incorrect ZOC and dive threshold parameters used for your data, which lead to invalid dives. I can't look at it in greater detail at the moment, but there doesn't seem to be justification for the filter ZOC method; a 1.5 m or so offset is enough. Your sampling interval also leaves a lot of noise down to about 10 m (assuming those are the units), so 4 m doesn't seem enough. Your wet and dry thresholds may also need adjusting to make sure they make sense.

[JessHCarroll]

@spluque that's incredibly helpful thank you so much!!!! Really appreciate it :D