eTUFF Microwave Telemetry Inputs

[marosteg]

1. For filtering of delta-limited depth and temperature: Incorporate an IF statement in eTUFF generation for Microwave including "delta" columns for depth and temperature within etuff$etuff (maybe rename to 'deltaD' and 'deltaT') to enable exclusion of these spurious measurements from the eventual PDTs.

2. For filtering light-based location estimates: Incorporate an IF statement in eTUFF generation for Microwave including "mean" columns for latitude and longitude to enable eventual detection of raw locations that are >/= X degrees from the mean to be excluded from likelihoods.

[camrinbraun]

1. For filtering of delta-limited depth and temperature: Incorporate an IF statement in eTUFF generation for Microwave including "delta" columns for depth and temperature within etuff$etuff (maybe rename to 'deltaD' and 'deltaT') to enable exclusion of these spurious measurements from the eventual PDTs.

What is it that you want from here? Just the Delta(val) column? Can you provide a little more info on why you want this so we have a record of that here?

[marosteg]

The delta value corresponds to the change in the metric from one hour previous and reflects how MTI compresses data. The simplified version is that if depth and/or temperature change by more than a certain amount, the resulting measurement is "delta-limited" and likely not accurate. However, the "Delta(val)" column does not indicate which entries are delta-limited (the 'X' indicates an accurate reference value, it does not mean that the entry is bad). Instead, the measurements that are delta-limited are provided in two columns for each metric: 1) TEMPERATURE "Δ Lim +Temp" and "Δ Lim -Temp" , 2) DEPTH "Δ Lim Dives" and "Δ Lim Ascents". A delta-limited measurement can only be one of these at a time, given that they have explicit directionality relative to the reference value (up or down).

In a given time step, only depth, only temperature, or both may be delta-limited. Thus, we need to incorporate the "Δ Lim..." columns that are time-matched to the depth and temperature measurements. This will facilitate rapid exclusion of delta-limited entries that could introduce error into the resulting PDTs and HYCOM-related likelihoods.

[camrinbraun]

this is ready for testing in #28

[marosteg]

Providing this as an example of how to filter out delta-limited temperature entries using the new etuff$qc addition:

#### 3.1.2 Depth-Temperature Data ####
# DEPTH-TEMPERATURE TIME SERIES DATA example showing how to coerce
# depth-temp time series to a PDT-like summarized product
ts <- as.data.frame(cbind(etuff$etuff$DateTime,etuff$etuff$depth,etuff$etuff$temperature))
colnames(ts) <- c("Date","Depth","Temperature")
ts$Date <- as.POSIXct(ts$Date,origin='1970-01-01',tz="UTC")

## filter out delta-limited entries (MTI-specific)
# creates new df only with the rows containing delta-limited temperatures (up or down)
qc <- etuff$qc[which(is.na(etuff$qc$temperature_deltaLimPos)==F|is.na(etuff$qc$temperature_deltaLimNeg)==F),]
# removes columns not needed for matching or filtering
qc <- qc %>% select(-latitude_avg,-longitude_avg,-id)
# rename column name to match ts format
colnames(qc)[which(colnames(qc)=="DateTime")] <- "Date"
# merge qc df to ts df by the date/time
ts <- left_join(ts, qc, by = "Date")
# eliminate rows with missing temp/depth data or flagged as delta-limited
ts_index <- is.na(ts$Temperature)|is.na(ts$Depth)|is.na(ts$temperature_deltaLimPos)==F|is.na(ts$temperature_deltaLimNeg)==F
ts <- ts[!ts_index,]

#business as usual below
pdt_dates <- seq.POSIXt(tag, pop, by = "day") ## generate example daily summary of depth-temp profiles
## generate depth-temp summary from time series
pdt <- bin_TempTS(ts, out_dates = pdt_dates, bin_res = 5)
pdt <- pdt[, c("Date", "Depth", "MinTemp", "MaxTemp")]
head(pdt)

[marosteg]

Providing this as an example of how to filter out light-based lat/lon entries using the new etuff$qc addition:

#### 3.1.3 Light Data ####
### Light-based position estimates
ll <- as.data.frame(cbind(etuff$etuff$DateTime,etuff$etuff$longitude,etuff$etuff$latitude))
colnames(ll) <- c("Date","Longitude","Latitude")
ll$Date <- as.POSIXct(ll$Date,origin='1970-01-01',tz="UTC")
ll_index <- is.na(ll$Longitude)|is.na(ll$Latitude)
ll <- ll[!ll_index,]
ll <- ll[which(ll$Date <= pop),]
ll$Error.Semi.minor.axis = .7 * 1000 * 111

## filter out lat/lon entries by difference from average (MTI-specific)
# creates new df only with the rows containing avg lat/lon estimates
qc <- etuff$qc[which(is.na(etuff$qc$latitude_avg)==F|is.na(etuff$qc$longitude_avg)==F),]
# removes columns not needed for matching or filtering
qc <- qc %>% select(-temperature_deltaLimPos,-temperature_deltaLimNeg,-id)
# rename column name to match ts format
colnames(qc)[which(colnames(qc)=="DateTime")] <- "Date"
# merge qc df to ll df by the date/time
ll <- left_join(ll, qc, by = "Date")
# calculate absolute difference between raw and avg lat/lon estimates
ll$latitude_diff <- abs(ll$Latitude-ll$latitude_avg)
ll$longitude_diff <- abs(ll$Longitude-ll$longitude_avg)

# In place of the old lightloc, create separate latloc and lonloc dfs, filtering out raw estimates > 1 degree from average
# The > X degree should be chosen in the context of your focal species' max daily displacement
latloc <- ll[which(ll$latitude_diff<=1),]
lonloc <- ll[which(ll$longitude_diff<=1),]

head(latloc)
head(lonloc)