detecthist

[jimcasaer]

something apparently goes wrong when unsing the detectionHistory command in the latest version to perform occ calculations the cam_op now looks perfect and the o1,o2, also but the first and last day in the station are all decimals in the result of detectionHistory and should be 1 or 0 in order to get correct results for occ calculation

[jimcasaer]

error message what is the meaning of following error message (see at the end) start_date <- lubridate::as_datetime("2019-10-01") end_date <- lubridate::as_datetime("2020-3-31")

cam_op <- get_cam_op(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), station_col = "gridcell", use_prefix = FALSE)

rec_table <- get_record_table(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), stationCol = "gridcell", exclude = NULL, minDeltaTime = 0, deltaTimeComparedTo = NULL )

Dethist_wb <- detectionHistory(recordTable = rec_table, camOp = cam_op,
stationCol = "Station", speciesCol = "Species", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat ="%Y-%m-%d %H:%M:%S", species = "Sus scrofa", occasionLength = 1, day1 = "station", datesAsOccasionNames = FALSE, occasionStartTime = 0, includeEffort = FALSE, timeZone = "UTC", writecsv = FALSE)

recordTable was converted from tibble to data.frame Error in if (any(date_ranges2$start_first_occasion > date_ranges2$end_last_occasion)) { : missing value where TRUE/FALSE needed

[damianooldoni]

It means something strange happens within detectionHistory 😄 the any() statement doesn't return TRUE or FALSE and so the if() statement returns the error you see. I will debug your case study and I will get to the source of the error.

[jimcasaer]

I broke the issue rules by putting two different problems in one issue :-) so the first remark deals with another problem of detecthistory than the second :-)

[damianooldoni]

Thanks @jimcasaer for telling me this 😄 ! Still, as your issue title is generic, any example where the fucntion doesn't return what it is expected to return can be mentioned in this issue. @jimcasaer: could you please write here below the code which returns the decimal values in detectionHistory? Thanks.

[jimcasaer]

same as previous ex. but with these start and end date no mistakes however had to add line : dethist <- dethist[, 2:25] to get correct results because of decimals in detecthistory otherwise

start_date <- lubridate::as_datetime("2018-04-01") end_date <- lubridate::as_datetime("2018-08-31")

cam_op <- get_cam_op(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), station_col = "gridcell", use_prefix = FALSE )

rec_table <- get_record_table(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), stationCol = "gridcell", exclude = NULL, minDeltaTime = 0, deltaTimeComparedTo = NULL )

Dethist_wb <- detectionHistory(recordTable = rec_table, camOp = cam_op,
stationCol = "Station", speciesCol = "Species", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat ="%Y-%m-%d %H:%M:%S", species = "Sus scrofa", occasionLength = 1, day1 = "station", datesAsOccasionNames = FALSE, occasionStartTime = 0, includeEffort = FALSE, timeZone = "UTC", writecsv = FALSE)

dethist.temp <- Dethist_wb[[1]] dethist <- as.data.frame(dethist.temp) dethist <- dethist[, 2:25] occupcov <- unmarkedFrameOccu(y = dethist) null_p <- occu(~1~1, occupcov) # Model without covariates
backTransform(null_p, 'det') # Backtransformation backTransform(null_p, 'state')

[damianooldoni]

Different things after meeting with @jimcasaer and some tests:

1. The strange decimals (equal to effort values) at the first and end day of a station are not due to the absence of observations during that day. I simulated such situation with mica dummy datapackage and everything works perfect
2. @jimcasaer: the column "gridcell" doesn't exist in deployments and so I get an error while using get_cam_op() to create the camera operation matrix. Notice that this will make me improve the code as at the moment the error is quite obscure just because I forgot to check the argument station_col at the very beginning of the function (defensive programming strategy always helps in this case!). To be posted in a separate issue.
3. I am afraid the warnings we get while reading GMU8 are quite important as they are linked to 13 "missing" timestamps in observations.csv! I see that the datetime in these cases is equal to just +01:00 and is therefore imported as "NA"I am just wondering what happens if you remove these rows from the observations:

   problems_idx <- c(21220,
                 24177,
                 28261,
                 30361,
                 73332,
                 100780,
                 102968,
                 102969,
                 102970,
                 102971,
                 109909,
                 109910,
                 114661)
   GMU8$observations <- GMU8$observations[-problems_idx,]

[jimcasaer]

gridcell is a selfcreated station column --therefore it is missing when you do not run the full script replacing it by locationName does not solve the decimal problem :-(

[damianooldoni]

Indeed, but if you remove the observations with NA as timestamp everything works fine on my laptop.

Try this code:

GMU8 <- read_camtrap_dp(file = "./inst/extdata/gmu8/datapackage.json", media = TRUE)
problems_idx <- c(21220,
                  24177,
                  28261,
                  30361,
                  73332,
                  100780,
                  102968,
                  102969,
                  102970,
                  102971,
                  109909,
                  109910,
                  114661)
GMU8$observations <- GMU8$observations[-problems_idx,]

start_date <- lubridate::as_datetime("2019-04-01")
end_date <- lubridate::as_datetime("2019-08-31")

cam_op <- get_cam_op(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)),
                     station_col = "locationName",
                     use_prefix = FALSE
)

rec_table <- get_record_table(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)),
                              stationCol = "locationName",
                              exclude = NULL,
                              minDeltaTime = 0,
                              deltaTimeComparedTo = NULL
)

Dethist_wb <- detectionHistory(recordTable = rec_table,
                               camOp = cam_op,
                               stationCol = "Station",
                               speciesCol = "Species",
                               recordDateTimeCol = "DateTimeOriginal",
                               recordDateTimeFormat ="%Y-%m-%d %H:%M:%S",
                               species = "Sus scrofa",
                               occasionLength = 1,
                               day1 = "station",
                               datesAsOccasionNames = FALSE,
                               occasionStartTime = 0,
                               includeEffort = FALSE,
                               timeZone = "UTC",
                               writecsv = FALSE)

dethist <- as.data.frame(Dethist_wb$detection_history)
View(dethist)

I get this detection history (screenshot):

And as my eyes are bad, I prefer to perform a robust check by checking if all columns contain only NAs or integer values (floor(x) == x):

# perform the check column-wise using purrr pacakge
are_cols_whole_numbers <- purrr::map_lgl(
  dethist, 
  function(x) {
    x <- x[!is.na(x)]
    if (length(x) > 0) 
      {all(floor(x) == x)
    } else {
      TRUE
    }
  })
# is the result of the check equal to TRUE for all columns?
all(are_cols_whole_numbers == TRUE)

So, I really think the parsing issues while reading the datetime of the observations are really the source of all problems.

[jimcasaer]

seems that there also can be problems with Date time format of start and end date of deployments

see example I sent you jim

Op ma 2 mei 2022 21:54 schreef Damiano Oldoni @.***>:

Indeed, but if you remove the observations with NA as timestamp everything works fine on my laptop.

Try this code:

GMU8 <- read_camtrap_dp(file = "./inst/extdata/gmu8/datapackage.json", media = TRUE)problems_idx <- c(21220, 24177, 28261, 30361, 73332, 100780, 102968, 102969, 102970, 102971, 109909, 109910, 114661)GMU8$observations <- GMU8$observations[-problems_idx,] start_date <- lubridate::as_datetime("2019-04-01")end_date <- lubridate::as_datetime("2019-08-31") cam_op <- get_cam_op(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), station_col = "locationName", use_prefix = FALSE ) rec_table <- get_record_table(GMU8, pred_and(pred_gte("start", start_date), pred_lte("end", end_date)), stationCol = "locationName", exclude = NULL, minDeltaTime = 0, deltaTimeComparedTo = NULL ) Dethist_wb <- detectionHistory(recordTable = rec_table, camOp = cam_op, stationCol = "Station", speciesCol = "Species", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat ="%Y-%m-%d %H:%M:%S", species = "Sus scrofa", occasionLength = 1, day1 = "station", datesAsOccasionNames = FALSE, occasionStartTime = 0, includeEffort = FALSE, timeZone = "UTC", writecsv = FALSE) dethist <- as.data.frame(Dethist_wb$detection_history) View(dethist)

I get this detection history (screenshot): [image: image] https://user-images.githubusercontent.com/33662631/166313349-189680da-63d5-4d1c-899d-3de8d37edfc7.png

And as my eyes are bad, I prefer to perform a robust check by checking if all columns contain only NAs or integer values (floor(x) == x):

perform the check column-wise using purrr pacakgeare_cols_whole_numbers <- purrr::map_lgl(

dethist, function(x) { x <- x[!is.na(x)] if (length(x) > 0) {all(floor(x) == x) } else { TRUE } })# is the result of the check equal to TRUE for all columns? all(are_cols_whole_numbers == TRUE)

So, I really think the parsing issues while reading the datetime of the observations are really the source of all problems.

— Reply to this email directly, view it on GitHub https://github.com/inbo/camtraptor/issues/105#issuecomment-1115298974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHMZ2B3ICE4HUNUFQ4Y3XTVIAXGNANCNFSM5UZ3AD4Q . You are receiving this because you were mentioned.Message ID: @.***>

[damianooldoni]

Yes, I will check them too, thanks @jimcasaer! What I was trying to say is that parsing issues while reading the datapackage (in observations and deployments) can appear quite "innocent" warnings, but can result in quite dramatic consequences.

[jimcasaer]

strange, when I run those lines (first comment) -- I still see the same decimal problem ,

[damianooldoni]

It seems I cannot reproduce the error and you cannot reproduce my example without error. So, I suggest you to:

1. Close all open RStudio sessions.
2. Open basic R. Yes, the old school R with the basic console. If more than one R version are installed, please open the one which is coupled to your RStudio. How do you know which one it is? Via Tools -> Global Options and check path in R version field, see screenshot.
3. In console, run update.packages(ask=FALSE) to upload ALL packages on your laptop. Argument ask=FALSE is just to not be annoyed by pop ups where you have to confirm the update by clicking "Yes" for each package. It can be really annoying!
4. Try again my code above in RStudio and I hope you are rid of decimals. Meanwhile I check your other datapckage, saint.zip.

[jimcasaer]

error still when using "gridcell" but I suspect this is because the same gridcell is visited more than once => I therefore close the issue for now

[jimcasaer]

or do you suggest to leave it open ? @damianooldoni

[damianooldoni]

I think we can close this issue which focuses on decimal values in the detection history. In this way we have a better view of what is working well and what is not working. So, if you think you get an error which shouldn't occur, please, open a new issue.

[damianooldoni]

Closing this issue as a new has been opened (#110 ) as I preferred.