search

paulhibbing / AGread

Read Accelerometer Files from ActiGraph Accelerometers
Other
15 stars 4 forks source link

Unable to get raw acc data from gt3x file with read_gt3x() #7

Closed martakarass closed 5 years ago

martakarass commented 5 years ago

I have gt3x file with ~40 minutes of raw accelerometry data collected at 100 Hz, with idle mode enabled. I have corresponding ActiLife raw data output CSV. I try to use AGread::read_gt3x to replicate ActiLife raw data output. My attempts failed (see "Read gt3x file with read_gt3x: Attempt 1", "Read gt3x file with read_gt3x: Attempt 2" bullet points below). Perhaps I am misusing the function? 

rm(list = ls())
library(data.table)
library(AGread)

## Personal dropbox sharing links
csv.fpath <- "https://www.dropbox.com/s/hhdqjntarqxgpuu/TAS1H30182785%20%282019-09-17%29RAW.csv?dl=1"
gt3x.fpath <- "https://www.dropbox.com/s/pi7m9b75gqzl64g/TAS1H30182785%20%282019-09-17%29.gt3x?dl=1"
gt3x.destfile <- "TAS1H30182785 (2019-09-17).gt3x"

## Download gt3x to wd
if (!file.exists(gt3x.destfile)) download.file(gt3x.fpath, gt3x.destfile) 
as.character(unlist(read.csv(file = csv.fpath, nrows = 6)))
# [1] "Serial Number: TAS1H30182785"     "Start Time 18:40:00"             
# [3] "Start Date 9/17/2019"             "Epoch Period (hh:mm:ss) 00:00:00"
# [5] "Download Time 19:20:05"           "Download Date 9/17/2019" 

dat1 <- as.data.frame(fread(csv.fpath))

## Expected vs actual number of observations in raw data CSV Actilife output 
hz <- 100 
collection_dur_s <- as.numeric(difftime(as.POSIXct("2019-09-17 19:20:05"), as.POSIXct("2019-09-17 18:40:00"), units = "secs"))
nrow_exp <- hz * collection_dur_s
nrow_act <- nrow(dat1)
c(nrow_exp, nrow_act) 
# [1] 240500 240500

Fails.

dat4 <- read_gt3x(gt3x.destfile, verbose = TRUE)
# Processing TAS1H30182785 (2019-09-17).gt3x 
# 
# Parsing info.txt  ............. COMPLETE
# 
# Will parse the following packet types, if available:
#   METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY 
# EVENT, TAG, ACTIVITY, HEART_RATE_BPM 
# HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE 
# EPOCH, EPOCH2, EPOCH3, EPOCH4 
# ACTIVITY2, SENSOR_DATA 
# 
# Reading log.bin  ............. COMPLETE
# Getting record headers............... COMPLETE
# Parsing PARAMETERS packet(s)   ............. COMPLETE                      
# Parsing METADATA packet(s)   ............. COMPLETE                      
# Parsing BATTERY packet(s)   ............. COMPLETE                      
# Parsing EVENT packet(s)   ............. COMPLETE                      
# Parsing CAPSENSE packet(s)   ............. COMPLETE                      
# Parsing ACTIVITY2 packet(s)   ............. 96%Error in payload_parse_activity2_26(payload, info, is_last_packet) : 
#   test_pass is not TRUE

No "ACTIVITY" packet in the resulted object? Hence no activity data have been read? I cannot identify it/retrieve it from resulted object dat4. 

## Comment out "ACTIVITY2" which caused problem above
dat4 <- read_gt3x(gt3x.destfile, verbose = TRUE, include = c(
  "METADATA", "PARAMETERS", "SENSOR_SCHEMA", "BATTERY", "EVENT",
  "TAG", "ACTIVITY", "HEART_RATE_BPM", "HEART_RATE_ANT", "HEART_RATE_BLE",
  "LUX", "CAPSENSE", "EPOCH", "EPOCH2", "EPOCH3", "EPOCH4", # "ACTIVITY2", 
  "SENSOR_DATA"))

# Processing TAS1H30182785 (2019-09-17).gt3x 
# 
# Parsing info.txt  ............. COMPLETE
# 
# Will parse the following packet types, if available:
#   METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY 
# EVENT, TAG, ACTIVITY, HEART_RATE_BPM 
# HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE 
# EPOCH, EPOCH2, EPOCH3, EPOCH4 
# SENSOR_DATA 
# 
# Reading log.bin  ............. COMPLETE
# Getting record headers............... COMPLETE
# Parsing PARAMETERS packet(s)   ............. COMPLETE                      
# Parsing METADATA packet(s)   ............. COMPLETE                      
# Parsing BATTERY packet(s)   ............. COMPLETE                      
# Parsing EVENT packet(s)   ............. COMPLETE                      
# Parsing CAPSENSE packet(s)   ............. COMPLETE                      
# 
# Processing complete. Elapsed time 0.005 minutes.

names(dat4)
# [1] "METADATA"   "BATTERY"    "EVENT"      "CAPSENSE"   "PARAMETERS"
paulhibbing commented 5 years ago

@martakarass From what I'm seeing, my guess is you don't have v1.0.0 installed. Most likely that's my fault for advertising it as "available on CRAN" when it may not be built for your operating system for another 24 h or so. You can install it now using `devtools::install_github("paulhibbing/AGread").

That said, I'm glad you shared your files, because I'm seeing new differences between read_gt3x and read_AG_raw, which appear to be related to either USB events or idle sleep mode periods. I thought I had that figured out, but I guess not. I'm going to look into it more and report back in another issue. In the meantime, here's a script you may find useful.

# Setup -------------------------------------------------------------------

    rm(list = ls())

  ## Check that AGread v1.0.0 is installed

    stopifnot("AGread" %in% installed.packages())

    expected_version <- structure(
      list(c(1L, 0L, 0L)),
      class = c("package_version", "numeric_version")
    )

    installed_version <- packageVersion("AGread")

    if (!identical(expected_version, installed_version)) {
      stop("It appears you don\'t have AGread v1.0.0 installed.")
    } else {
      rm(expected_version, installed_version)
    }

  ## Check that R can locate the gt3x and RAW.csv files

    file_directory <- getwd()
    file_3x <- file.path(
      file_directory, "TAS1H30182785 (2019-09-17).gt3x"
    )
    file_csv <- file.path(
      file_directory, "TAS1H30182785 (2019-09-17)RAW.csv"
    )

    if (!all(file.exists(c(file_3x, file_csv)))) {
      stop(paste(
        "Can\'t find the data files. Try adjusting the",
        "\n  value of `file_directory` object."
      ))
    } else {
      rm(file_directory)
      library(AGread)
    }

# Read Files and prepare for comparison -----------------------------------

    all_3x <- read_gt3x(file_3x, verbose = TRUE)
    test_3x <- all_3x$RAW
    class(test_3x) <- "data.frame"

    test_RAW <- read_AG_raw(
      file_csv, return_raw = TRUE, verbose = TRUE
    )
    test_RAW <- test_RAW[ ,names(test_3x)]

# Compare the output of both files ----------------------------------------

    accel_names <- setdiff(names(test_3x), "Timestamp")

    PAutilities::test_errors(
      test_3x, test_RAW, accel_names,
      return_logical = FALSE
    )

    all.equal(test_3x, test_RAW, scale = 1) # Timestamps are OK

    sapply(
      accel_names,
      function(x) summary(
      abs(test_3x[ ,x] - test_RAW[ ,x])
      ),
      simplify = FALSE
    )
martakarass commented 5 years ago

Thanks a ton! The AGread:: read_gt3x does complete for me with v1.0.0! I see discrepancies your code shows. They are having quite a systematic-ish pattern. Pasting here for reference to compare with that file on future versions. 

library(ggplot2)

## Value of difference 
agree_x <- test_RAW$Accelerometer_X - test_3x$Accelerometer_X
agree_y <- test_RAW$Accelerometer_Y - test_3x$Accelerometer_Y
agree_z <- test_RAW$Accelerometer_Z - test_3x$Accelerometer_Z

## Plot them
rn_n <- length(agree_x)
df <- data.frame(x = rep(1:rn_n, 3),
                 y = c(agree_x, agree_y, agree_z),
                 gr = c(rep("Accelerometer_X", rn_n), rep("Accelerometer_Y", rn_n), rep("Accelerometer_Z", rn_n)))
plt <- 
  ggplot(df, aes(x = x, y = y, group = 1)) + 
  geom_hline(yintercept = 0, color  = "red", alpha = 0.5) + 
  geom_line() + 
  facet_grid(gr ~ .) + 
  labs(title = "Value of difference in measurement\n(ActiLife raw data output) - (AGread::read_gt3x read)", 
       x = "obs_idx",
       y = "(ActiLife raw data output) - (AGread::read_gt3x read)") + 
  theme_grey(base_size = 10)
plt

paulhibbing commented 5 years ago

@martakarass This should be fixed as of https://github.com/paulhibbing/AGread/commit/627b6d28d5fb8b0d26144e8655b992fae7954059.

Running devtools::install_github("paulhibbing/AGread") should install v1.1.0.9000 (under development), and you should see identical results for primary accelerometer data via read_gt3x and read_AG_raw. Let me know if not.

paulhibbing commented 5 years ago

Also, I forgot to mention you can now view the idle sleep mode information in the EVENT component of the output, e.g. test_RAW$EVENT.

martakarass commented 5 years ago

Works 100% correct on this example! Thanks. I appreciate a ton.