No Activity Data Parsed

[muschellij2]

Issue

Example gt3x data has no parsed Activity - maybe Activity2 packet?

I'm adding @THLfi to the mix because I want to note that AGread parses a lot of additional, likely useful information from gt3x compared to read.gt3x. Also, the timing differences are quite different, but again the amount of info parsed is very different.

library(AGread)
#> package 'AGread' was built under R version 3.5.0
library(read.gt3x)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
library(readr)
zipfile = "https://web.stanford.edu/~hyatt4/content/software/padaco/sampleData.zip"
outfile = file.path(tempdir(), basename(zipfile))
if (!file.exists(outfile)) {
  dl = download.file(zipfile, outfile)
}
unz = unzip(outfile, exdir = tempdir())

csv_file = unz[basename(unz) == "sample_1sec.csv"]
csv_data <- readr::read_csv(
  csv_file, skip = 10,
  col_types = 
    cols(
      Date = col_character(),
      Time = col_character(),
      Axis1 = col_double(),
      Axis2 = col_double(),
      Axis3 = col_double(),
      Steps = col_double(),
      Lux = col_double(),
      `Inclinometer Off` = col_double(),
      `Inclinometer Standing` = col_double(),
      `Inclinometer Sitting` = col_double(),
      `Inclinometer Lying` = col_double(),
      `Vector Magnitude` = col_double()
    ))
csv_data$Date = paste0(csv_data$Date, " ", csv_data$Time)
csv_data$Date = mdy_hms(csv_data$Date)
head(csv_data)
#> # A tibble: 6 x 12
#>   Date                Time  Axis1 Axis2 Axis3 Steps   Lux `Inclinometer O…
#>   <dttm>              <chr> <dbl> <dbl> <dbl> <dbl> <dbl>            <dbl>
#> 1 2015-12-09 00:00:00 00:0…    35     0   148     0     0                0
#> 2 2015-12-09 00:00:01 00:0…     0     0    65     0     0                1
#> 3 2015-12-09 00:00:02 00:0…     0     0    39     0     0                1
#> 4 2015-12-09 00:00:03 00:0…     0     0     0     0     0                1
#> 5 2015-12-09 00:00:04 00:0…     0     0     0     0     0                1
#> 6 2015-12-09 00:00:05 00:0…     0     0     0     0     0                1
#> # … with 4 more variables: `Inclinometer Standing` <dbl>, `Inclinometer
#> #   Sitting` <dbl>, `Inclinometer Lying` <dbl>, `Vector Magnitude` <dbl>

gt3x_file = unz[basename(unz) == "sample.gt3x"]
system.time({
  ag = read_gt3x(gt3x_file)
})
#> Warning in parse_packet_set.default(X[[i]], ...): No method exists yet for
#> parsing ACTIVITY packets -- returning NULL.
#> Warning in parse_packet_set.default(X[[i]], ...): No method exists yet for
#> parsing LUX packets -- returning NULL.
#>    user  system elapsed 
#>  16.743   3.715  29.590
head(ag$CAPSENSE)
#>             Timestamp signal reference    state bursts
#> 1 2015-12-09 00:00:59    546       546 Not Worn     10
#> 2 2015-12-09 00:01:59    546       546 Not Worn     10
#> 3 2015-12-09 00:02:59    546       546 Not Worn     10
#> 4 2015-12-09 00:03:59    546       546 Not Worn     10
#> 5 2015-12-09 00:04:59    546       546 Not Worn     10
#> 6 2015-12-09 00:05:59    546       546 Not Worn     10
head(ag$ACTIVITY)
#> NULL
head(ag$BATTERY)
#>             Timestamp battery_voltage
#> 1 2015-12-08 18:55:40           4.164
#> 2 2015-12-08 18:56:40           4.164
#> 3 2015-12-08 18:57:40           4.164
#> 4 2015-12-08 19:05:40           4.175
#> 5 2015-12-08 19:06:40           4.175
#> 6 2015-12-08 19:07:40           4.175

system.time({
  rg = read.gt3x(gt3x_file, asDataFrame = TRUE, 
                 imputeZeroes = TRUE, debug = TRUE)
})
#> Reading Stream...
#> imputing 40 values at index 29100720 
#> imputing 4080 values at index 29100760
#>    user  system elapsed 
#>   4.192   2.115  13.663
hdr = attributes(rg)
sr = hdr$sample_rate
rg$ms = as.numeric(rg$time) %% 1
rg$diff_time = c(NA, diff(rg$time))
# 
as.data.frame(rg[ 1:(sr), ])
#>        X      Y      Z                time         ms  diff_time
#> 1  0.000  0.000  0.000 2015-12-09 00:00:00 0.00000000         NA
#> 2  0.102 -0.145  0.094 2015-12-09 00:00:00 0.07500005 0.07500005
#> 3  0.262  0.020 -0.848 2015-12-09 00:00:00 0.12500000 0.04999995
#> 4  0.262  0.031 -0.926 2015-12-09 00:00:00 0.20000005 0.07500005
#> 5  0.266  0.035 -0.938 2015-12-09 00:00:00 0.25000000 0.04999995
#> 6  0.262  0.031 -0.938 2015-12-09 00:00:00 0.32500005 0.07500005
#> 7  0.262  0.035 -0.938 2015-12-09 00:00:00 0.37500000 0.04999995
#> 8  0.266  0.035 -0.934 2015-12-09 00:00:00 0.45000005 0.07500005
#> 9  0.262  0.039 -0.934 2015-12-09 00:00:00 0.50000000 0.04999995
#> 10 0.266  0.035 -0.934 2015-12-09 00:00:00 0.57500005 0.07500005
#> 11 0.258  0.035 -0.926 2015-12-09 00:00:00 0.62500000 0.04999995
#> 12 0.266  0.039 -0.934 2015-12-09 00:00:00 0.70000005 0.07500005
#> 13 0.262  0.035 -0.934 2015-12-09 00:00:00 0.75000000 0.04999995
#> 14 0.266  0.035 -0.938 2015-12-09 00:00:00 0.82500005 0.07500005
#> 15 0.262  0.035 -0.934 2015-12-09 00:00:00 0.87500000 0.04999995
#> 16 0.262  0.035 -0.934 2015-12-09 00:00:00 0.95000005 0.07500005
#> 17 0.262  0.035 -0.930 2015-12-09 00:00:01 0.00000000 0.04999995
#> 18 0.266  0.035 -0.934 2015-12-09 00:00:01 0.07500005 0.07500005
#> 19 0.270  0.031 -0.934 2015-12-09 00:00:01 0.12500000 0.04999995
#> 20 0.270  0.035 -0.938 2015-12-09 00:00:01 0.20000005 0.07500005
#> 21 0.270  0.035 -0.938 2015-12-09 00:00:01 0.25000000 0.04999995
#> 22 0.266  0.039 -0.934 2015-12-09 00:00:01 0.32500005 0.07500005
#> 23 0.266  0.035 -0.934 2015-12-09 00:00:01 0.37500000 0.04999995
#> 24 0.270  0.031 -0.934 2015-12-09 00:00:01 0.45000005 0.07500005
#> 25 0.266  0.035 -0.938 2015-12-09 00:00:01 0.50000000 0.04999995
#> 26 0.262  0.035 -0.930 2015-12-09 00:00:01 0.57500005 0.07500005
#> 27 0.262  0.031 -0.938 2015-12-09 00:00:01 0.62500000 0.04999995
#> 28 0.262  0.035 -0.934 2015-12-09 00:00:01 0.70000005 0.07500005
#> 29 0.262  0.031 -0.938 2015-12-09 00:00:01 0.75000000 0.04999995
#> 30 0.266  0.035 -0.934 2015-12-09 00:00:01 0.82500005 0.07500005
#> 31 0.266  0.035 -0.930 2015-12-09 00:00:01 0.87500000 0.04999995
#> 32 0.262  0.035 -0.934 2015-12-09 00:00:01 0.95000005 0.07500005
#> 33 0.266  0.035 -0.938 2015-12-09 00:00:02 0.00000000 0.04999995
#> 34 0.262  0.039 -0.934 2015-12-09 00:00:02 0.07500005 0.07500005
#> 35 0.262  0.035 -0.934 2015-12-09 00:00:02 0.12500000 0.04999995
#> 36 0.266  0.035 -0.930 2015-12-09 00:00:02 0.20000005 0.07500005
#> 37 0.262  0.039 -0.938 2015-12-09 00:00:02 0.25000000 0.04999995
#> 38 0.266  0.035 -0.934 2015-12-09 00:00:02 0.32500005 0.07500005
#> 39 0.262  0.035 -0.930 2015-12-09 00:00:02 0.37500000 0.04999995
#> 40 0.262  0.039 -0.938 2015-12-09 00:00:02 0.45000005 0.07500005

^{Created on 2020-02-12 by the reprex package (v0.3.0.9001)}

Session info

``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 3.6.2 (2019-12-12) #> os macOS Mojave 10.14.6 #> system x86_64, darwin15.6.0 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz America/New_York #> date 2020-02-12 #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date lib source #> AGread * 1.1.0.9000 2019-10-03 [1] local #> anytime 0.3.6 2019-08-29 [1] CRAN (R 3.6.0) #> assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0) #> backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.0) #> binaryLogic 0.3.9 2017-12-13 [1] CRAN (R 3.6.0) #> cli 2.0.1 2020-01-08 [1] CRAN (R 3.6.0) #> colorspace 1.4-1 2019-03-18 [1] CRAN (R 3.6.0) #> crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0) #> digest 0.6.23 2019-11-23 [1] CRAN (R 3.6.0) #> dplyr 0.8.3 2019-07-04 [1] CRAN (R 3.6.0) #> evaluate 0.14 2019-05-28 [1] CRAN (R 3.6.0) #> fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.0) #> fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.0) #> ggplot2 3.2.1 2019-08-10 [1] CRAN (R 3.6.0) #> glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.0) #> gtable 0.3.0 2019-03-25 [1] CRAN (R 3.6.0) #> highr 0.8 2019-03-20 [1] CRAN (R 3.6.0) #> hms 0.5.2 2019-10-30 [1] CRAN (R 3.6.0) #> htmltools 0.4.0 2019-10-04 [1] CRAN (R 3.6.0) #> knitr 1.26.1 2020-01-05 [1] Github (muschellij2/knitr@f5af631) #> lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.0) #> lifecycle 0.1.0 2019-08-01 [1] CRAN (R 3.6.0) #> lubridate * 1.7.4 2018-04-11 [1] CRAN (R 3.6.0) #> magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.0) #> munsell 0.5.0 2018-06-12 [1] CRAN (R 3.6.0) #> PAutilities 0.2.0 2019-07-10 [1] CRAN (R 3.6.0) #> pillar 1.4.3 2019-12-20 [1] CRAN (R 3.6.0) #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 3.6.0) #> purrr 0.3.3 2019-10-18 [1] CRAN (R 3.6.0) #> R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.0) #> Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.0) #> read.gt3x * 0.1.0.9000 2020-01-28 [1] local #> readr * 1.3.1 2018-12-21 [1] CRAN (R 3.6.0) #> reprex 0.3.0.9001 2020-01-05 [1] Github (tidyverse/reprex@5ae0b29) #> rlang 0.4.2 2019-11-23 [1] CRAN (R 3.6.0) #> rmarkdown 2.0.7 2020-01-17 [1] Github (rstudio/rmarkdown@2faf16a) #> rstudioapi 0.10.0-9003 2020-01-05 [1] Github (rstudio/rstudioapi@abe596d) #> scales 1.1.0 2019-11-18 [1] CRAN (R 3.6.0) #> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.0) #> stringi 1.4.5 2020-01-11 [1] CRAN (R 3.6.0) #> stringr 1.4.0 2019-02-10 [1] CRAN (R 3.6.0) #> styler 1.1.1 2019-05-06 [1] CRAN (R 3.6.0) #> tibble 2.1.3 2019-06-06 [1] CRAN (R 3.6.0) #> tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.6.0) #> utf8 1.1.4 2018-05-24 [1] CRAN (R 3.6.0) #> vctrs 0.2.1 2019-12-17 [1] CRAN (R 3.6.0) #> withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0) #> xfun 0.11 2019-11-12 [1] CRAN (R 3.6.0) #> yaml 2.2.0 2018-07-25 [1] CRAN (R 3.6.0) #> zeallot 0.1.0 2018-01-28 [1] CRAN (R 3.6.0) #> #> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library ```

[paulhibbing]

Thanks @muschellij2. This is a robustness issue, somewhat similar to #10. In this file the accelerometer packets are in ACTIVITY format, whereas read_gt3x currently supports only ACTIVITY2 parsing.

One of the limitations for me has been that I haven't encountered certain packet types in the wild. This file has both ACTIVITY and LUX packets (neither of which I've dealt with), so it will potentially serve as a nice way to get those set up. That said, it's going to be months before I can get to it.

[muschellij2]

I agree - definitely an enhancement. The lessons from this example is read_gt3x can't read ACTIVITY packets, ACTIVITY packets are not constrained to "old" GT3X formats, and read.gt3x from @THLfi can read it, but doesn't parse additional information such as LUX or info like CAPSENSE.

[JWiley]

Just a note that if more examples with with the newer activity format and LUX would help I have plenty and would be happy to share some. In sleep research, the lux data could be very interesting and would be neat to get through R rather than having to export it manually.

[TheTS]

Just experienced this - good to know it's because of the ACTIVITY packets!

>   df <- read_gt3x(file, verbose = T)

Processing Laura (2020-08-16).gt3x 

  Unzipping  ............. COMPLETE
  Checking components  ............. COMPLETE
  Retrieving info from the zip archive  ............. COMPLETE

  Will parse the following packet types, if available:
    METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY 
    EVENT, TAG, ACTIVITY, HEART_RATE_BPM 
    HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE 
    EPOCH, EPOCH2, EPOCH3, EPOCH4 
    ACTIVITY2, SENSOR_DATA 

  Reading log.bin  ............. COMPLETE
  Getting record headers  ............... COMPLETE
  Parsing PARAMETERS packet(s)   ............. COMPLETE                      
  Parsing EVENT packet(s)   ............. COMPLETE                      
  Parsing METADATA packet(s)   ............. COMPLETE                      
  Parsing BATTERY packet(s)   ............. COMPLETE                      
  Parsing ACTIVITY packet(s)
  Parsing LUX packet(s)

Processing complete. Elapsed time 0.08 minutes.
Warning messages:
1: In parse_packet_set.default(X[[i]], ...) :
  No method exists yet for parsing ACTIVITY packets -- returning NULL.
2: In parse_packet_set.default(X[[i]], ...) :
  No method exists yet for parsing LUX packets -- returning NULL.

[muschellij2]

You can try read.gt3x. https://github.com/THLfi/read.gt3x

On Fri, Aug 28, 2020 at 1:18 AM Tom Stewart notifications@github.com wrote:

Just experienced this - good to know it's because of the ACTIVITY packets!

df <- read_gt3x(file, verbose = T)

Processing Laura (2020-08-16).gt3x

Unzipping ............. COMPLETE

Checking components ............. COMPLETE

Retrieving info from the zip archive ............. COMPLETE

Will parse the following packet types, if available:

METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY

EVENT, TAG, ACTIVITY, HEART_RATE_BPM

HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE

EPOCH, EPOCH2, EPOCH3, EPOCH4

ACTIVITY2, SENSOR_DATA

Reading log.bin ............. COMPLETE

Getting record headers ............... COMPLETE

Parsing PARAMETERS packet(s) ............. COMPLETE

Parsing EVENT packet(s) ............. COMPLETE

Parsing METADATA packet(s) ............. COMPLETE

Parsing BATTERY packet(s) ............. COMPLETE

Parsing ACTIVITY packet(s)

Parsing LUX packet(s)

Processing complete. Elapsed time 0.08 minutes.

Warning messages:

1: In parse_packet_set.default(X[[i]], ...) :

No method exists yet for parsing ACTIVITY packets -- returning NULL.

2: In parse_packet_set.default(X[[i]], ...) :

No method exists yet for parsing LUX packets -- returning NULL.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/paulhibbing/AGread/issues/19#issuecomment-682330296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIGPLXDO5ORRCXJSZTABSTSC44ZXANCNFSM4KUDXXUA .

-- Best, John

[paulhibbing]

Thanks for all this feedback. I've now had a first go at introducing an ACTIVITY packet parser, which you can invoke when you set parser = "dev".

I haven't done a lot of testing on it, but so far it's looked alright. I'd recommend comparing outputs against read_AG_raw for a few files before using this more widely.

My guess is read.gt3x is still faster. Not sure I'll solve that... But this fits right into the recent parser design update, so I do think it flows well.

Let me know if anything isn't working. The build is failing right now on Travis, but it looks like that's due to rJava configuration. Shouldn't be a problem if you install with dependencies = FALSE. (rJava isn't a strong dependency of AGread -- at this point I don't think the issue is related to paulhibbing/PAutilities#2 since RJava is relegated to suggests via matchingMarkets. But that's neither here nor there as far as this issue is concerned.)

[paulhibbing]

Travis is (apparently) configured in both repos to install all packages (including those in suggests). After clearing the PAutilities cache, that build fails with the same error AGread is having. That leads me to believe the current AGread failure is related to paulhibbing/PAutilities#2 after all.

Looks like there's already a request for the Travis folks to look into it.