search

openmovementproject / openmovement

Open Movement devices are miniature, embeddable, open source sensors developed at Newcastle University, UK. The source code for the firmware and software is available under a BSD 2-clause license, and the hardware (PCB designs, layouts and schematics), enclosure designs and documentation are available under a Creative Commons 3.0 BY Attribution License.
https://openmovement.dev
146 stars 76 forks source link

Over Sampling Rate per second #40

Closed muschellij2 closed 3 years ago

muschellij2 commented 3 years ago

I'm reading in the sample CWA file from UK Biobank from this page https://biobank.ctsu.ox.ac.uk/crystal/field.cgi?id=90001 (Resources tab). I wrapped https://github.com/digitalinteraction/openmovement/blob/master/Software/AX3/cwa-convert/c/main.c into the read.cwa package (https://github.com/muschellij2/read.cwa) and the pycwa wraps the accProcess.py code from https://github.com/activityMonitoring/biobankAccelerometerAnalysis/tree/master/accelerometer to compare the results. As you can see from the counts from the data indicate some seconds have > 100 (the sampling rate) records, but I'm wondering if some resampling is done in any way.

 library(readr)
library(GGIR)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(pycwa)
library(read.cwa)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following objects are masked from 'package:base':
#> 
#>     date, intersect, setdiff, union
destfile = file.path(tempdir(), "accsamp.zip")
file = file.path(tempdir(), "accsamp.cwa")
if (!file.exists(file)) {
  if (!file.exists(destfile)) {
    download.file("https://biobank.ctsu.ox.ac.uk/crystal/crystal/examples/accsamp.zip",
                  destfile = destfile, quiet = FALSE)
  }
  unz = unzip(destfile, exdir = tempdir(), overwrite = TRUE)
}

xyz = c("x", "y", "z")

round_to_seconds = function(df) {
  df %>% mutate(time = lubridate::floor_date(time, "1 second"))
}
get_ranges = function(x) {
  sapply(x[xyz], range)
}

############################################
# Reader from OpenMovement/om
############################################
om = read_cwa(file, verbose = FALSE)
om = om$data
colnames(om) = tolower(colnames(om))
om = om[, c("time", xyz)]
nrow(om)
#> [1] 62227320
get_ranges(om)
#>              x         y         z
#> [1,] -8.000000 -8.000000 -8.000000
#> [2,]  7.984375  7.984375  7.984375
om = round_to_seconds(om)

############################################
# Reader from https://github.com/activityMonitoring/biobankAccelerometerAnalysis
############################################
py = py_read_cwa(file, verbose = FALSE)
py = py[, c("time", xyz)]
nrow(py)
#> [1] 60477000
get_ranges(py)
#>           x      y      z
#> [1,] -8.000 -8.000 -8.000
#> [2,]  7.984  7.984  7.984
py = round_to_seconds(py)

head(om)
#> # A tibble: 6 x 4
#>   time                      x      y      z
#>   <dttm>                <dbl>  <dbl>  <dbl>
#> 1 2013-07-13 10:00:05 -0.672   0.125 -1.30 
#> 2 2013-07-13 10:00:05 -0.0312 -0.109  0.844
#> 3 2013-07-13 10:00:05 -0.0469 -0.125  0.891
#> 4 2013-07-13 10:00:05 -0.0312 -0.125  0.891
#> 5 2013-07-13 10:00:05 -0.0312 -0.125  0.891
#> 6 2013-07-13 10:00:05 -0.0312 -0.125  0.891
head(py)
#> # A tibble: 6 x 4
#>   time                     x      y      z
#>   <dttm>               <dbl>  <dbl>  <dbl>
#> 1 2013-07-13 09:00:05 -0.672  0.125 -1.30 
#> 2 2013-07-13 09:00:05 -0.031 -0.109  0.844
#> 3 2013-07-13 09:00:05 -0.047 -0.125  0.891
#> 4 2013-07-13 09:00:05 -0.031 -0.125  0.891
#> 5 2013-07-13 09:00:05 -0.031 -0.125  0.891
#> 6 2013-07-13 09:00:05 -0.031 -0.125  0.891

# need to add because of DST
py$time = py$time + as.period(1, "hour")

om %>% filter(time <= time[1] + as.period(30, "seconds")) %>% group_by(time) %>% count()
#> # A tibble: 31 x 2
#> # Groups:   time [31]
#>    time                    n
#>    <dttm>              <int>
#>  1 2013-07-13 10:00:05    23
#>  2 2013-07-13 10:00:06   101
#>  3 2013-07-13 10:00:07   103
#>  4 2013-07-13 10:00:08   103
#>  5 2013-07-13 10:00:09   104
#>  6 2013-07-13 10:00:10   103
#>  7 2013-07-13 10:00:11   104
#>  8 2013-07-13 10:00:12   103
#>  9 2013-07-13 10:00:13   104
#> 10 2013-07-13 10:00:14   104
#> # … with 21 more rows
py %>% filter(time <= time[1] + as.period(30, "seconds")) %>% group_by(time) %>% count()
#> # A tibble: 31 x 2
#> # Groups:   time [31]
#>    time                    n
#>    <dttm>              <int>
#>  1 2013-07-13 10:00:05    23
#>  2 2013-07-13 10:00:06   100
#>  3 2013-07-13 10:00:07   100
#>  4 2013-07-13 10:00:08   100
#>  5 2013-07-13 10:00:09   100
#>  6 2013-07-13 10:00:10   100
#>  7 2013-07-13 10:00:11   100
#>  8 2013-07-13 10:00:12   100
#>  9 2013-07-13 10:00:13   100
#> 10 2013-07-13 10:00:14   100
#> # … with 21 more rows

gg = GGIR::g.cwaread(file, end = Inf, progressBar = TRUE, desiredtz = "UTC")
#>   |                                                                              |                                                                      |   0%  |                                                                              |                                                                      |   1%  |                                                                              |=                                                                     |   1%  |                                                                              |=                                                                     |   2%  |                                                                              |==                                                                    |   2%  |                                                                              |==                                                                    |   3%  |                                                                              |==                                                                    |   4%  |                                                                              |===                                                                   |   4%  |                                                                              |===                                                                   |   5%  |                                                                              |====                                                                  |   5%  |                                                                              |====                                                                  |   6%  |                                                                              |=====                                                                 |   6%  |                                                                              |=====                                                                 |   7%  |                                                                              |=====                                                                 |   8%  |                                                                              |======                                                                |   8%  |                                                                              |======                                                                |   9%  |                                                                              |=======                                                               |   9%  |                                                                              |=======                                                               |  10%  |                                                                              |=======                                                               |  11%  |                                                                              |========                                                              |  11%  |                                                                              |========                                                              |  12%  |                                                                              |=========                                                             |  12%  |                                                                              |=========                                                             |  13%  |                                                                              |=========                                                             |  14%  |                                                                              |==========                                                            |  14%  |                                                                              |==========                                                            |  15%  |                                                                              |===========                                                           |  15%  |                                                                              |===========                                                           |  16%  |                                                                              |============                                                          |  16%  |                                                                              |============                                                          |  17%  |                                                                              |============                                                          |  18%  |                                                                              |=============                                                         |  18%  |                                                                              |=============                                                         |  19%  |                                                                              |==============                                                        |  19%  |                                                                              |==============                                                        |  20%  |                                                                              |==============                                                        |  21%  |                                                                              |===============                                                       |  21%  |                                                                              |===============                                                       |  22%  |                                                                              |================                                                      |  22%  |                                                                              |================                                                      |  23%  |                                                                              |================                                                      |  24%  |                                                                              |=================                                                     |  24%  |                                                                              |=================                                                     |  25%  |                                                                              |==================                                                    |  25%  |                                                                              |==================                                                    |  26%  |                                                                              |===================                                                   |  26%  |                                                                              |===================                                                   |  27%  |                                                                              |===================                                                   |  28%  |                                                                              |====================                                                  |  28%  |                                                                              |====================                                                  |  29%  |                                                                              |=====================                                                 |  29%  |                                                                              |=====================                                                 |  30%  |                                                                              |=====================                                                 |  31%  |                                                                              |======================                                                |  31%  |                                                                              |======================                                                |  32%  |                                                                              |=======================                                               |  32%  |                                                                              |=======================                                               |  33%  |                                                                              |=======================                                               |  34%  |                                                                              |========================                                              |  34%  |                                                                              |========================                                              |  35%  |                                                                              |=========================                                             |  35%  |                                                                              |=========================                                             |  36%  |                                                                              |==========================                                            |  36%  |                                                                              |==========================                                            |  37%  |                                                                              |==========================                                            |  38%  |                                                                              |===========================                                           |  38%  |                                                                              |===========================                                           |  39%  |                                                                              |============================                                          |  39%  |                                                                              |============================                                          |  40%  |                                                                              |============================                                          |  41%  |                                                                              |=============================                                         |  41%  |                                                                              |=============================                                         |  42%  |                                                                              |==============================                                        |  42%  |                                                                              |==============================                                        |  43%  |                                                                              |==============================                                        |  44%  |                                                                              |===============================                                       |  44%  |                                                                              |===============================                                       |  45%  |                                                                              |================================                                      |  45%  |                                                                              |================================                                      |  46%  |                                                                              |=================================                                     |  46%  |                                                                              |=================================                                     |  47%  |                                                                              |=================================                                     |  48%  |                                                                              |==================================                                    |  48%  |                                                                              |==================================                                    |  49%  |                                                                              |===================================                                   |  49%  |                                                                              |===================================                                   |  50%  |                                                                              |===================================                                   |  51%  |                                                                              |====================================                                  |  51%  |                                                                              |====================================                                  |  52%  |                                                                              |=====================================                                 |  52%  |                                                                              |=====================================                                 |  53%  |                                                                              |=====================================                                 |  54%  |                                                                              |======================================                                |  54%  |                                                                              |======================================                                |  55%  |                                                                              |=======================================                               |  55%  |                                                                              |=======================================                               |  56%  |                                                                              |========================================                              |  56%  |                                                                              |========================================                              |  57%  |                                                                              |========================================                              |  58%  |                                                                              |=========================================                             |  58%  |                                                                              |=========================================                             |  59%  |                                                                              |==========================================                            |  59%  |                                                                              |==========================================                            |  60%  |                                                                              |==========================================                            |  61%  |                                                                              |===========================================                           |  61%  |                                                                              |===========================================                           |  62%  |                                                                              |============================================                          |  62%  |                                                                              |============================================                          |  63%  |                                                                              |============================================                          |  64%  |                                                                              |=============================================                         |  64%  |                                                                              |=============================================                         |  65%  |                                                                              |==============================================                        |  65%  |                                                                              |==============================================                        |  66%  |                                                                              |===============================================                       |  66%  |                                                                              |===============================================                       |  67%  |                                                                              |===============================================                       |  68%  |                                                                              |================================================                      |  68%  |                                                                              |================================================                      |  69%  |                                                                              |=================================================                     |  69%  |                                                                              |=================================================                     |  70%  |                                                                              |=================================================                     |  71%  |                                                                              |==================================================                    |  71%  |                                                                              |==================================================                    |  72%  |                                                                              |===================================================                   |  72%  |                                                                              |===================================================                   |  73%  |                                                                              |===================================================                   |  74%  |                                                                              |====================================================                  |  74%  |                                                                              |====================================================                  |  75%  |                                                                              |=====================================================                 |  75%  |                                                                              |=====================================================                 |  76%  |                                                                              |======================================================                |  76%  |                                                                              |======================================================                |  77%  |                                                                              |======================================================                |  78%
gg = gg$data
gg$time = lubridate::as_datetime(gg$time)
gg %>% filter(time <= time[1] + as.period(30, "seconds")) %>%
  round_to_seconds() %>% group_by(time) %>% count()
#> # A tibble: 31 x 2
#> # Groups:   time [31]
#>    time                    n
#>    <dttm>              <int>
#>  1 2013-07-13 10:00:05    23
#>  2 2013-07-13 10:00:06   100
#>  3 2013-07-13 10:00:07   100
#>  4 2013-07-13 10:00:08   100
#>  5 2013-07-13 10:00:09   100
#>  6 2013-07-13 10:00:10   100
#>  7 2013-07-13 10:00:11   100
#>  8 2013-07-13 10:00:12   100
#>  9 2013-07-13 10:00:13   100
#> 10 2013-07-13 10:00:14   100
#> # … with 21 more rows

Created on 2020-11-02 by the reprex package (v0.3.0.9001)

Session info sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.0.2 (2020-06-22) #> os macOS Catalina 10.15.6 #> system x86_64, darwin17.0 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz America/New_York #> date 2020-11-02 #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date lib source #> assertthat 0.2.1 2019-03-21 [2] CRAN (R 4.0.0) #> backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2) #> cli 2.1.0 2020-10-12 [1] CRAN (R 4.0.2) #> colorout * 1.2-2 2020-06-01 [2] Github (jalvesaq/colorout@726d681) #> crayon 1.3.4 2017-09-16 [2] CRAN (R 4.0.0) #> data.table 1.13.2 2020-10-19 [1] CRAN (R 4.0.2) #> digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.2) #> dplyr * 1.0.2 2020-08-18 [2] CRAN (R 4.0.2) #> ellipsis 0.3.1 2020-05-15 [2] CRAN (R 4.0.0) #> evaluate 0.14 2019-05-28 [2] CRAN (R 4.0.0) #> fansi 0.4.1 2020-01-08 [2] CRAN (R 4.0.0) #> fs 1.5.0 2020-07-31 [2] CRAN (R 4.0.2) #> generics 0.0.2 2018-11-29 [2] CRAN (R 4.0.0) #> GGIR * 2.1-3 2020-10-22 [1] Github (wadpac/GGIR@49aedcd) #> glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2) #> highr 0.8 2019-03-20 [2] CRAN (R 4.0.0) #> hms 0.5.3 2020-01-08 [2] CRAN (R 4.0.0) #> htmltools 0.5.0 2020-06-16 [2] CRAN (R 4.0.0) #> jsonlite 1.7.1 2020-09-07 [1] CRAN (R 4.0.2) #> knitr 1.30 2020-09-22 [1] CRAN (R 4.0.2) #> lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.2) #> lifecycle 0.2.0 2020-03-06 [2] CRAN (R 4.0.0) #> lubridate * 1.7.9 2020-06-08 [2] CRAN (R 4.0.0) #> magrittr 1.5 2014-11-22 [2] CRAN (R 4.0.0) #> Matrix 1.2-18 2019-11-27 [2] CRAN (R 4.0.2) #> pillar 1.4.6 2020-07-10 [2] CRAN (R 4.0.2) #> pkgconfig 2.0.3 2019-09-22 [2] CRAN (R 4.0.0) #> purrr 0.3.4 2020-04-17 [2] CRAN (R 4.0.0) #> pycwa * 0.1.0 2020-10-30 [1] local #> R.methodsS3 1.8.1 2020-08-26 [1] CRAN (R 4.0.2) #> R.oo 1.24.0 2020-08-26 [1] CRAN (R 4.0.2) #> R.utils 2.10.1 2020-08-26 [1] CRAN (R 4.0.2) #> R6 2.4.1 2019-11-12 [2] CRAN (R 4.0.0) #> Rcpp 1.0.5 2020-07-06 [2] CRAN (R 4.0.0) #> read.cwa * 0.2.1 2020-10-26 [1] local #> readr * 1.4.0 2020-10-05 [1] CRAN (R 4.0.2) #> reprex 0.3.0.9001 2020-09-30 [1] Github (tidyverse/reprex@d3fc4b8) #> reticulate 1.18 2020-10-25 [1] CRAN (R 4.0.2) #> rlang 0.4.8.9000 2020-10-22 [1] Github (r-lib/rlang@7a36238) #> rmarkdown 2.4 2020-09-30 [1] CRAN (R 4.0.2) #> rstudioapi 0.11 2020-02-07 [2] CRAN (R 4.0.0) #> sessioninfo 1.1.1 2018-11-05 [2] CRAN (R 4.0.0) #> stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2) #> stringr 1.4.0 2019-02-10 [2] CRAN (R 4.0.0) #> styler 1.3.2 2020-02-23 [2] CRAN (R 4.0.0) #> tibble 3.0.4 2020-10-12 [1] CRAN (R 4.0.2) #> tidyselect 1.1.0 2020-05-11 [2] CRAN (R 4.0.0) #> utf8 1.1.4 2018-05-24 [2] CRAN (R 4.0.0) #> vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2) #> withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2) #> xfun 0.18 2020-09-29 [1] CRAN (R 4.0.2) #> yaml 2.2.1 2020-02-01 [2] CRAN (R 4.0.0) #> #> [1] /Users/johnmuschelli/Library/R/4.0/library #> [2] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

Just the r code (without output):

library(readr)
library(GGIR)
library(dplyr)
library(pycwa)
library(read.cwa)
library(lubridate)
destfile = file.path(tempdir(), "accsamp.zip")
file = file.path(tempdir(), "accsamp.cwa")
if (!file.exists(file)) {
  if (!file.exists(destfile)) {
    download.file("https://biobank.ctsu.ox.ac.uk/crystal/crystal/examples/accsamp.zip",
                  destfile = destfile, quiet = FALSE)
  }
  unz = unzip(destfile, exdir = tempdir(), overwrite = TRUE)
}

xyz = c("x", "y", "z")

round_to_seconds = function(df) {
  df %>% mutate(time = lubridate::floor_date(time, "1 second"))
}
get_ranges = function(x) {
  sapply(x[xyz], range)
}

############################################
# Reader from OpenMovement/om
############################################
om = read_cwa(file, verbose = FALSE)
om = om$data
colnames(om) = tolower(colnames(om))
om = om[, c("time", xyz)]
nrow(om)
get_ranges(om)
om = round_to_seconds(om)

############################################
# Reader from https://github.com/activityMonitoring/biobankAccelerometerAnalysis
############################################
py = py_read_cwa(file, verbose = FALSE)
py = py[, c("time", xyz)]
nrow(py)
get_ranges(py)
py = round_to_seconds(py)

head(om)
head(py)

# need to add because of DST
py$time = py$time + as.period(1, "hour")

om %>% filter(time <= time[1] + as.period(30, "seconds")) %>% group_by(time) %>% count()
py %>% filter(time <= time[1] + as.period(30, "seconds")) %>% group_by(time) %>% count()

gg = GGIR::g.cwaread(file, end = Inf, progressBar = TRUE, desiredtz = "UTC")
gg = gg$data
gg$time = lubridate::as_datetime(gg$time)
gg %>% filter(time <= time[1] + as.period(30, "seconds")) %>%
  round_to_seconds() %>% group_by(time) %>% count()
danielgjackson commented 3 years ago

Hi,

Unlike other accelerometer loggers, the explicit aim of the AX devices was to record the direct, raw output of the underlying sensor. This avoids any blocking/aliasing which might occur if the resampling was done on the device itself, while also allowing the user the choice of whether to process the data later if they would like to simulate device-based resampling at an exact rate. To enable this, while the underlying sensor's internal clock is slightly variable, samples periodically receive a time stamp from a more precise real-time clock.

The means that, while you might see a slightly variable number of raw samples per given epoch, you can choose to use the true sample values over the epoch, or choose to simulate a fixed rate by resampling with any number of approaches. Because this was not done on the device itself before recording, you are free to change your mind later.