Test app behavior with various data sets and usages

[xhdong-umd]

In the development I mostly tested the app with 3 - 4 data sets. It will be very helpful to test the app with different data sets and usages.

I plan to make a beta release this week. Any test from team members and users are welcome and appreciated.

Thanks!

[vestlink]

101684640 Ovis aries, Hol, Buskerud, Norway downloaded from Movebnak everything ok, except home range

sessionInfo("ctmm") R version 3.4.0 (2017-04-21) Platform: x86_64-apple-darwin15.6.0 (64-bit) Running under: macOS Sierra 10.12.4

Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: character(0)

other attached packages: [1] ctmm_0.4.0

loaded via a namespace (and not attached): [1] Rcpp_0.12.10 compiler_3.4.0 git2r_0.18.0
[4] plyr_1.8.4 methods_3.4.0 utils_3.4.0
[7] tools_3.4.0 grDevices_3.4.0 digest_0.6.12
[10] packrat_0.4.8-1 jsonlite_1.4 lubridate_1.6.0
[13] memoise_1.1.0 tibble_1.3.0 gtable_0.2.0
[16] lattice_0.20-35 shiny_1.0.3.9000 rgdal_1.2-7
[19] yaml_2.1.14 curl_2.6 gridExtra_2.2.1
[22] withr_1.0.2 httr_1.2.1 stringr_1.2.0
[25] raster_2.5-8 graphics_3.4.0 htmlwidgets_0.8
[28] datasets_3.4.0 stats_3.4.0 devtools_1.13.0
[31] grid_3.4.0 DT_0.2 shinydashboard_0.5.3 [34] fasttime_1.0-2 base_3.4.0 data.table_1.10.4
[37] R6_2.2.0 XML_3.98-1.7 pacman_0.4.5
[40] sp_1.2-4 reshape2_1.4.2 ggplot2_2.2.1
[43] magrittr_1.5 scales_0.4.1 htmltools_0.3.6
[46] mime_0.5 xtable_1.8-2 colorspace_1.3-2
[49] numDeriv_2016.8-1 httpuv_1.3.3 labeling_0.3
[52] stringi_1.1.5 lazyeval_0.2.0 munsell_0.4.3
[55] markdown_0.8

[vestlink]

70397061 Ovis aries, Nordøsterdalen, Norway downloaded from Movebank

Failed at import

Listening on http://127.0.0.1:4041 Maximum speed of 0.931742399929265 m/s observed in 71608843 Minimum sampling interval of 60 minutes in 71608843 Maximum speed of 0.674794205728709 m/s observed in 71608847 Minimum sampling interval of 60 minutes in 71608847 Maximum speed of 2.76815758956926 m/s observed in 71608850 Minimum sampling interval of 60 minutes in 71608850 Maximum speed of 2.76527530675793 m/s observed in 71608853 Minimum sampling interval of 60 minutes in 71608853 Maximum speed of 0.997728365682784 m/s observed in 71608856 Minimum sampling interval of 60 minutes in 71608856 Maximum speed of 3.11082402717296 m/s observed in 71608859 Minimum sampling interval of 60 minutes in 71608859 Maximum speed of 0.912273659962597 m/s observed in 71608862 Minimum sampling interval of 60 seconds in 71608862 Maximum speed of 2.78820429032318 m/s observed in 71608865 Minimum sampling interval of 60 minutes in 71608865 Maximum speed of 0.948370142282912 m/s observed in 71608868 Minimum sampling interval of 60 minutes in 71608868 Maximum speed of 0.946988220939844 m/s observed in 71608871 Minimum sampling interval of 60 minutes in 71608871 Maximum speed of 2.78775035257792 m/s observed in 71608874 Minimum sampling interval of 60 minutes in 71608874 Maximum speed of 2.76908792122087 m/s observed in 71608877 Minimum sampling interval of 60 minutes in 71608877 Maximum speed of 0.040155455208739 m/s observed in 71608881 Minimum sampling interval of 22 hours in 71608881 Maximum speed of 0.613787638482886 m/s observed in 71608884 Minimum sampling interval of 60 minutes in 71608884 Maximum speed of 0.57280220377731 m/s observed in 71608887 Minimum sampling interval of 60 minutes in 71608887 Maximum speed of 1.0503998234458 m/s observed in 71608890 Minimum sampling interval of 30 minutes in 71608890 Maximum speed of 1.62446291434064 m/s observed in 71608893 Minimum sampling interval of 30 minutes in 71608893 Maximum speed of 1.19775383354831 m/s observed in 71608896 Minimum sampling interval of 30 minutes in 71608896 Maximum speed of 2.97220782248906 m/s observed in 71608899 Minimum sampling interval of 30 minutes in 71608899 Maximum speed of 1.13781006015034 m/s observed in 71608902 Minimum sampling interval of 30 minutes in 71608902 Maximum speed of 0.97976256458531 m/s observed in 71608905 Minimum sampling interval of 30 minutes in 71608905 Maximum speed of 2.80869691037339 m/s observed in 71610139 Minimum sampling interval of 34 minutes in 71610139 Maximum speed of 0.99385062493792 m/s observed in 71610143 Minimum sampling interval of 60 minutes in 71610143 Maximum speed of 1.86438641847641 m/s observed in 71610146 Minimum sampling interval of 60 minutes in 71610146 Maximum speed of 2.91045222967905 m/s observed in 71610149 Minimum sampling interval of 60 minutes in 71610149 Maximum speed of 0.554262956811187 m/s observed in 71610152 Minimum sampling interval of 33 minutes in 71610152 Maximum speed of 0.648483921836115 m/s observed in 71610155 Minimum sampling interval of 5 minutes in 71610155 Maximum speed of 2.98279051779543 m/s observed in 71610158 Minimum sampling interval of 24 minutes in 71610158 Maximum speed of 1.57412407192828 m/s observed in 71610161 Minimum sampling interval of 60 minutes in 71610161 Maximum speed of 1.53710198361756 m/s observed in 71610164 Minimum sampling interval of 60 minutes in 71610164 Maximum speed of 2.819588125937 m/s observed in 71610167 Minimum sampling interval of 60 minutes in 71610167 Maximum speed of 0.751145367890969 m/s observed in 71610170 Minimum sampling interval of 60 minutes in 71610170 Maximum speed of 1.99046703571182 m/s observed in 71610173 Minimum sampling interval of 60 minutes in 71610173 Maximum speed of 1.17659848781512 m/s observed in 71610176 Minimum sampling interval of 60 minutes in 71610176 Maximum speed of 1.98951321360075 m/s observed in 71610179 Minimum sampling interval of 60 minutes in 71610179 Maximum speed of 2.89068307837532 m/s observed in 71610182 Minimum sampling interval of 60 minutes in 71610182 Maximum speed of 0.754840969321976 m/s observed in 71610185 Minimum sampling interval of 60 minutes in 71610185 Maximum speed of 4.72373312097498 m/s observed in 71610188 Minimum sampling interval of 30 minutes in 71610188 Maximum speed of 5.24885457694219 m/s observed in 71610191 Minimum sampling interval of 30 minutes in 71610191 Maximum speed of 4.35362118227563 m/s observed in 71610194 Minimum sampling interval of 30 minutes in 71610194 Maximum speed of 4.7050367650959 m/s observed in 71610197 Minimum sampling interval of 30 minutes in 71610197 Maximum speed of 7.07619588837674 m/s observed in 71610200 Minimum sampling interval of 60 seconds in 71610200 Maximum speed of 0.922837233606031 m/s observed in 71610203 Minimum sampling interval of 60 seconds in 71610203 Maximum speed of 2.84192669976294 m/s observed in 71610206 Minimum sampling interval of 30 minutes in 71610206 Maximum speed of 4.64612317549794 m/s observed in 71610209 Minimum sampling interval of 30 minutes in 71610209 Maximum speed of 6.92878898890962 m/s observed in 71610212 Minimum sampling interval of 30 minutes in 71610212 Maximum speed of 4.63331793180993 m/s observed in 71610215 Minimum sampling interval of 30 minutes in 71610215 Maximum speed of 7.18903941399057 m/s observed in 71610218 Minimum sampling interval of 30 minutes in 71610218 Maximum speed of 5.25885071336233 m/s observed in 71610221 Minimum sampling interval of 30 minutes in 71610221 Warning: Error in if: argument is not interpretable as logical Stack trace (innermost first): 75: speedMLE 74: ctmm:::assign_speeds 73: [.data.table 72: [ [helpers.R#159] 71: calculate_speed [helpers.R#159] 70: dt_tele_objs [helpers.R#209] 69: merge_animals [helpers.R#214] 68: data_import [/private/var/folders/qw/xjq4_wxj239b56vwdtj7gjhh0000gn/T/RtmpDunElS/shinyapp1531218a36f/ctmm-webapp-master/server.R#36] 67: observeEventHandler [/private/var/folders/qw/xjq4_wxj239b56vwdtj7gjhh0000gn/T/RtmpDunElS/shinyapp1531218a36f/ctmm-webapp-master/server.R#260] 3: runApp 2: runUrl 1: shiny::runGitHub ERROR: [on_request_read] connection reset by peer

sessionInfo("ctmm") R version 3.4.0 (2017-04-21) Platform: x86_64-apple-darwin15.6.0 (64-bit) Running under: macOS Sierra 10.12.4 Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: character(0)

other attached packages: [1] ctmm_0.4.0

loaded via a namespace (and not attached): [1] Rcpp_0.12.10 compiler_3.4.0 git2r_0.18.0 [4] plyr_1.8.4 methods_3.4.0 utils_3.4.0 [7] tools_3.4.0 grDevices_3.4.0 digest_0.6.12 [10] packrat_0.4.8-1 jsonlite_1.4 lubridate_1.6.0 [13] memoise_1.1.0 tibble_1.3.0 gtable_0.2.0 [16] lattice_0.20-35 shiny_1.0.3.9000 rgdal_1.2-7 [19] yaml_2.1.14 curl_2.6 gridExtra_2.2.1 [22] withr_1.0.2 httr_1.2.1 stringr_1.2.0 [25] raster_2.5-8 graphics_3.4.0 htmlwidgets_0.8 [28] datasets_3.4.0 stats_3.4.0 devtools_1.13.0 [31] grid_3.4.0 DT_0.2 shinydashboard_0.5.3 [34] fasttime_1.0-2 base_3.4.0 data.table_1.10.4 [37] R6_2.2.0 XML_3.98-1.7 pacman_0.4.5 [40] sp_1.2-4 reshape2_1.4.2 ggplot2_2.2.1 [43] magrittr_1.5 scales_0.4.1 htmltools_0.3.6 [46] mime_0.5 xtable_1.8-2 colorspace_1.3-2 [49] numDeriv_2016.8-1 httpuv_1.3.3 labeling_0.3 [52] stringi_1.1.5 lazyeval_0.2.0 munsell_0.4.3 [55] markdown_0.8

[xhdong-umd]

Currently only pages up to Visual Diagnostics are finished. The latter pages used to have very simple features in the previous version, but now they are disabled because the app structure has changed a lot.

For Home Range page I just removed all the boxes so it may look weird. I probably should add something like "page under construction".

[xhdong-umd]

Thanks for this test. It looks like the speed outlier calculation met some problem with the data. I cannot download this data so I cannot test it now.

Maybe I should change app so that it will still work with the data just with the speed outlier feature disabled.

[vestlink]

I could make the data available for you via dropbox

[xhdong-umd]

@vestlink , thanks! That will be very helpful.

I'll also try to make the app work even if it had errors with speed outlier calculation.

[vestlink]

https://www.dropbox.com/sh/v5q9kd2jb6xryk7/AADUKqM_4KLpdeP-4DFz-RiRa?dl=0 both the linux vm and the failing data

[vestlink]

Ubuntu 16.10 64 bit works like a charm...

[xhdong-umd]

Interesting, the error message posted above said it was having problem in speed outlier calculation, but I can't even import this csv file.

Error in `colnames<-`(`*tmp*`, value = c("x", "y")) : 
  attempt to set 'colnames' on an object with less than two dimensions
In addition: Warning message:
In as.telemetry.data.frame(data, timeformat = timeformat, timezone = timezone,  :
  VDOP values found but UERE not specified. See help("uere").

@vestlink Did you download the data through the movebank import feature, instead of uploading the csv file? The movebank import have less columns so maybe that's why it passed the import process.

[vestlink]

I used the import feature yes

On 10 May 2017 20:05, "xianghui dong" notifications@github.com wrote:

Interesting, the error message posted above said it was having problem in speed outlier calculation, but I can't even import this csv file.

Error in colnames<-(*tmp*, value = c("x", "y")) : attempt to set 'colnames' on an object with less than two dimensions In addition: Warning message: In as.telemetry.data.frame(data, timeformat = timeformat, timezone = timezone, : VDOP values found but UERE not specified. See help("uere").

@vestlink https://github.com/vestlink Did you download the data through the movebank import feature, instead of uploading the csv file? The movebank import have less columns so maybe that's why it passed the import process.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ctmm-initiative/ctmm-webapp/issues/23#issuecomment-300566073, or mute the thread https://github.com/notifications/unsubscribe-auth/AT0_HLXPTP3quzVaHGmg-zXQ2HC9niuhks5r4fxwgaJpZM4NWrC2 .

[xhdong-umd]

Can you save the imorted data as csv and share that csv? The are two problems here:

• import full csv into ctmm
• speed outlier calculation

With the imported csv I can look at the 2nd problem a little bit. For the first import problem I will ask @chfleming , probably some data format compatibility issues.

[vestlink]

i see issues regarding the naming of the csv when saved. See the shared dropbox for the requested csv

[xhdong-umd]

The original file name may cause some problem, but I modified the file name in my test already. It's more about column names or formats for importing.

When you trying to save the csv in app, is there any problem with the proposed file name? I used a filter to remove special characters in that proposed file name, but I'm not sure if it can handle this case properly.

[vestlink]

no problem with the file name, it is just wrong and do not reflect the movebank project that i've downloaded

[vestlink]

it should have been 70397061, Ovis aries, Nordøsterdalen, Norway

[xhdong-umd]

Got it, this is still the same bug with the data manager.

[xhdong-umd]

I updated the repo, now the save file for movebank import should give proper file name. The app also isolated the outlier with other pages. So it can work with this data as long as you don't use the outlier page. I'm still working on this part so there may be other problems.

[xhdong-umd]

@chfleming It seemed to be problem with ctmm:::time_res. My code here used ctmm:::time_res on data frame of each individual.

Error in if (dt) { : argument is not interpretable as logical 
5. speedMLE(data, dt = dt, UERE = UERE, CTMM = CTMM) 
4.ctmm:::assign_speeds(.SD, dt = ctmm:::time_res(.SD), UERE = 0, 
    method = "min") 

The previous naive leaving speed definition don't have problem, probably too simple to fail.

[chfleming]

I will look into it.

[vestlink]

Listening on http://127.0.0.1:4616 Warning: Error in : Please install xml2 package Stack trace (innermost first): 73: need_package 72: parser 71: parse_auto 70: httr::content 69: request [helpers.R#282] 68: get_all_studies [helpers.R#291] 67: observeEventHandler [C:\temp\RtmpAhtiK8\shinyappd182dbc43df\ctmm-webapp-master/server.R#125] 3: runApp 2: runUrl 1: shiny::runGitHub

sessionInfo("ctmm") R version 3.3.3 (2017-03-06) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

locale: [1] LC_COLLATE=Norwegian (Bokmål)_Norway.1252 LC_CTYPE=Norwegian (Bokmål)_Norway.1252
[3] LC_MONETARY=Norwegian (Bokmål)_Norway.1252 LC_NUMERIC=C
[5] LC_TIME=Norwegian (Bokmål)_Norway.1252

attached base packages: character(0)

other attached packages: [1] ctmm_0.4.0

loaded via a namespace (and not attached): [1] Rcpp_0.12.10 git2r_0.18.0 plyr_1.8.4 methods_3.3.3
[5] utils_3.3.3 tools_3.3.3 grDevices_3.3.3 digest_0.6.12
[9] packrat_0.4.8-1 jsonlite_1.4 lubridate_1.6.0 memoise_1.1.0
[13] tibble_1.3.0 gtable_0.2.0 lattice_0.20-34 shiny_1.0.3.9000
[17] yaml_2.1.14 curl_2.6 gridExtra_2.2.1 withr_1.0.2
[21] httr_1.2.1 stringr_1.2.0 raster_2.5-8 graphics_3.3.3
[25] htmlwidgets_0.8 datasets_3.3.3 stats_3.3.3 devtools_1.13.0
[29] grid_3.3.3 DT_0.2 shinydashboard_0.5.3 base_3.3.3
[33] data.table_1.10.4 R6_2.2.1 XML_3.98-1.7 pacman_0.4.5
[37] sp_1.2-4 ggplot2_2.2.1 magrittr_1.5 scales_0.4.1
[41] htmltools_0.3.6 mime_0.5 xtable_1.8-2 colorspace_1.3-2
[45] httpuv_1.3.3 stringi_1.1.5 lazyeval_0.2.0 munsell_0.4.3
[49] markdown_0.8

[xhdong-umd]

Got it. I need to add xml2 to package dependency list. My test in the isolated R libraries didn't test the movebank import so I didn't catch this. Thanks!

I'm still working on some substantial changes on code so probably will not push the changes to repo now.

[xhdong-umd]

I updated the repo with the xml2 fix, speed calculation now have fall back method, and it's isolated in outlier page, so new data should not crash the app.

[chfleming]

@xhdong-umd I have it fixed on GitHub now. There was no logic in time_res for individuals with only 1-2 locations.

I also had to fix as.telemetry to only take extra columns that are not filled with NA values.

[xhdong-umd]

I tested with the current github verion ctmm. The original csv can be imported without problem, but I still saw the speed calculation had error and fell back to alternative method. This is not a problem though, the app will still behave and we have the fall back method. We can wait for more data cases before revisiting this again.

source("shiny/app/helpers.R")
vest <- "/Users/xhdong/Projects/ctmm-Shiny/data/vestlink/Movebank 70397061 - A week in the life of a pygmy blue whale.csv"
data <- as.telemetry(vest)
merged <- merge_animals(data)
animals_dt <- merged$data
calculate_speed_ctmm(animals_dt)

Error in if (any(SUB)) { : missing value where TRUE/FALSE needed 
5.
speedMLE(data[c(1, 3), ], dt = dt, UERE = UERE, CTMM = CTMM) 
4.
ctmm:::assign_speeds(.SD, dt = ctmm:::time_res(.SD), UERE = 0, 
    method = "max") 
3.
`[.data.table`(animals_dt, , `:=`(speed, ctmm:::assign_speeds(.SD, 
    dt = ctmm:::time_res(.SD), UERE = 0, method = "max")), by = identity) at helpers.R#165
2.
animals_dt[, `:=`(speed, ctmm:::assign_speeds(.SD, dt = ctmm:::time_res(.SD), 
    UERE = 0, method = "max")), by = identity] at helpers.R#165
1.
calculate_speed_ctmm(animals_dt) 

[chfleming]

I tested time_res and forgot to test the other functions, which also needed similar logic to handle the case of two data points. Its fixed now.

[xhdong-umd]

Yes, it's almost impossible for us to predict all kinds of corner cases... Now we have the delete individuals button, user can also sort by point count then delete them if that's acceptable.