Unable to unpack Coordinates from PolyLine

[ChrisWoodsSays]

Hi there, thanks for a great package. This was previously working, but doesn't now, maybe due to the API update or changes to nest.

The code is largely from Daniel Padfield's helpful post here

Here is the reproducable code.

`

Load the rStrava Package

install.packages('devtools')
devtools::install_github('fawda123/rStrava', force=TRUE)

# Load the libraries
library(tidyverse)
library(rStrava)
library(ggmap)

app_name <- 'xxxx'
app_client_id  <- 'nnnnn' # an integer, assigned by Strava
app_secret <- 'xxxxxxxxxxxxxxxxxxxxx' # an alphanumeric secret, assigned by Strava

# create the Strava authentication token
stoken <- httr::config(token = strava_oauth(app_name, app_client_id, app_secret,         app_scope="activity:read_all"))
#stoken <- httr::config(token = readRDS('.httr-oauth')[[1]])

# retrieve the Google key, restart R if not found
mykey <- Sys.getenv("google_key")
# Register Google key into R Session
register_google(mykey)

# Get an activities list of the desired type (club, friends, user)
my_acts <- get_activity_list(stoken)

#  Convert list of activities into a dataframe
my_actsDF <- compile_activities(my_acts) 

# columns to keep
desired_columns <- c('id', 'name', 'gear_id',
                     'achievement_count', 'athlete_count', 'kudos_count', 
                     'average_speed', 'max_speed', 
                     'average_watts', 'kilojoules', 'average_temp',
                     'distance', 'elapsed_time', 'moving_time', 
                     'start_date', 'start_date_local', 
                     'map.summary_polyline', 'location_city', 
                     'start_latitude', 'start_longitude')

# keep only desired columns and this year
my_actsDF <- dplyr::select(my_actsDF, match(desired_columns, names(my_actsDF))) %>%
    filter(start_date >=  '2019-01-01')

# transformations
my_actsDF <- mutate(my_actsDF,
                  activity_no = seq(1,n(), 1),
                  elapsed_time = elapsed_time/60/60,
                  moving_time = moving_time/60/60, 
                  date = gsub("T.*$", '', start_date) %>%
                      as.POSIXct(., format = '%Y-%m-%d'),
                  EUdate = format(date, '%d/%m/%Y'),
                  month = format(date, "%m"),
                  day = format(date, "%d"),
                  year = format(date, "%Y")) %>%
    mutate_at(., c('month', 'day'), as.numeric)

# get lat lon and distance of every ride
lat_lon <- my_actsDF %>%
    filter(!is.na(map.summary_polyline)) %>%
    nest(., -activity_no) %>%
    mutate(coords = map(data, get_latlon),
           distance = map(coords, ~get_dists(.x$lon, .x$lat))) %>%
    unnest(., data) %>%
    unnest(., coords, distance)`

Everything is fine until the last statement, when getting lat_lon when I get the messages

Error in googleway::google_elevation(polyline = polyline, key = key) : please only specify a single polyline In addition: Warning message: All elements of ... must be named. Did you want data = c(id, name, gear_id, achievement_count, athlete_count, kudos_count, average_speed, max_speed, average_watts, kilojoules, average_temp, distance, elapsed_time, moving_time, start_date, start_date_local, map.summary_polyline, location_city, start_latitude, start_longitude, date, EUdate, month, day, year)?

I have tried adding in the 'data = c(id..' suggested code and also using the nest_legacy and unnest_legacy, but to no avail.

Please can you help?

[padpadpadpad]

Hey Chris

Where does this code break for you?

get_activity_list() is not working for me. Says page not found. This is even after adding activity:read_all to the OAuth.

[ChrisWoodsSays]

get_activity_list() is working for me. I am successfully downloading the data and I can see the poly lines for each ride, just struggling to unpack them.

I did do a force package download of rStrava as I was having various issues. I've never had page not found.

Is your Authorisation Call Back Domain set correctly on your Strava API settings page. Mine is set to localhost.

[padpadpadpad]

Hi Chris

It is now! Still not working though. :(

So I do think the nest() and unnest() have changed...

Could you upload a couple a very small subset of the dataset and I can try and work through it? Would do it on my own data but I can't seem to access the API right now.

[ChrisWoodsSays]

Hi Daniel. I've saved it here.

https://www.dropbox.com/s/79s36k6cn9vemba/my_actsDF_subset.RDS?dl=0

Its produced by the lines (and then subsetted to give just 2 rows)

`

# Get an activities list of the desired type (club, friends, user)
my_acts <- get_activity_list(stoken)

#  Convert list of activities into a dataframe
my_actsDF <- compile_activities(my_acts) `

Thanks,

Chris

[fawda123]

@ChrisWoodsSays I think I figured it out. It's not an issue with the Google API for unpacking the polylines. You just need to change the last bit of your code to this:

# get lat lon and distance of every ride
lat_lon <- my_actsDF %>%
    filter(!is.na(map.summary_polyline)) %>%
    group_by(activity_no) %>% 
    nest() %>%
    mutate(
        coords = map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)),
        distance = map(coords, ~get_dists(.x$lon, .x$lat))
        ) %>%
    select(activity_no, coords, distance) %>% 
    unnest(., c(coords, distance))

The get_latlon function takes the actual polyline object as input. As it was setup originally, you were passing an entire row of a data frame, of which only one column was the polyline. Make sense?

[ChrisWoodsSays]

@fawda123, I'm now getting

Error in UseMethod("mutate") : no applicable method for 'mutate' applied to an object of class "list"

[fawda123]

Can you post your output from sessionInfo()?

[ChrisWoodsSays]

R version 3.6.1 (2019-07-05) Platform: x86_64-apple-darwin15.6.0 (64-bit) Running under: macOS Mojave 10.14.6

Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale: [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] plotly_4.9.0 rgdal_1.4-6 sp_1.3-1 sf_0.8-0 ggthemes_4.2.0 skimr_1.0.7
[7] rStrava_1.1.0 networkD3_0.4 RColorBrewer_1.1-2 Hmisc_4.2-0 Formula_1.2-3 survival_2.44-1.1 [13] lattice_0.20-38 viridis_0.5.1 viridisLite_0.3.0 forcats_0.4.0 stringr_1.4.0 dplyr_0.8.3
[19] purrr_0.3.3 readr_1.3.1 tidyr_1.0.0 tibble_2.1.3 tidyverse_1.2.1 ggmap_3.0.0
[25] ggplot2_3.2.1 openxlsx_4.1.0.1 maps_3.3.0

loaded via a namespace (and not attached): [1] colorspace_1.4-1 rjson_0.2.20 class_7.3-15 ellipsis_0.3.0 rprojroot_1.3-2 htmlTable_1.13.2
[7] base64enc_0.1-3 fs_1.3.1 rstudioapi_0.10 roxygen2_6.1.1 remotes_2.1.0 ggrepel_0.8.1
[13] fansi_0.4.0 lubridate_1.7.4 xml2_1.2.2 splines_3.6.1 knitr_1.25 pkgload_1.0.2
[19] zeallot_0.1.0 jsonlite_1.6 packrat_0.5.0 broom_0.5.2 cluster_2.1.0 png_0.1-7
[25] shiny_1.4.0 compiler_3.6.1 httr_1.4.1 backports_1.1.5 assertthat_0.2.1 Matrix_1.2-17
[31] fastmap_1.0.1 lazyeval_0.2.2 cli_1.1.0 later_1.0.0 acepack_1.4.1 htmltools_0.4.0
[37] prettyunits_1.0.2 tools_3.6.1 igraph_1.2.4.1 gtable_0.3.0 glue_1.3.1 V8_2.3
[43] Rcpp_1.0.2 cellranger_1.1.0 vctrs_0.2.0 nlme_3.1-140 xfun_0.9 ps_1.3.0
[49] testthat_2.2.1 rvest_0.3.4 mime_0.7 lifecycle_0.1.0 devtools_2.2.1 XML_3.98-1.20
[55] scales_1.0.0 hms_0.5.1 promises_1.1.0 yaml_2.2.0 curl_4.2 geosphere_1.5-10
[61] memoise_1.1.0 gridExtra_2.3 rpart_4.1-15 latticeExtra_0.6-28 stringi_1.4.3 desc_1.2.0
[67] e1071_1.7-2 checkmate_1.9.4 pkgbuild_1.0.5 zip_2.0.4 RgoogleMaps_1.4.4 rlang_0.4.1
[73] pkgconfig_2.0.3 commonmark_1.7 bitops_1.0-6 htmlwidgets_1.5.1 labeling_0.3 processx_3.4.1
[79] tidyselect_0.2.5 plyr_1.8.4 magrittr_1.5 R6_2.4.0 generics_0.0.2 DBI_1.0.0
[85] googleway_2.7.1 pillar_1.4.2 haven_2.1.1 foreign_0.8-71 withr_2.1.2 units_0.6-4
[91] RCurl_1.95-4.12 nnet_7.3-12 modelr_0.1.5 crayon_1.3.4 KernSmooth_2.23-15 utf8_1.1.4
[97] jpeg_0.1-8 usethis_1.5.1 grid_3.6.1 readxl_1.3.1 data.table_1.12.2 callr_3.3.1
[103] classInt_0.4-1 digest_0.6.21 xtable_1.8-4 httpuv_1.5.2 munsell_0.5.0 sessioninfo_1.1.1

[fawda123]

Works fine on my end, does this not work (note the different date filter and I could not get the select working)?

library(tidyverse)
library(rStrava)

# create the authentication token
stoken <- httr::config(token = strava_oauth(app_name, app_client_id, app_secret, cache = T, app_scope="activity:read_all"))

# retrieve the key, restart R if not found
mykey <- Sys.getenv("google_key")

my_acts <- get_activity_list(stoken)

#  Convert list of activities into a dataframe
my_actsDF <- compile_activities(my_acts) 

# keep only desired columns and this year
my_actsDF <- my_actsDF %>%
    dplyr::filter(start_date >=  '2019-10-15')

# transformations
my_actsDF <- mutate(my_actsDF,
        activity_no = seq(1,n(), 1),
        elapsed_time = elapsed_time/60/60,
        moving_time = moving_time/60/60, 
        date = gsub("T.*$", '', start_date) %>%
            as.POSIXct(., format = '%Y-%m-%d'),
        EUdate = format(date, '%d/%m/%Y'),
        month = format(date, "%m"),
        day = format(date, "%d"),
        year = format(date, "%Y")
        ) %>%
    mutate_at(., c('month', 'day'), as.numeric)

# get lat lon and distance of every ride
lat_lon <- my_actsDF %>%
    filter(!is.na(map.summary_polyline)) %>%
    group_by(activity_no) %>% 
    nest() %>%
    mutate(
        coords = map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)),
        distance = map(coords, ~get_dists(.x$lon, .x$lat))
        ) %>%
    select(activity_no, coords, distance) %>% 
    unnest(., c(coords, distance))

[ChrisWoodsSays]

Am afraid not, still get same error. Re installed r-strava with force and restarted R Studio.

[fawda123]

Okay, can you share on dropbox the my_actsDF data object that gets assigned to lat_lon?

[ChrisWoodsSays]

I've added to dropbox here, just two rides, but I get the same error from that file.

https://www.dropbox.com/s/z87q2z9otgbzq6i/my_actsDF_subset_for_Marcus?dl=0

Its produced by

my_actsDF <- mutate(my_actsDF, activity_no = seq(1,n(), 1), elapsed_time = elapsed_time/60/60, moving_time = moving_time/60/60, date = gsub("T.*$", '', start_date) %>% as.POSIXct(., format = '%Y-%m-%d'), EUdate = format(date, '%d/%m/%Y'), month = format(date, "%m"), day = format(date, "%d"), year = format(date, "%Y")) %>% mutate_at(., c('month', 'day'), as.numeric)

Error is

Error in UseMethod("mutate") : no applicable method for 'mutate' applied to an object of class "list"

[ChrisWoodsSays]

I appreciate your help on this Marcus.

[fawda123]

Still works on my end... what does your data look like up to this line?

lat_lon <- my_actsDF %>%
    filter(!is.na(map.summary_polyline)) %>%
    group_by(activity_no) %>% 
    nest()
lat_lon
# # A tibble: 2 x 2
# # Groups:   activity_no [2]
# activity_no            data
#        <dbl> <list<df[,25]>>
# 1           1        [1 x 25]
# 2           3        [1 x 25]

[ChrisWoodsSays]

Looks very similar

# get lat lon and distance of every ride
lat_lon <- my_actsDF_subset %>%
+     filter(!is.na(map.summary_polyline)) %>%
+     group_by(activity_no) %>% 
+     nest() 
lat_lon
# A tibble: 2 x 2
# Groups:   activity_no [2]
  activity_no            data
        <dbl> <list<df[,25]>>
1           1        [1 × 25]
2           3        [1 × 25]

https://www.dropbox.com/s/7hnwyegfypbh7mg/lat_lon_for_Marcus_upto_nest?dl=0

[fawda123]

Then it returns an error when you run the next line with mutate?

[ChrisWoodsSays]

Yes, that’s it.

[fawda123]

Okay, we are getting close to a minimal working example.

This works fine for me.

library(dplyr)
library(purrr)
library(rStrava)

# retrieve the key, restart R if not found
mykey <- Sys.getenv("google_key")

my_actsDF <- readRDS('C:/Users/Marcus.SCCWRP2K/Desktop/lat_lon_for_Marcus_upto_nest.RData') 

my_actsDF <- my_actsDF %>% 
    mutate(
        coords = map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)),
        distance = map(coords, ~get_dists(.x$lon, .x$lat))
    )

And my sessionInfo(). Anything that doesn't match with yours? I know we are on different operating systems, but that shouldn't be the issue.

R version 3.6.0 (2019-04-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rStrava_1.1.0 purrr_0.3.3   dplyr_0.8.3  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        pillar_1.4.2      compiler_3.6.0    later_1.0.0       plyr_1.8.4        bitops_1.0-6     
 [7] tools_3.6.0       digest_0.6.22     packrat_0.5.0     lattice_0.20-38   jsonlite_1.6      tibble_2.1.3     
[13] pkgconfig_2.0.3   rlang_0.4.1       shiny_1.4.0       googleway_2.7.1   rstudioapi_0.10   curl_4.2         
[19] fastmap_1.0.1     httr_1.4.1        xml2_1.2.2        htmlwidgets_1.5.1 grid_3.6.0        tidyselect_0.2.5 
[25] glue_1.3.1        R6_2.4.0          XML_3.98-1.20     sp_1.3-1          magrittr_1.5      promises_1.1.0   
[31] htmltools_0.4.0   assertthat_0.2.1  mime_0.7          xtable_1.8-4      geosphere_1.5-10  httpuv_1.5.2     
[37] RCurl_1.95-4.12   crayon_1.3.4 

[ChrisWoodsSays]

I have these extra (just below Matrix products)

BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

[1] ggmap_3.0.0 rStrava_1.1.0 forcats_0.4.0 stringr_1.4.0
[5] readr_1.3.1 tidyr_1.0.0
[9] tibble_2.1.3 ggplot2_3.2.1 tidyverse_1.2.1

Then a load of extra packages loaded via a namespace

[ChrisWoodsSays]

I'm on R 3.6.1, you are on 3.6.0. I know nest changed recently and there is nest_legacy that supposedly is as per the old version, but that doesn't seem to help.

[ChrisWoodsSays]

Does the traceback help?

Error in UseMethod("mutate") : no applicable method for 'mutate' applied to an object of class "list" 24. mutate_(.data, .dots = compat_as_lazy_dots(...)) 23. mutate.default(., lat = location$lat, lon = location$lng) 22. dplyr::mutate(., lat = location$lat, lon = location$lng) 21. function_list[i] 20. freduce(value, _function_list) 19. _fseq(_lhs) 18. eval(quote(_fseq(_lhs)), env, env) 17. eval(quote(_fseq(_lhs)), env, env) 16. withVisible(eval(quote(_fseq(_lhs)), env, env)) 15. googleway::google_elevation(polyline = polyline, key = key) %>% .[["results"]] %>% dplyr::mutate(lat = location$lat, lon = location$lng) %>% dplyr::select(-location, -resolution) %>% dplyr::rename(ele = elevation) 14. get_latlon(x$map.summary_polyline, key = mykey) 13. .f(.x[[i]], ...) 12. purrr::map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)) 11. mutate_impl(.data, dots, caller_env()) 10. mutate.tbl_df(., coords = purrr::map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)), distance = map(coords, ~get_dists(.x$lon, .x$lat))) 9. mutate(., coords = purrr::map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)), distance = map(coords, ~get_dists(.x$lon, .x$lat))) 8. function_list[k] 7. withVisible(function_list[k]) 6. freduce(value, _function_list) 5. _fseq(_lhs) 4. eval(quote(_fseq(_lhs)), env, env) 3. eval(quote(_fseq(_lhs)), env, env) 2. withVisible(eval(quote(_fseq(_lhs)), env, env)) 1. my_actsDF_subset %>% filter(!is.na(map.summary_polyline)) %>% group_by(activity_no) %>% nest_legacy() %>% mutate(coords = purrr::map(data, function(x) get_latlon(x$map.summary_polyline, key = mykey)), distance = map(coords, ~get_dists(.x$lon, .x$lat)))

[fawda123]

Okay, I'm at a loss now - I tried the example on three machines (two windows, one linux) and three versions of R (3.6.0., 3.6.1, 3.6.4) and I have not been able to reproduce the problem. This doesn't seem to be an rStrava issue, I don't think. Can you try on another machine and see if you have the same issue?

[ChrisWoodsSays]

I wonder if it’s one of the extra packages. I’m inclined to scrub my r installation and install afresh. If that doesn’t work, I have an old Windows machine I can try. Thanks for your persistence, I’ll let you know. May be tomorrow though.

[ChrisWoodsSays]

I think I know what it is. get_latlon calls googleway::google_elevation which in turn calls googleway::decode to decode the polyline. I tried passing a singular polyline and received an empty list. On further examination Google is rejecting it. I’m travelling at the moment and cannot get into the google developer's console, but I expect that’s where the issues lies. It would also explain why it works for you too.

Get polyline and confirm its a straightforward character string

polyline<-my_actsDF$data[[1]]$map.summary_polyline

Use google_elevation as per rStrava’s get_latlon to unpack polyline

googleway::google_elevation(polyline = polyline, key = mykey)

$error_message [1] "This API project is not authorized to use this API."

$results list()

$status [1] "REQUEST_DENIED"

On 3 Nov 2019, at 03:29, Marcus W Beck notifications@github.com wrote:

Okay, I'm at a loss now - I tried the example on three machines (two windows, one linux) and three versions of R (3.6.0., 3.6.1, 3.6.4) and I have not been able to reproduce the problem. This doesn't seem to be an rStrava issue, I don't think. Can you try on another machine and see if you have the same issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fawda123/rStrava/issues/66?email_source=notifications&email_token=AKAYUTGITGKUMSAYP47IUE3QRZARVA5CNFSM4JFRUGPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5KC5Y#issuecomment-549101943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKAYUTAWN6QFFFXOTGVEEI3QRZARVANCNFSM4JFRUGPA.

[ChrisWoodsSays]

The Maps Elevation API was disabled. I've now enabled it and it works. I'm sorry II sent you on a wild goose chase. I've learnt quite a bit. Thanks for your help Marcus and Daniel, I really appreciate it.

[fawda123]

Dang, that's a tricky one! Glad you got it sorted out though and happy to help for what it's worth.

[fawda123]

In hindsight, I'm going to add an informative error message if the Google API is disabled - this was a beast to diagnose for something simple, so we need to make sure it's clear when users have this disabled.