`arc_select()` error - `Error in x[[i]]: subscript out of bounds`

ar-puuk commented 4 days ago

Describe the bug arc_select() shows an error: Error in x[[i]]: subscript out of bounds

To Reproduce

library(arcgislayers)

furl <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Public_Schools/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: Public_Schools (esriGeometryPoint)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: Public_Schools
#> Geometry Type: esriGeometryPoint
#> CRS: 4326
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[i]]: subscript out of bounds
head(data_sf)
#> Error in eval(expr, envir, enclos): object 'data_sf' not found

^{Created on 2024-09-16 with reprex v2.1.1}

JosiahParry commented 4 days ago

Perhaps you can update the package?

install.packages("arcgislayers", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"))

ar-puuk commented 4 days ago

Thanks for your quick response. I updated the package, but unfortunately, the issue persists on my device. I checked with a couple of other feature service layers, neither of them worked.

install.packages("arcgislayers", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"))
#> Installing package into 'C:/Users/pukar/AppData/Local/R/win-library/4.4'
#> (as 'lib' is unspecified)
#> package 'arcgislayers' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\pukar\AppData\Local\Temp\RtmpILHgjU\downloaded_packages

library(arcgislayers)

furl <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Public_Schools/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: Public_Schools (esriGeometryPoint)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: Public_Schools
#> Geometry Type: esriGeometryPoint
#> CRS: 4326
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[i]]: subscript out of bounds
head(data_sf)
#> Error in eval(expr, envir, enclos): object 'data_sf' not found

^{Created on 2024-09-16 with reprex v2.1.1}

JosiahParry commented 4 days ago

Update the dependency {arcpbf} too install.packages("arcpbf", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"))

ar-puuk commented 4 days ago

Unfortunately, that did not work either:

install.packages("arcgislayers", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"))
#> Installing package into 'C:/Users/pukar/AppData/Local/R/win-library/4.4'
#> (as 'lib' is unspecified)
#> package 'arcgislayers' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\pukar\AppData\Local\Temp\RtmpiIDbmS\downloaded_packages

install.packages("arcpbf", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"))
#> Installing package into 'C:/Users/pukar/AppData/Local/R/win-library/4.4'
#> (as 'lib' is unspecified)
#> package 'arcpbf' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\pukar\AppData\Local\Temp\RtmpiIDbmS\downloaded_packages

library(arcgislayers)

furl <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Public_Schools/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: Public_Schools (esriGeometryPoint)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: Public_Schools
#> Geometry Type: esriGeometryPoint
#> CRS: 4326
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[i]]: subscript out of bounds
head(data_sf)
#> Error in eval(expr, envir, enclos): object 'data_sf' not found

^{Created on 2024-09-16 with reprex v2.1.1}

JosiahParry commented 4 days ago

Have you restarted your R session between installing new versions? You should update your dependencies too.

Please report your sessionInfo()

install.packages(
  "arcgislayers",
  repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"),
  dependencies = TRUE
)

ar-puuk commented 4 days ago

Yes, I have closed and restarted RStudio between each of the attempts.

install.packages("arcgislayers", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"), dependencies = TRUE)
#> Installing package into 'C:/Users/pukar/AppData/Local/R/win-library/4.4'
#> (as 'lib' is unspecified)
#> package 'arcgislayers' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\pukar\AppData\Local\Temp\RtmpQzN3kA\downloaded_packages

install.packages("arcpbf", repos = c("https://r-arcgis.r-universe.dev", "https://cloud.r-project.org"), dependencies = TRUE)
#> Installing package into 'C:/Users/pukar/AppData/Local/R/win-library/4.4'
#> (as 'lib' is unspecified)
#> package 'arcpbf' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\pukar\AppData\Local\Temp\RtmpQzN3kA\downloaded_packages

library(arcgislayers)

furl <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Public_Schools/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: Public_Schools (esriGeometryPoint)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: Public_Schools
#> Geometry Type: esriGeometryPoint
#> CRS: 4326
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[i]]: subscript out of bounds
head(data_sf)
#> Error in eval(expr, envir, enclos): object 'data_sf' not found

^{Created on 2024-09-16 with reprex v2.1.1}

and here is the session info:

> sessionInfo()
R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
  [1] DBI_1.2.3               httr2_1.0.4             tmaptools_3.1-1         tabulizerjars_1.0.1     rlang_1.1.4             magrittr_2.0.3         
  [7] lehdr_1.1.3             snakecase_0.11.1        arcgisutils_0.3.0       e1071_1.7-15            compiler_4.4.1          callr_3.7.6            
 [13] png_0.1-8               vctrs_0.6.5             rvest_1.0.4             stringr_1.5.1           pkgconfig_2.0.3         crayon_1.5.3           
 [19] fastmap_1.2.0           lwgeom_0.2-14           leafem_0.2.3            utf8_1.2.4              promises_1.3.0          rmarkdown_2.28         
 [25] tzdb_0.4.0              ps_1.8.0                spacesXYZ_1.3-0         purrr_1.0.2             xfun_0.47               satellite_1.0.5        
 [31] reprex_2.1.1            jsonlite_1.8.8          later_1.3.2             mapview_2.11.2          uuid_1.2-1              terra_1.7-78           
 [37] parallel_4.4.1          R6_2.5.1                stringi_1.8.4           RColorBrewer_1.1-3      lubridate_1.9.3         stars_0.6-6            
 [43] knitr_1.48              Rcpp_1.0.13             base64enc_0.1-3         pacman_0.5.1            readr_2.1.5             leaflet.providers_2.0.0
 [49] timechange_0.3.0        tidyselect_1.2.1        yaml_2.3.10             rstudioapi_0.16.0       dichromat_2.0-1         abind_1.4-8            
 [55] codetools_0.2-20        websocket_1.4.2         processx_3.8.4          lattice_0.22-6          tibble_3.2.1            leafsync_0.1.0         
 [61] withr_3.0.1             evaluate_0.24.0         tidycensus_1.6.5        tabulizer_0.2.3         sf_1.0-17               arcgislayers_0.3.0.9000
 [67] rJava_1.0-11            units_0.8-5             proxy_0.4-27            zip_2.3.1               xml2_1.3.6              pillar_1.9.0           
 [73] tigris_2.1              KernSmooth_2.23-24      stats4_4.4.1            generics_0.1.3          sp_2.1-4                chromote_0.3.1         
 [79] hms_1.1.3               munsell_0.5.1           scales_1.3.0            openxlsx2_1.9           class_7.3-22            glue_1.7.0             
 [85] janitor_2.2.0.9000      clipr_0.8.0             tools_4.4.1             leaflegend_1.2.1        data.table_1.16.0       openxlsx_4.2.7         
 [91] fs_1.6.4                XML_3.99-0.17           grid_4.4.1              tidyr_1.3.1             crosstalk_1.2.1         colorspace_2.1-1       
 [97] cols4all_0.7-1          raster_3.6-26           cli_3.6.3               rappdirs_0.3.3          fansi_1.0.6             tmap_3.99.9002         
[103] viridisLite_0.4.2       dplyr_1.1.4             digest_0.6.37           widgetframe_0.3.1       classInt_0.4-10         htmlwidgets_1.6.4      
[109] colorblindcheck_1.0.2   htmltools_0.5.8.1       lifecycle_1.0.4         leaflet_2.2.2           httr_1.4.7              microbenchmark_1.5.0

JosiahParry commented 4 days ago

You also should remove head() because the error is before this. Do not install the package in your reprex call. Also after the error please run rlang::last_trace().

After, try installing collapse and see if that fixes it? I'm curious if this is in rbind_results()

ar-puuk commented 4 days ago

Installing collapse didnt help either:

furl <- "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/arcgis/rest/services/Public_Schools/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 4326
#> Capabilities: Query,Extract
#>   0: Public_Schools (esriGeometryPoint)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: Public_Schools
#> Geometry Type: esriGeometryPoint
#> CRS: 4326
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[i]]: subscript out of bounds

rlang::last_trace()
#> Error: Can't show last error because no error was recorded yet

sessionInfo()
#> R version 4.4.1 (2024-06-14 ucrt)
#> Platform: x86_64-w64-mingw32/x64
#> Running under: Windows 11 x64 (build 22631)
#> 
#> Matrix products: default
#> 
#> 
#> locale:
#> [1] LC_COLLATE=English_United States.utf8 
#> [2] LC_CTYPE=English_United States.utf8   
#> [3] LC_MONETARY=English_United States.utf8
#> [4] LC_NUMERIC=C                          
#> [5] LC_TIME=English_United States.utf8    
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] cli_3.6.3               knitr_1.48              rlang_1.1.4            
#>  [4] xfun_0.47               KernSmooth_2.23-24      DBI_1.2.3              
#>  [7] sf_1.0-17               glue_1.7.0              htmltools_0.5.8.1      
#> [10] e1071_1.7-15            arcgisutils_0.3.0       RcppSimdJson_0.1.12    
#> [13] arcgislayers_0.3.0.9000 rmarkdown_2.28          rappdirs_0.3.3         
#> [16] grid_4.4.1              evaluate_0.24.0         classInt_0.4-10        
#> [19] fastmap_1.2.0           jsonify_1.2.2           yaml_2.3.10            
#> [22] lifecycle_1.0.4         httr2_1.0.4             compiler_4.4.1         
#> [25] fs_1.6.4                Rcpp_1.0.13             rstudioapi_0.16.0      
#> [28] digest_0.6.37           R6_2.5.1                class_7.3-22           
#> [31] reprex_2.1.1            curl_5.2.2              magrittr_2.0.3         
#> [34] proxy_0.4-27            tools_4.4.1             withr_3.0.1            
#> [37] arcpbf_0.1.5            units_0.8-5

^{Created on 2024-09-16 with reprex v2.1.1}

The session info within reprex seems different than what I get in my console so attaching that separately as well:

> sessionInfo()
R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] arcgislayers_0.3.0.9000

loaded via a namespace (and not attached):
  [1] DBI_1.2.3               httr2_1.0.4             tmaptools_3.1-1         tabulizerjars_1.0.1     rlang_1.1.4             magrittr_2.0.3         
  [7] lehdr_1.1.3             snakecase_0.11.1        arcgisutils_0.3.0       e1071_1.7-15            compiler_4.4.1          callr_3.7.6            
 [13] png_0.1-8               vctrs_0.6.5             rvest_1.0.4             stringr_1.5.1           pkgconfig_2.0.3         crayon_1.5.3           
 [19] fastmap_1.2.0           lwgeom_0.2-14           leafem_0.2.3            utf8_1.2.4              promises_1.3.0          rmarkdown_2.28         
 [25] tzdb_0.4.0              ps_1.8.0                spacesXYZ_1.3-0         purrr_1.0.2             xfun_0.47               reprex_2.1.1           
 [31] satellite_1.0.5         jsonlite_1.8.8          later_1.3.2             mapview_2.11.2          uuid_1.2-1              terra_1.7-78           
 [37] parallel_4.4.1          R6_2.5.1                stringi_1.8.4           RColorBrewer_1.1-3      lubridate_1.9.3         stars_0.6-6            
 [43] knitr_1.48              Rcpp_1.0.13             base64enc_0.1-3         pacman_0.5.1            readr_2.1.5             leaflet.providers_2.0.0
 [49] timechange_0.3.0        tidyselect_1.2.1        yaml_2.3.10             rstudioapi_0.16.0       dichromat_2.0-1         abind_1.4-8            
 [55] RcppSimdJson_0.1.12     codetools_0.2-20        websocket_1.4.2         curl_5.2.2              processx_3.8.4          lattice_0.22-6         
 [61] tibble_3.2.1            leafsync_0.1.0          withr_3.0.1             evaluate_0.24.0         tidycensus_1.6.5        tabulizer_0.2.3        
 [67] sf_1.0-17               rJava_1.0-11            units_0.8-5             proxy_0.4-27            zip_2.3.1               xml2_1.3.6             
 [73] pillar_1.9.0            tigris_2.1              KernSmooth_2.23-24      stats4_4.4.1            generics_0.1.3          sp_2.1-4               
 [79] chromote_0.3.1          jsonify_1.2.2           hms_1.1.3               munsell_0.5.1           scales_1.3.0            openxlsx2_1.9          
 [85] class_7.3-22            glue_1.7.0              janitor_2.2.0.9000      clipr_0.8.0             tools_4.4.1             leaflegend_1.2.1       
 [91] data.table_1.16.0       openxlsx_4.2.7          fs_1.6.4                XML_3.99-0.17           grid_4.4.1              tidyr_1.3.1            
 [97] crosstalk_1.2.1         colorspace_2.1-1        cols4all_0.7-1          raster_3.6-26           cli_3.6.3               rappdirs_0.3.3         
[103] fansi_1.0.6             tmap_3.99.9002          viridisLite_0.4.2       dplyr_1.1.4             digest_0.6.37           widgetframe_0.3.1      
[109] classInt_0.4-10         arcpbf_0.1.5            htmlwidgets_1.6.4       colorblindcheck_1.0.2   htmltools_0.5.8.1       lifecycle_1.0.4        
[115] leaflet_2.2.2           httr_1.4.7              microbenchmark_1.5.0

JosiahParry commented 4 days ago

I'm genuinely at a loss. Do you know how to step through the debugger?

I would ask that you run and find where the code crashes. I cannot reproduce this error and I don't have any idea where it might be coming from.

debugonce(arcgislayers:::collect_layer)
data_sf <- arcgislayers::arc_select(layer)

ar-puuk commented 4 days ago

I would have to teach myself. Let me get back once I figure out further details.

JosiahParry commented 4 days ago

I believe in you! All it does is open up the internals of the function and lets you run it line by line. It's a gateway into powerful debugging and package development!

JosiahParry commented 4 days ago

I just ran into this error but then was unable to reproduce it again... I wonder if you are getting rate limited by the service intermittently? Or if the protocol buffer is sometimes getting corrupted

ar-puuk commented 4 days ago

That is interesting. I have tried couple other feature servers that have worked in the past, and I get the same error.

furl <- "https://services1.arcgis.com/99lidPhWCzftIe9K/arcgis/rest/services/UtahZipCodeAreas/FeatureServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <FeatureServer <1 layer, 0 tables>>
#> CRS: 3857
#> Capabilities: Query,Extract
#>   0: ZipCodes (esriGeometryPolygon)

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: ZipCodes
#> Geometry Type: esriGeometryPolygon
#> CRS: 3857
#> Capabilities: Query,Extract

data_sf <- arcgislayers::arc_select(layer)
#> Error in x[[1]]: subscript out of bounds

rlang::last_trace()
#> Error: Can't show last error because no error was recorded yet

^{Created on 2024-09-16 with reprex v2.1.1}

Interestingly, one of the feature/map server I just tried gave a strange error. I don't know if both of these errors have to do anything with each other or not.

furl <- "https://rnhp.dot.ga.gov/hosting/rest/services/GDOT_Boundaries/MapServer"

conn <- arcgislayers::arc_open(furl)
conn
#> <MapServer <7 layers, 0 tables>>
#> CRS: 3857
#> Capabilities: Map,Query,Data
#>   0: State Boundary  ()
#>   1: Counties ()
#>   2: GDOT Maintenance Areas ()
#>   3: GDOT Districts ()
#>   4: State House Districts ()
#>   5: State Senate Districts ()
#>   6: Congressional Districts ()

layer <- arcgislayers::get_layer(conn, id = 0)
layer
#> <FeatureLayer>
#> Name: State Boundary 
#> Geometry Type: esriGeometryPolygon
#> CRS: 3857
#> Capabilities: Map,Query,Data

# debugonce(arcgislayers:::collect_layer)
data_sf <- arcgislayers::arc_select(layer)
#> Error in .deserialize_json(json = json, query = query, empty_array = empty_array, : TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc.

^{Created on 2024-09-16 with reprex v2.1.1}

JosiahParry commented 4 days ago

I really wonder if you're being rate limited. Try running:


resp <- httr2::request("https://rnhp.dot.ga.gov/hosting/rest/services/GDOT_Boundaries/MapServer/0/query") |> 
  httr2::req_body_form(where = "1=1", outFields = "*", f = "pjson") |> 
  httr2::req_perform()

# extract json
json <- httr2::resp_body_string(resp)

# parse the result
arcgisutils::parse_esri_json(json)

# if that didn't work run
json

ar-puuk commented 4 days ago

That one works:

but this one still doesn't:

JosiahParry commented 4 days ago

Ah! I updated from httr2 1.0.1 to httr2 1.0.4 and can reproduce this issue!

httr2 must have introduced a break change somewhere.

ar-puuk commented 4 days ago

Great. Is httr2 just responsible for the json issue or is it for the subscript issue as well? Looks like I will be debugging for a while.. lol

JosiahParry commented 4 days ago

I'll need to figure out how to make a repro for httr2 but it looks like that somewhere between the versions the req_body_form() function became broken.

Can you run remotes::install_version("httr2", "1.0.1") for the time being?

ar-puuk commented 4 days ago

That seems to solve the issue. Thanks a lot.

JosiahParry commented 4 days ago

The issue is still there! We just don't know how to fix it for now. I'll dig into it more.

JosiahParry commented 4 days ago

Thank you for your patience and help reporting this @ar-puuk i've reported a bug in httr2 https://github.com/r-lib/httr2/issues/549

ar-puuk commented 4 days ago

Thanks a lot for your support in troubleshooting and your help in identifying a temporary solution.

R-ArcGIS / arcgislayers

`arc_select()` error - `Error in x[[i]]: subscript out of bounds` #219