osmdata_sf() returning a CURL error

[ttuff]

This code worked until I updated to R 4.2 and now it throws a new error. Regardless of the key/value pair that I choose or the location I select, every query that includes osmdata_sf() returns a curl error.

library(osmdata)
bb <- getbb("Lawrence, Kansas")
my_boundary <- opq(bb) %>% 
  add_osm_feature(key = "building", value = "university") %>% 
osmdata_sf()

WITHOUT osmdata_sf() it returns: $bbox [1] "38.9044669,-95.3445398,39.0335024,-95.166617"

$prefix [1] "[out:xml][timeout:25];\n(\n"

$suffix [1] ");\n(._;>;);\nout body;"

$features [1] " [\"building\"=\"university\"]"

attr(,"class") [1] "list" "overpass_query" attr(,"nodes_only") [1] FALSE

WITH osmdata_sf() it returns: Error: ! HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1) Backtrace:

1. ... %>% osmdata_sf()
2. osmdata::osmdata_sf(.)
3. osmdata:::fill_overpass_data(obj, doc, quiet = quiet)
4. osmdata:::overpass_query(...)
5. osmdata::overpass_status(quiet)
6. httr2::req_perform(req)
7. base::tryCatch(...)
8. base (local) tryCatchList(expr, classes, parentenv, handlers)
9. base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
   10. value[3L]

Info about my session:

packageVersion("osmdata") [1] ‘0.1.10.4’ R.Version()$version.string [1] "R version 4.2.2 (2022-10-31)" sessionInfo() R version 4.2.2 (2022-10-31) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Ventura 13.0.1

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

[AdriMichelson]

I am having the same issue here as well when I backtrace to the error:

 rlang::last_error()
<error/httr2_failed>
Error:
! HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
---
Backtrace:
  1. ... %>% osmdata_sf()
  2. osmdata::osmdata_sf(.)
  3. osmdata:::fill_overpass_data(obj, doc, quiet = quiet)
  4. osmdata:::overpass_query(...)
  5. osmdata::overpass_status(quiet)
  6. httr2::req_perform(req)
  7. base::tryCatch(...)
  8. base (local) tryCatchList(expr, classes, parentenv, handlers)
  9. base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 10. value[[3L]](cond)
Run `rlang::last_trace()` to see the full context.
> rlang::last_trace()
<error/httr2_failed>
Error:
! HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
---
Backtrace:
     ▆
  1. ├─... %>% osmdata_sf()
  2. └─osmdata::osmdata_sf(.)
  3.   └─osmdata:::fill_overpass_data(obj, doc, quiet = quiet)
  4.     └─osmdata:::overpass_query(...)
  5.       └─osmdata::overpass_status(quiet)
  6.         └─httr2::req_perform(req)
  7.           └─base::tryCatch(...)
  8.             └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  9.               └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 10.                 └─value[[3L]](cond)

[mpadge]

The overpass server that delivers the data is somehow malfunctioning. This is the direct cause:

u <- "https://overpass.kumi.systems/api/status"
httr2::request (u) |>
    httr2::req_method ("GET") |>
    httr2::req_perform (req)
#> Error:
#> ! HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

#> Backtrace:
#>     ▆
#>  1. └─httr2::req_perform(req)
#>  2.   └─base::tryCatch(...)
#>  3.     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#>  4.       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>  5.         └─value[[3L]](cond)

^{Created on 2022-11-15 with reprex v2.0.2}

The server status nevertheless appears normal - but it is failing to deliver external requests. For the moment, we just have to hope that the necessary fix magically appears, but I'll definitely keep this open until it does. Thanks

[ttuff]

Could it be this: https://blog.balasundar.com/curl-92-http2-stream-0-was-not-closed-cleanly-protocolerror-err-1 ?

[Robsteranium]

Indeed I can recreate this with curl. The request works if we force HTTP 1.1:

$ curl https://overpass.kumi.systems/api/status 
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

$ curl --http1.1 https://overpass.kumi.systems/api/status
Connected as: 177471748
Current time: 2022-11-15T20:23:24Z
Announced endpoint: none
Rate limit: 0
Currently running queries (pid, space limit, time limit, start time):
1170965 536870912       180     2022-11-15T20:23:23Z
1170939 536870912       600     2022-11-15T20:23:21Z
1170967 536870912       180     2022-11-15T20:23:23Z
1170968 536870912       180     2022-11-15T20:23:23Z
1170969 536870912       180     2022-11-15T20:23:23Z
1170970 536870912       180     2022-11-15T20:23:23Z
1170971 536870912       180     2022-11-15T20:23:23Z

$ curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

We can get this working like:

url <- "https://overpass.kumi.systems/api/status"
resp <- httr2::request(url) |>
  httr2::req_method("GET") |>
  httr2::req_options(ssl_enable_alpn=F) |>
  httr2::req_perform()

[Robsteranium]

You can work around this by using httr2's mocking facilities.

The mock disables ALPN before continuing with the request:

http1_mock <- function(req) {
  req |>
    httr2::req_options(ssl_enable_alpn=F) |>
    httr2::req_perform(mock=NULL)
}

You can then wrap an individual request:

httr2::with_mock(http1_mock, status <- overpass_status())

Or mutate the local environment state (this will affect all httr2 requests of course):

httr2::local_mock(http1_mock)
status <- overpass_status()

[mpadge]

Thanks @Robsteranium , that indeed solves the problem, but ... that problem should not exist anyway, as the kumi overpass API should accept HTTP/2 requests, and did so successfully until yesterday, even with latest version of curl which defaults to HTTP/2. I'll implement what i hope will be only a temporary fix here. Temporary because it shouldn't be necessary - I've been succesfully using the latest curl with default HTTP/2 up until yesterday. I've contacted kumi, and hope that they'll be able to help on their side, so we can revert the changes here.

I'll notify here in a moment as soon as the change has been implemented.

[mpadge]

It seems that the issue has resolved itself in the meantime. I'll wait to see whether anybody else still has this problem, but everything is now fine for me, with curl v7.86 and HTTP/2:

u <- "https://overpass.kumi.systems/api/status"
status <- httr2::request (u) |>
    httr2::req_options (ssl_enable_alpn=T) |>
    httr2::req_perform ()
status <- httr2::resp_body_string (status)
strsplit (status, "\n") [[1]] [1:4]
#> [1] "Connected as: 177471748"            "Current time: 2022-11-16T10:21:18Z"
#> [3] "Announced endpoint: none"           "Rate limit: 0"

status <- httr2::request (u) |>
    httr2::req_options (ssl_enable_alpn=F) |>
    httr2::req_perform ()
status <- httr2::resp_body_string (status)
strsplit (status, "\n") [[1]] [1:4]
#> [1] "Connected as: 177471748"            "Current time: 2022-11-16T10:21:25Z"
#> [3] "Announced endpoint: none"           "Rate limit: 0"

^{Created on 2022-11-16 with reprex v2.0.2}

If nobody reports any further issues, I'll close after a day or two. Thanks :+1:

[Robsteranium]

I also contacted Kumi this morning, they just replied to say "we were able to reproduce and identify the problem, and it seems to be fixed now". I can confirm this is working for me now too.

I had this problem on Monday and it went away after I installed {osmdata} from GitHub instead of CRAN. This seems to have been coincidental as the problem returned (the next time I tried) yesterday. I wouldn't rule out some sort of non-deterministic behaviour here (e.g. one server behind the load balancer with a different configuration).

That said, we can always reopen the issue if it comes back...

[abdulrr]

I think I have faced a similar issue

my code: region = "Helsinki, Finland" opq(bbox = region)

I am not sure what's the issue.

Error in `httr2::req_perform()`:
! HTTP 405 Method Not Allowed.

Backtrace:
    ▆
 1. └─osmdata::opq(bbox = region)
 2.   └─osmdata::bbox_to_string(bbox)
 3.     └─osmdata::getbb(bbox)
 4.       └─osmdata:::get_nominatim_query(...)
 5.         └─httr2::req_perform(req)
 6.           └─global mock(req)
 7.             └─httr2::req_perform(httr2::req_options(req, ssl_enable_alpn = F), mock = NULL)

Sessioninfo()

R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] osmdata_0.2.3     tidytransit_1.6.0 forcats_1.0.0     stringr_1.5.0     dplyr_1.1.2       purrr_1.0.1       readr_2.1.4       tidyr_1.3.0      
 [9] tibble_3.2.1      ggplot2_3.4.2     tidyverse_2.0.0   lubridate_1.9.2   sf_1.0-14        

loaded via a namespace (and not attached):
 [1] httr2_0.2.3        tidyselect_1.2.0   geodist_0.0.8      colorspace_2.1-0   vctrs_0.6.3        generics_0.1.3     yaml_2.3.7        
 [8] utf8_1.2.3         rlang_1.1.1        e1071_1.7-13       pillar_1.9.0       glue_1.6.2         withr_2.5.0        DBI_1.1.3         
[15] rappdirs_0.3.3     lifecycle_1.0.3    gtfsio_1.1.0       munsell_0.5.0      gtable_0.3.3       tzdb_0.4.0         curl_5.0.1        
[22] class_7.3-20       fansi_1.0.4        Rcpp_1.0.11        KernSmooth_2.23-20 renv_0.12.3        scales_1.2.1       classInt_0.4-9    
[29] osmextract_0.4.1   config_0.3.1       hms_1.1.3          digest_0.6.33      stringi_1.7.12     grid_4.2.2         rprojroot_2.0.3   
[36] here_1.0.1         cli_3.6.1          tools_4.2.2        magrittr_2.0.3     proxy_0.4-27       pkgconfig_2.0.3    data.table_1.14.8 
[43] timechange_0.2.0   rstudioapi_0.15.0  R6_2.5.1           units_0.8-2        compiler_4.2.2

[mpadge]

Thanks @abdulrr , that's already been fixed via #328. Just install latest dev version and you should be good.

[ruimgbarros]

hi. I've just ran a query using the 0.2.5.1 version and I'm still getting this error.

Sessioninfo()

R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS 13.4.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] leaflet_2.1.2     ggmap_3.0.2       sf_1.0-10        
 [4] osmdata_0.2.5.001 lubridate_1.9.2   forcats_1.0.0    
 [7] stringr_1.5.0     dplyr_1.1.2       purrr_1.0.1      
[10] readr_2.1.4       tidyr_1.3.0       tibble_3.2.1     
[13] ggplot2_3.4.2     tidyverse_2.0.0  

loaded via a namespace (and not attached):
 [1] tidyselect_1.2.0    httr2_0.2.3         remotes_2.4.2      
 [4] lattice_0.21-8      colorspace_2.1-0    vctrs_0.6.3        
 [7] generics_0.1.3      htmltools_0.5.5     pkgbuild_1.4.2     
[10] utf8_1.2.3          rlang_1.1.1         e1071_1.7-13       
[13] pillar_1.9.0        glue_1.6.2          withr_2.5.0        
[16] DBI_1.1.3           rappdirs_0.3.3      sp_2.0-0           
[19] jpeg_0.1-10         lifecycle_1.0.3     plyr_1.8.8         
[22] munsell_0.5.0       gtable_0.3.3        RgoogleMaps_1.4.5.3
[25] htmlwidgets_1.6.2   callr_3.7.3         tzdb_0.4.0         
[28] fastmap_1.1.1       ps_1.7.5            crosstalk_1.2.0    
[31] curl_5.0.2          class_7.3-22        fansi_1.0.4        
[34] Rcpp_1.0.11         KernSmooth_2.23-20  scales_1.2.1       
[37] classInt_0.4-9      desc_1.4.2          hms_1.1.3          
[40] png_0.1-8           digest_0.6.33       stringi_1.7.12     
[43] processx_3.8.2      rprojroot_2.0.3     grid_4.1.0         
[46] cli_3.6.1           tools_4.1.0         bitops_1.0-7       
[49] magrittr_2.0.3      proxy_0.4-27        crayon_1.5.2       
[52] pkgconfig_2.0.3     xml2_1.3.5          prettyunits_1.1.1  
[55] timechange_0.2.0    httr_1.4.7          rstudioapi_0.15.0  
[58] R6_2.5.1            units_0.8-2         compiler_4.1.0     

[mpadge]

@ruimgbarros As with all of the above, such errors are never related to the package itself, and you'll just have to wait for a resolution elsewhere. Sorry.

[ruimgbarros]

Hi! After restarting my computer, I was able to run it. Thank you!