search

r-lib / httr

httr: a friendly http package for R
https://httr.r-lib.org
Other
988 stars 1.99k forks source link

RETRY "HEAD" failure #499

Closed mmuurr closed 5 years ago

mmuurr commented 6 years ago

I've found that RETRY seems to hang when the verb is "HEAD". (Other verbs work just fine.) This behavior is consistent for me across two platforms (Amazon Linux and Mac OS X) using three different libcurl variants. It appears to be a connection-closing issue, and I've observed it with multiple distinct HTTP servers/endpoints. Can anyone else verify that they see this, too?

httr::GET("https://httpbin.org")
#> Response [https://httpbin.org/]
#>   Date: 2017-11-21 21:52
#>   Status: 200
#>   Content-Type: text/html; charset=utf-8
#>   Size: 13 kB
#> <!DOCTYPE html>
#> <html>
#> ...

httr::RETRY("GET", "https://httpbin.org")
#> Response [https://httpbin.org/]
#>   Date: 2017-11-21 21:52
#>   Status: 200
#>   Content-Type: text/html; charset=utf-8
#>   Size: 13 kB
#> <!DOCTYPE html>
#> ...

httr::HEAD("https://httpbin.org")
#> Response [https://httpbin.org/]
#>   Date: 2017-11-21 21:52
#>   Status: 200
#>   Content-Type: text/html; charset=utf-8
#> <EMPTY BODY>

httr::RETRY("HEAD", "https://httpbin.org")
#> Error in curl::curl_fetch_memory(url, handle = handle): transfer closed with 13011 bytes remaining to read
#> Request failed [ERROR]. Retrying in 1 seconds...
#> Error in curl::curl_fetch_memory(url, handle = handle): transfer closed with 13011 bytes remaining to read
#> Request failed [ERROR]. Retrying in 3 seconds...
#> Error in curl::curl_fetch_memory(url, handle = handle): transfer closed with 13011 bytes remaining to read
Session info ```R ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value version R version 3.4.1 (2017-06-30) os OS X El Capitan 10.11 system x86_64, darwin15.0.0 ui RStudio language (EN) collate en_US.UTF-8 tz America/Denver date 2017-11-21 ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── package * version date source backports 1.1.0 2017-05-22 CRAN (R 3.4.1) callr 1.0.0 2016-06-18 cran (@1.0.0) clipr 0.3.3 2017-06-19 cran (@0.3.3) clisymbols 1.2.0 2017-05-21 cran (@1.2.0) curl 2.8.1 2017-07-21 CRAN (R 3.4.1) DBI 0.7 2017-06-18 CRAN (R 3.4.1) devtools 1.13.3 2017-08-02 CRAN (R 3.4.1) digest 0.6.12 2017-01-27 CRAN (R 3.4.1) evaluate 0.10.1 2017-06-24 cran (@0.10.1) htmltools 0.3.6 2017-04-28 cran (@0.3.6) httr 1.3.1 2017-08-20 CRAN (R 3.4.1) knitr 1.17 2017-08-10 cran (@1.17) magrittr 1.5 2014-11-22 CRAN (R 3.4.1) memoise 1.1.0 2017-04-21 CRAN (R 3.4.1) R6 2.2.2 2017-06-17 CRAN (R 3.4.1) Rcpp 0.12.12 2017-07-15 CRAN (R 3.4.1) reprex 0.1.1 2017-01-13 CRAN (R 3.4.1) rmarkdown 1.6 2017-06-15 cran (@1.6) RPostgreSQL 1000.6-2 2017-09-14 local rprojroot 1.2 2017-01-16 CRAN (R 3.4.1) sessioninfo 1.0.0 2017-06-21 CRAN (R 3.4.1) stringi 1.1.5 2017-04-07 CRAN (R 3.4.1) stringr 1.2.0 2017-02-18 CRAN (R 3.4.1) whisker 0.3-2 2013-04-28 CRAN (R 3.4.1) withr 2.0.0 2017-07-28 CRAN (R 3.4.1) yaml 2.1.14 2016-11-12 cran (@2.1.14) ```
mmuurr commented 6 years ago

Solved by examining the request_build call in HEAD itself. This works:

httr::RETRY("HEAD", "https://httpbin.org", httr::config(nobody = TRUE))

Worth adding this edge-case into the RETRY wrapper? (I'm happy to make a quick pull-request, if so.)