r-lib / httr

httr: a friendly http package for R
https://httr.r-lib.org
Other
986 stars 1.99k forks source link

Unexpected results with http_error() #707

Closed jacpete closed 2 years ago

jacpete commented 2 years ago

I am having some issues with unexpected results from http_error(). Documentation and examples show that the input can pass a url to the function or a response object. However, I am having issues with the url: https://egisdata.baltimorecity.gov/egis/rest/services/CityView/Liquor_Licenses/MapServer/0. If I just pass the url to http_error I get TRUE (there was an error), but if I pass a GET response for the url to http_error I get FALSE (no error; the expected result). Am I missing something or is this an issue with http_error?

library(httr)
url <- "https://egisdata.baltimorecity.gov/egis/rest/services/CityView/Liquor_Licenses/MapServer/0"

Unexpected Results

httr::http_error(url)
# [1] TRUE

Expected Results

httr::GET(url)
# Response [https://egisdata.baltimorecity.gov/egis/rest/services/CityView/Liquor_Licenses/MapServer/0]
#   Date: 2021-10-27 17:09
#   Status: 200
#   Content-Type: text/html;charset=utf-8
#   Size: 4.92 kB
# <html lang="en">
# <head>
# <title>Layer: Liquor Licenses (ID: 0)</title>
# <link href="/egis/rest/static/main.css" rel="stylesheet" type="text/css"/>
# </head>
# <body>
# <table width="100%" class="userTable">
# <tr>
# <td class="titlecell">
# ArcGIS REST Services Directory

httr::http_status(httr::GET(url))
# $category
# [1] "Success"
# 
# $reason
# [1] "OK"
# 
# $message
# [1] "Success: (200) OK"

httr::http_error(httr::GET(url))
#[1] FALSE

Environment Information

sessionInfo()
# R version 4.1.1 (2021-08-10)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: Pop!_OS 21.04
# 
# Matrix products: default
# BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
# LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
# 
# locale:
#  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
#  [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
# [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
# 
# attached base packages:
# [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
# [1] httr_1.4.2
# 
# loaded via a namespace (and not attached):
#  [1] Rcpp_1.0.7         pillar_1.6.2       compiler_4.1.1     class_7.3-19       tools_4.1.1        bit_4.0.4          jsonlite_1.7.2     RSQLite_2.2.8     
#  [9] memoise_2.0.0      lifecycle_1.0.0    tibble_3.1.4       pkgconfig_2.0.3    rlang_0.4.11       DBI_1.1.1          rstudioapi_0.13    curl_4.3.2        
# [17] parallel_4.1.1     fastmap_1.1.0      e1071_1.7-8        dplyr_1.0.7        xml2_1.3.2         generics_0.1.0     vctrs_0.3.8        classInt_0.4-3    
# [25] bit64_4.0.5        grid_4.1.1         tidyselect_1.1.1   glue_1.4.2         sf_1.0-2           R6_2.5.1           fansi_0.5.0        pbapply_1.5-0     
# [33] purrr_0.3.4        blob_1.2.2         magrittr_2.0.1     ellipsis_0.3.2     units_0.7-2        rvest_1.0.1        utf8_1.2.2         KernSmooth_2.23-20
# [41] proxy_0.4-26       cachem_1.0.6       crayon_1.4.1      
jacpete commented 2 years ago

After doing some exploring of the source code. It looks like the http_error.character method uses HEAD() under the hood retrieves the error code and assesses if its greater than 400.

https://github.com/r-lib/httr/blob/21ff69f219ad11298854a63b8f753389088cf382/R/response-status.r#L144-L181

The response it returns is error 405 which is "Method Not Allowed". This tells me that it is likely that the server at that URL has disabled HEAD requests, but in my example I know a GET request succeeds. The solution in the package I am working on is to use httr::http_error(httr::GET(url)) as the url's I am working with never have a very big body and not a concern to retrieve it as well.

I will close the issue as this is a sufficient workaround.