When querying an endpoint of a REST API I set up with R/plumber, I experience substantial time differences between R/httr, R/curl and download.file. While download.file takes around 8 seconds, httr and curl take 25 seconds or more for exactly the same query using the same client and host server (both CentOS 7). With wget and curl on command line it also takes 8 seconds.
Querying the API
system.time({
r = GET("http://myserver.com/path/to/endpoint")
warn_for_status(r)
stop_for_status(r)
x = content(r)
})
# user system elapsed
# 9.329 0.345 32.946
system.time({
req = curl_fetch_memory("http://myserver.com/path/to/endpoint")
})
# user system elapsed
# 1.707 0.268 25.160
system.time({
curl_download("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 1.608 0.417 25.237
system.time({
curl_fetch_disk("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 1.656 0.467 25.468
system.time({
download.file("http://myserver.com/path/to/endpoint", tempfile())
})
# user system elapsed
# 0.980 0.469 8.260
Sounds like the problem arises from curl, not from httr. I'd suggest filing an issue for the curl package, preferably including a reproducible example so there's some chance of being able to fix it.
When querying an endpoint of a REST API I set up with R/plumber, I experience substantial time differences between
R/httr
,R/curl
anddownload.file
. Whiledownload.file
takes around 8 seconds,httr
andcurl
take 25 seconds or more for exactly the same query using the same client and host server (both CentOS 7). Withwget
andcurl
on command line it also takes 8 seconds.Querying the API
Response header
sessionInfo()
libcurl version
libcurl 7.29.0 with NSS/3.53.1
Update 04-10-2021
There seems to be a discrepancy between result of
headers()
and what the actual header is: