r-lib / httr

httr: a friendly http package for R
https://httr.r-lib.org
Other
986 stars 1.99k forks source link

Workaround for FTP crashes in some libcurl versions #755

Closed jeroen closed 1 month ago

jeroen commented 1 month ago

There is a bug in libcurl 8.5.0 - 8.7.1 that causes a crash if you mix http and ftp requests in the same handle. Unfortunately these versions of libcurl are common on both Linux and MacOS.

A reprex that crashes both on Ubuntu 24.04 and all current MacOS versions:

library(httr)
url1 <- "https://en.wikipedia.org/wiki/Bioconductor"
response1 <- GET(url1)
url2 <- "ftp://ftp.ensembl.org/pub/release-71/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.71.gtf.gz"
response2a <- GET(url2, write_disk(tempfile()))
response2b <- GET(url2, write_disk(tempfile()))

Pooling handles is a legacy feature that is not recommended anymore. The only use is sharing cookies between requests. However FTP requests don't have cookies, so it makes no sense to pool the handle. A straightforward fix therefore is to only pool handles for http requests.

Even though httr is not intended for FTP and not actively maintained, I still think we should push this fix.

jeroen commented 1 month ago

Actually I should better try to work around it in curl.