Cannot upload more than 128 files using req_perform_parallel

[maxsutton]

When uploading more than 128 files, req_perform_parallel fails with an error message:

Error in gzfile(file, "rb") : all 128 connections are in use

This occurs even when restricting the curl pool to only use 2 connections at once.

It seems that httr2 opens some files by itself (such as by checking utils::packageVersion("httr2"), so the real limitation is more like 124 files. I have tested the same setup below using req_perform_sequential() and it succeeds.

Code

Note: With an endpoint of example_url(), a good request fails with an HTTP 404 error, but req_perform_parallel continues. When using more than 128 files, the error occurs before the first request is initiated, so the endpoint isn't a concern.

library(httr2)

N <- 10

test_upload <- function(N, endpoint = example_url()) {
  pool <- curl::new_pool(total_con = 2, host_con = 2)

  files <- sapply(1:N, function(x) {f <- tempfile(); writeLines(as.character(x), f); f})

  reqs <- lapply(files, function(f) {request(endpoint) |> req_body_file(f)})

  parallel <- req_perform_parallel(reqs, pool = pool, on_error = "continue")
}

Good (and expected behaviour)

small <- test_upload(10)
good <- test_upload(120)

Expected output: a progress bar and no errors. small and good are lists of the status of the requests made.

Bug

bug <- test_upload(129)

Error in gzfile(file, "rb") : all 128 connections are in use

## Session info ``` ─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value version R version 4.4.0 (2024-04-24 ucrt) os Windows 10 x64 (build 19045) system x86_64, mingw32 ui RStudio language (EN) collate English_Australia.utf8 ctype English_Australia.utf8 tz Australia/Sydney date 2024-07-10 rstudio 2024.04.0+735 Chocolate Cosmos (desktop) pandoc NA ─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── package * version date (UTC) lib source cachem 1.1.0 2024-05-16 [1] CRAN (R 4.4.0) callr 3.7.6 2024-03-25 [1] CRAN (R 4.4.0) cli 3.6.2 2023-12-11 [1] CRAN (R 4.4.0) curl 5.2.1 2024-03-01 [1] CRAN (R 4.4.0) devtools 2.4.5 2022-10-11 [1] CRAN (R 4.4.0) digest 0.6.35 2024-03-11 [1] CRAN (R 4.4.0) ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.4.0) fansi 1.0.6 2023-12-08 [1] CRAN (R 4.4.0) fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.4.0) fs 1.6.4 2024-04-25 [1] CRAN (R 4.4.0) glue 1.7.0 2024-01-09 [1] CRAN (R 4.4.0) htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0) htmlwidgets 1.6.4 2023-12-06 [1] CRAN (R 4.4.0) httpuv 1.6.15 2024-03-26 [1] CRAN (R 4.4.0) httr2 * 1.0.1 2024-04-01 [1] CRAN (R 4.4.0) later 1.3.2 2023-12-06 [1] CRAN (R 4.4.0) lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0) magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.4.0) memoise 2.0.1 2021-11-26 [1] CRAN (R 4.4.0) mime 0.12 2021-09-28 [1] CRAN (R 4.4.0) miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.4.0) pillar 1.9.0 2023-03-22 [1] CRAN (R 4.4.0) pkgbuild 1.4.4 2024-03-17 [1] CRAN (R 4.4.0) pkgload 1.3.4 2024-01-16 [1] CRAN (R 4.4.0) processx 3.8.4 2024-03-16 [1] CRAN (R 4.4.0) profvis 0.3.8 2023-05-02 [1] CRAN (R 4.4.0) promises 1.3.0 2024-04-05 [1] CRAN (R 4.4.0) ps 1.7.6 2024-01-18 [1] CRAN (R 4.4.0) purrr 1.0.2 2023-08-10 [1] CRAN (R 4.4.0) R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0) rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.4.0) Rcpp 1.0.12 2024-01-09 [1] CRAN (R 4.4.0) remotes 2.5.0 2024-03-17 [1] CRAN (R 4.4.0) rlang 1.1.3 2024-01-10 [1] CRAN (R 4.4.0) rstudioapi 0.16.0 2024-03-24 [1] CRAN (R 4.4.0) sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.0) shiny 1.8.1.1 2024-04-02 [1] CRAN (R 4.4.0) stringi 1.8.4 2024-05-06 [1] CRAN (R 4.4.0) stringr 1.5.1 2023-11-14 [1] CRAN (R 4.4.0) urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.4.0) usethis 2.2.3 2024-02-19 [1] CRAN (R 4.4.0) utf8 1.2.4 2023-10-22 [1] CRAN (R 4.4.0) vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.4.0) webfakes 1.3.1 2024-04-25 [1] CRAN (R 4.4.0) xtable 1.8-4 2019-04-21 [1] CRAN (R 4.4.0) ```

[hadley]

Slightly simpler reprex:

library(httr2)

temp <- tempfile()
writeLines("Hello, world!", temp)

# OK
reqs <- lapply(1:129, function(i) request(example_url()))
parallel <- req_perform_parallel(reqs, on_error = "continue")

# NOT OK
reqs <- lapply(1:129, function(i) request(example_url()) |> req_body_file(temp))
parallel <- req_perform_parallel(reqs, on_error = "continue")

So I'm holding on to connections somewhere in the path way that serves the file.

...

Hmmm, I bet this is it:

    con <- file(data, "rb")
    # Leaks connection if request doesn't complete

So I'll need to think of a better way to clean up the connection created here.

[hadley]

Maybe req_body_apply() needs to include some callback that will close the open connection? And then maybe give req_handle() some sort of done() callback which we can call from $suceed() and req_perform()?

...

Hmmm, it must be more complicated than that because I do see the correct close function getting called.

...

Ooooh, the problem is that we're opening the connection when we register the call, but we only close it when it's done.

[maxsutton]

Thanks @hadley! I didn't have the headspace to dig into the package -- I've only just started to use it. So I really appreciated you sharing your thought process. Immensely helpful.