Closed dselivanov closed 4 years ago
Can you provide a code to reproduce this?
Same as in the #150. This takes more than 2gb of RAM during request processing. Seems related to the https://github.com/rexyai/RestRserve/blob/25ca00a784e3d1681c84867954ba93b95466bca1/src/parse_multipart.cpp#L106
library(data.table)
library(httr)
# Generate data
n<-20000000 # observations
dta<-data.frame(
numVar=round(rnorm(n),3),
charVar=c("foo", "bar")[(runif(n)<0.5)+1]
)
# Write to file
tmp <- tempfile()
fwrite(dta, tmp)
utils:::format.object_size(file.info(tmp)$size, "auto")
#> [1] "198.1 Mb"
# POST request with file
time<-system.time({
rs <- POST(
url = "http://127.0.0.1:8080/echo",
body = list(dta = upload_file(tmp)),
encode = "multipart"
)
})
If issue in the cpp_parse_multipart_body
we can use it directly same as shown in test-parse-multipart.R
.
Surprisingly the issue seems is in the request$get_file()
The issue is in a way how raw vector is sliced: https://github.com/rexyai/RestRserve/blob/25ca00a784e3d1681c84867954ba93b95466bca1/R/Request.R#L211.
In case of large file (200m as in the example) the sequence of indices to get the file from the blob is large (200m integers or numeric which is is 800mb/1600mb of RAM). Even with R's ALTREP these indices are materialized during the raw vector subset.
An option would be to use R's readBin
function, but unfortunately it doesn't support offset
argument... Similar issue described here.
So I've created a small C++ function to slice raw vectors. @rplati see dev branch.
Now memory usage should be very small (~2x of the size of multipart request as we essentially doubling memory if we read file from blob and allocate a separate variable).
I can confirm that parsing the multipart body no longer draws unreasonable amounts of memory when starting from Docker image rexyai/restrserve:dev
. Many thanks for rapidly fixing the issue!
From #150 (and I can reproduce this):