Closed rplati closed 4 years ago
I can't reproduce. May be you have some container memory restrictions? Apart from that I recommend:
readBin
for example?Thanks for the reply, @dselivanov . I tried the rexyai/restrserve:0.2.2
image but it did’n change the container’s behavior.
The issue does seem to be related to the memory the container has available. I was able to reproduce the behavior on a linux server but at a much higher threshold (n=1600000000; 1.5 Gb). Starting the container with the option --memory=80m
reproduces the behavior at a lower filesize.
If it is possible to implement, an error message would be appreciated in this kind of situation.
Unfortunately we can't do anything about that - this is controlled by OS. Which means that if process consumes all the memory, OOM killer will kill it.
Also keep in mind that if you start containers on windows or os x, then the virtual machine has its own restriction on the max memory for the containers (on OS X it is 2gb by default). So even if you specify --memory=4g
it will be capped to to 2g. You need to control this separately in docker app options.
I’ve investigated the issue further. As far as I can gather, RestRserve quickly drains the container’s memory when the data passed to it are in multipart form.
I’ve monitored the container’s resources with docker stats
and varied the container’s available memory with the –-memory
option when starting the container. When the container has X amount of RAM available, a file of size Y will cause the OOM killer to kill the process resulting in an empty reply from server. When the body is passed as text/plain
, the ratio of X and Y is about 2, i.e. when the container has 1 GB of memory available, up to 500 MB can be passed to the endpoint successfully. With a multipart body, the ratio is about 7, i.e. we need to allocate about 7 times the amount of memory to the container as the largest file we wish to pass (not considering concurrent requests). I got roughly the same results running the container on windows and linux.
The multiplier of 7 is prohibitive with large file sizes and possible concurrent requests. Would you consider investigating and possibly optimizing the code?
@rplati thanks for investigation, I can reproduce. I've opened a separate ticket - #151
I’m running
RestRserve
in a linux container starting from imagerocker/r-ver:3.6.3
. I’ve noticed that when the size of the multipart body grows above a certain threshold, RestRserve returns an empty response. This does not happen when I runRestRserve
directly on my Windows machine.This is a minimal example: Dockerfile
restrserve.R
I build the image and start the container mapping tha container's port 80 to the host's port 5017 (port 80 not allowed on windows). Then I submit a post request to the endpoint.
On my machine, when n=10000000 (99.1 Mb) everything works fine and the container log looks like this.
But when the data size is above a certain threshold (on my machine n=20000000, 198.1 Mb) the endpoint returns an empty response and the log looks like this
i.e. the call seems to get stuck at parsing the multipart body. The container should have enough memory available. Any idea how to debug this? Is there some inherent limit I should be aware of? I would like to post files to the endpoint with max size of about 2 GB.