search

django / asgi_ipc

IPC-based ASGI channel layer
BSD 3-Clause "New" or "Revised" License
37 stars 13 forks source link

MemoryError on large uploads #33

Open scardine opened 7 years ago

scardine commented 7 years ago

Looks like Daphne will choke on large POST requests (in my case while uploading a large file).

out 16 16:48:12 icr daphne[29020]: 2017-10-16 16:48:12,424 ERROR    Traceback (most recent call last):
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/daphne/http_protocol.py", line 174, in process
out 16 16:48:12 icr daphne[29020]:     "server": self.server_addr,
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/core.py", line 68, in send
out 16 16:48:12 icr daphne[29020]:     time.time() + self.expiry,
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/store.py", line 116, in append
out 16 16:48:12 icr daphne[29020]:     value.setdefault(name, []).append((item, expiry))
out 16 16:48:12 icr daphne[29020]:   File "/usr/lib/python3.6/contextlib.py", line 89, in __exit__
out 16 16:48:12 icr daphne[29020]:     next(self.gen)
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/store.py", line 64, in mutate_value
out 16 16:48:12 icr daphne[29020]:     pickle.dump(value, self.mmap, protocol=2)
out 16 16:48:12 icr daphne[29020]: MemoryError

I'm using Python 3.6.1 (64bit) on Ubuntu 17.04 with 8G RAM. Uploaded file is 1.6GB.

Looks like a problem with pickle, is there any way to use other pickle-compatible serializer that is better for larger objects?

andrewgodwin commented 7 years ago

Yup, seems likely, we still didn't get proper large file streaming in yet. I'd recommend using a WSGI server for large uploads for now.