search

steiza / docstore

For any civics-minded organization that needs a simple place to host documents publicly
http://a2docs.org/
7 stars 4 forks source link

"413 Request Entity Too Large nginx/1.9.13" when uploading 10.6 megabyte file #24

Closed vielmetti closed 5 years ago

vielmetti commented 8 years ago

I attempted to upload a 10.6 megabyte site plan from a project in front of the Planning Commission, and got this error message. @eby - the logs should show this from June 6 at about 12:50 pm.

The docs I can find about nginx refer to a stanza declaring client_max_body_size as the thing to change.

eby commented 8 years ago

Should be all set. Messed it up when I commented out some of the ssl stuff.

vielmetti commented 5 years ago

There was a report from Kai about some kind of "file too big" problem, investigating. That was a 135MB document.

eby commented 5 years ago

I double-checked and Nginx is still 2G. If you have additional info on what the error was I can dig into it. Could be a timeout or something else

steiza commented 5 years ago

Aha! Tornado has a maximum upload size, but you can now customize it for the docstore app via max_file_size: in settings.yml. Note that the higher this is, the more memory the server will use!

I also updated docstore to use just Python 3. @eby if this is an issue I could add some backwards compatibility. Python 2 is end of life in 5 months, but I know all too well there's lots of Python 2 still out there!

eby commented 5 years ago

Alright. It is up now with a 500M max_file_size to see how it does with memory. Also switched to python3. Let me know if you notice any issues @vielmetti

eby commented 5 years ago

So again I'm not a python programmer. It looks like this has come up quite a bit. Not sure if the latest tornado already fixes the memory use but I found that there is a module to help buffer straight to disk so that memory isn't filled up: https://pythonhosted.org/tornadostreamform/

Supposedly bottle and flask already moved to a buffer to disk model when uploads are > small limit.

steiza commented 5 years ago

Yeah, tornado.web.stream_request_body looks promising, but from some quick investigation it looks like it might have some compatibility issues with XSRF cookies (at least in tornado 5).

vielmetti commented 5 years ago

I was able to upload http://a2docs.org/view/539 (about 130M) now with no issues.

I'll leave this issue open for you all to sort out if there should be any code changes (especially re tornado.web.stream_request_body) but am happy otherwise to close as "solved".

cdzombak commented 5 years ago

@vielmetti / @eby, can this ticket be closed out now?

steiza commented 5 years ago

I think someone should take a look at if tornado.web.stream_request_body (or something similar) could be used before closing this out.

vielmetti commented 5 years ago

Started a new #28 so this one can close.