search

kernelsauce / turbo

Turbo is a framework built for LuaJIT 2 to simplify the task of building fast and scalable network applications. It uses a event-driven, non-blocking, no thread design to deliver excellent performance and minimal footprint to high-load applications while also providing excellent support for embedded uses.
http://turbo.readthedocs.io/
Apache License 2.0
525 stars 84 forks source link

huge memory use when upload large file with form-data #352

Open gary8520 opened 4 years ago

gary8520 commented 4 years ago

Hi, We meet an issue that turbo will occupy memory up to three times of upload file size during the request. If we need to upload file with size 60MB, it will use memory up to 180MB. It would cause out of memory in the embedded system with limited memory.

After tracing code, the memory use as below: one is buffer enlarged in the iostream.IOStream:_read_to_buffer() calling self._read_buffer:append_right() one is convert ffi.string to lua string in iostream.IOStream:_consume() as request.body one is string slice argument[1] = data:sub(v1, b2) during parsing multipart httputil.parse_multipart_data()

In our scenario we need to save the file in the handler, it would cost one more time of memory.

hippi777 commented 4 years ago

hi there! :)

as a workaround, u could use a larger swap partition/file, so it wont crash; or just detect the size, and decide whether to save it, or keep it in memory...

otherwise i think its possible to reduce it to 2x the size by releasing the 1st copy before reaching the 3rd "stage"; or maybe even to keep only one and, use substrings (that can give u 2x the original size), but maybe even an iterator based access could be used, so that way it would consume insignificantly more mem than the original size.

however it depends on ur usecase, and im still not really familiar with the internals of turbo, but actually i already decided that i will use it, or write my own, based on it, or on naked syscalls. :D but most likely i wont be that much rich with my time, and actually i like what ive seen so far from the sauce, just ive got really unique funds in my app, and i wanna reshape things to align with those... (its not really oop based, but everything will be in a complex graph) - so i actually care about the future of turbo, just i still havent had the time to start to work with it, but i plan to do so asap, so i hope i will be able to join in the near future, and cuz of that, u can actually ask for my help with minor things. actually that can be a good thing for me, just to catch up with the flow, just i cant promise nothing right now other than my good will, and also, in that case please provide me with some pointers if u know an easy path to jump into the middle of the code base...

good luck, have fun, all the bests, and also, i wish u a happy new year! :D

niziak commented 4 years ago

You don't really want to use swap with ES. As a workaround you can let luajit to use all spare memory by changing behavior of kernel (i.e. in file /etc/sysctl.conf) with: vm.overcommit_memory=1

kernelsauce commented 3 years ago

For these kind of special events I would reimplement the HTTPServer class to stream the data required for your application. Turbo is quite flexible to handle this. Read the source and see where you need to override functions to not buffer the whole file.