Turbo is a framework built for LuaJIT 2 to simplify the task of building fast and scalable network applications. It uses a event-driven, non-blocking, no thread design to deliver excellent performance and minimal footprint to high-load applications while also providing excellent support for embedded uses.
isolated function which parse headers of multipart from parse_multipart_data() add kwargs.streaming_multipart_bytes in httpserver let user to parse multipart data in streaming and saved huge file (excess kwargs.large_body_bytes or 512 if no setting) to /tmp
the 3x memory of files as instance:
file about 100M so the multipart/form-data body is about 100M
in function iostream.IOStream:_read_to_buffer()self._read_buffer:append_right(ptr, sz) would expand the buffer size to 100M
in function iostream.IOStream:_consume(loc)chunk = ffi.string(ptr + self._read_buffer_offset, loc) converting/coping c string to lua string takes 100M
in function httputil.parse_multipart_data(data, boundary)argument[1] = data:sub(v1, b2) slicing the huge file content as new string takes 100M
the solution contains in the PR as below:
streaming parsing will parse every chunk to prevent to expand buffer size
parsing chunk with raw buffer struct to prevent to convert lua string
saving the hug file under /tmp in chunk as in handler user can os.rename() the file under /tmp to where they want
Testing result:
$ luajit examples/multipart.lua
response ok and the file uploaded/received are identical
isolated function which parse headers of multipart from parse_multipart_data() add kwargs.streaming_multipart_bytes in httpserver let user to parse multipart data in streaming and saved huge file (excess kwargs.large_body_bytes or 512 if no setting) to /tmp
the 3x memory of files as instance: file about 100M so the multipart/form-data body is about 100M
function iostream.IOStream:_read_to_buffer()
self._read_buffer:append_right(ptr, sz)
would expand the buffer size to 100Mfunction iostream.IOStream:_consume(loc)
chunk = ffi.string(ptr + self._read_buffer_offset, loc)
converting/coping c string to lua string takes 100Mfunction httputil.parse_multipart_data(data, boundary)
argument[1] = data:sub(v1, b2)
slicing the huge file content as new string takes 100Mthe solution contains in the PR as below:
os.rename()
the file under /tmp to where they wantTesting result:
$ luajit examples/multipart.lua
response ok and the file uploaded/received are identical