search

babelouest / ulfius

Web Framework to build REST APIs, Webservices or any HTTP endpoint in C language. Can stream large amount of data, integrate JSON data with Jansson, and create websocket services
https://babelouest.github.io/ulfius
GNU Lesser General Public License v2.1
1.07k stars 183 forks source link

chunk size in file_upload_callback #204

Closed elockman closed 2 years ago

elockman commented 2 years ago

When running an application much like sheep_counter, only the first chunk seems to be limited by the instance.max_post_param_size parameter.

Setting the limit to 1kB (1*1024), the first chunk is limited, but future chunks seem to be limited to about 16k. Setting the limit to 32kB still seems to have a limit of about 16kB, so the 16kB limit may be set elsewhere.

Below is example debug:

2021-11-08T18:40:38Z - webui_app DEBUG: Chunk 1 : from file 'xImage' of the key 'fileToUpload1', offset 0, size 604, cls is 'my cls'
2021-11-08T18:40:38Z - webui_app DEBUG: Chunk 2 : from file 'xImage' of the key 'fileToUpload1', offset 604, size 15718, cls is 'my cls'
2021-11-08T18:40:39Z - webui_app DEBUG: Chunk 3 : from file 'xImage' of the key 'fileToUpload1', offset 16322, size 15718, cls is 'my cls'
2021-11-08T18:40:39Z - webui_app DEBUG: Chunk 4 : from file 'xImage' of the key 'fileToUpload1', offset 32040, size 15718, cls is 'my cls'
2021-11-08T18:40:39Z - webui_app DEBUG: Chunk 5 : from file 'xImage' of the key 'fileToUpload1', offset 47758, size 15718, cls is 'my cls'
2021-11-08T18:40:39Z - webui_app DEBUG: Chunk 6 : from file 'xImage' of the key 'fileToUpload1', offset 63476, size 15718, cls is 'my cls'
2021-11-08T18:40:39Z - webui_app DEBUG: Chunk 7 : from file 'xImage' of the key 'fileToUpload1', offset 79194, size 15718, cls is 'my cls'

Note that the first chunk has a 604 size, while the rest are 15k+. All chunks are expected to be less than the instance.max_post_param_size parameter.

I am using a custom embedded linux platform, using the following make command: make CURLFLAG=1 GNUTLSFLAG=1 JANSSONFLAG=1

elockman commented 2 years ago

I've tried commenting out line 239 in ulfius.c, but that had no effect.

//con_info->max_post_param_size = 0;

babelouest commented 2 years ago

I assume you're talking about what's happening inside the function file_upload_callback.

When using this callback function, ulfius uses MHD_create_post_processor first: https://github.com/babelouest/ulfius/blob/master/src/ulfius.c#L528 Then, inside mhd_iterate_post_data, ulfius calls file_upload_callback if the current post parameter is a file: https://github.com/babelouest/ulfius/blob/master/src/ulfius.c#L333

The parameter max_post_param_size has no effect on file_upload_callback before this function is called before testing max_post_param_size.

And finally, the chunk size in file_upload_callback can't be forced by ulfius nor libmicrohttpd. Ulfius uses libmicrohttpd's MHD_create_post_processor with a fixed buffer_size parameter value of 65536. But even if the buffer_size could be changed in the application, it wouldn't be this value on every iteration, because this parameter is the max size.

If you explain what you're trying to achieve, I can try to help you find a more suitable solution using ulfius' current API.

elockman commented 2 years ago

Thank you. I'm running into RAM issues when uploading large files. I can SCP the files into my embedded system, but having issues with file uploads through a web interface. I realize these use different protocols, but I was hoping to change chunk sizes to investigate the source of the RAM usage.

I will close this "issue".