search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
3.67k stars 511 forks source link

File uploads are slow, increasing chunk size can help #6169

Closed HuFlungDu closed 3 weeks ago

HuFlungDu commented 3 weeks ago

Is your feature request related to a problem? Please describe. I noticed that libmeshctrl (and, by extension, probably meshctrl.js) can upload files to devices DRAMATICALLY faster than the web UI. On the order of a 30mb file taking 10 minutes through the web UI, but mere seconds when done through libmeshctrl. I tracked this issue down to the chunk size in in the web UI being considerably smaller than libmeshctrl uses, only 0x4000 bytes. This seems to cause a lot of overhead and makes the file upload incredibly slow. I tried increasing the chunk size to various values, specifically 0x10000 and 0x100000, and it dramatically increased the upload speed.

Describe the solution you'd like I created a commit to increase the chunk size to 0x100000, but I couldn't figure out how to properly generate all the translation files from the translation server. Also I think it makes more sense for people core to the meshcentral team to determine a chunk size that is appropriate for this application, as I don't know all the considerations that were put into choosing the number which was chosen.

Short of it: File uploads through the web UI seem like they are slower than they ought to be, it would be nice if the chunk size could be increased, or to have a config option to increase it in case the default needs to support embedded devices or something.

si458 commented 3 weeks ago

ok this is intresting. just did my own experiment and indeed increasing those values does speed up the upload! in theory we can also increase the file download too! so ill look into that i found setting the value to 65565 is better than 0x100000 because 0x100000 is 1,048,576, and you have to remember this is on the handlebar files which is client side! so the bigger the buffer the more memory the client side would need in there browser! but yes indeed just increasing from 16384 to 65565 does make a difference so nice catch!

HuFlungDu commented 3 weeks ago

Thanks Simon, that is a significant improvement!

I will say, 1,048,576 is still only 1MB, and by my testing meshcentral itself already takes up ~200MB on the user's device. Since the buffer gets discarded immediately after use and cleaned up by the garbage collector shortly afterward, I don't think that .5% increase during an upload operation should be too much of a burden, and it is much faster. I tried uploading a 10GB file with the 1MB block size and didn't see any noticeable jump in memory usage over the old 16k.

Admittedly, though, that's hardly an exhaustive test, and there could be edge cases I'm missing, hence the deference, just something to think on. Super happy to just have the 65k, though!

si458 commented 3 weeks ago

For my testing, i used my pi zero w (on wifi too) and uploaded a 58mb file. Originally, it took about 10 minutes, but after changing it to the 65565, it only took 3 minutes! Which is a big improvement!