search

meganz / MEGAsync

Easy automated syncing between your computers and your MEGA Cloud Drive
Other
1.58k stars 278 forks source link

Performance problem #820

Open LSmyrnaios opened 11 months ago

LSmyrnaios commented 11 months ago

Hi, I had a few thousand of files to sync and had to wait forever for them to finish.

When I opened the task manager, I noticed that MegaSync was undermining itself in terms of performance. It uses only 2 CPU cores (4 threads), while my CPU has 8 cores and 16 threads. Also, the network bandwidth used is 1% of my total bandwidth, which is 100 Mbps.

Please upgrade the paralilazation code to take advantage of the hardware leaps of the last 10 years.

Thank you.

Notes: 1) In settings, I have set "No limit" for download and upload. Also I have set the higher number of "connections per transfer", which is 6. 2) I am running MegaSync Desktop v. 4.9.5, on Windows 10 (latest build)

mattw-mega commented 11 months ago

Hi @LSmyrnaios , were your four threads at 100% CPU each? On modern CPUs that should be plenty to perform the encryption needed for upload, at a rate that would max out a typical network connection. More likely, it was limited by the max rate of file node creation. Or possibly the speed of your local disk. If you were to post pictures of the graphs from the performance tab of your Task Manager then we could have a better idea. thanks

LSmyrnaios commented 11 months ago

Hi @mattw-mega , the 4 threads are going up and down in terms of load.

In terms of network connection, it seems that when there are multiple small files, the used bandwidth is very very small. It is moderately used or even reaches 100 Mbps, when a single or more large files are downloaded.

Also, as you see in my screenshot, I got only SSDs, so the speed my local disk is not an issue.

Screenshot 2023-07-14 005643

mattw-mega commented 11 months ago

Right, certainly you are not CPU bound across multiple threads, so this is not an issue with encrypting/decrypting which can use multiple threads. Those graphs could possibly be one single thread at 100% spread across 4 CPUs though, with the OS swapping the core the thread is on for heat management etc. Do your syncs overall contain many hundreds of thousands of nodes (files/folders), or even millions? (not just the files to update, but the total in the sync/s). If so there are some inefficiencies that can show up when things get very large like that. The work in #770 addresses those so you could try the latest alpha version there - but please be aware it is alpha software, it could be buggy, and you should be careful and have other backups of your files. thanks

LSmyrnaios commented 11 months ago

@mattw-mega I have a bit over 110 thousand files and a bit less than 29 thousand folders.

Thank you for upgrading the app in the upcoming v.5 Unfortunately, I can't do alpha-testing at the moment.