search

meganz / MEGAsync

Easy automated syncing between your computers and your MEGA Cloud Drive
Other
1.64k stars 281 forks source link

Scanning takes a lot of time #596

Open alepulver opened 3 years ago

alepulver commented 3 years ago

I'm using MEGAsync 4.5.1 on Windows 10 x64 and after setting up a folder with 80k files (and 128GB) it takes about 20 min for "scanning...". My drive is an SSD and I'm surprised it takes so long, as Total Commander calculates the occupied space, number of files and directories in a few seconds. For the Windows home folder (187k files and 80GB) it still hasn't finished after almost an hour of "scanning".

It would also be useful to have a progress bar or some kind of indicator during this process. I tried checking the logs but they don't look like plain text.

zDEFz commented 3 years ago

Keep your machine on and shutdown as little as possible, sleep mode preferred. Don't nest your folders much etc.

Also refer to #151 Actually part of the answer stems from one of the replies I got from the MEGA support.

silvtal commented 3 years ago

Oh, so this hasn't been fixed yet. That's sad to say the least.

Why would I change my work organization just for this app to work?

I'll just have to look for an alternative

webdev69420 commented 3 years ago

Oh, so this hasn't been fixed yet. That's sad to say the least.

Why would I change my work organization just for this app to work?

I'll just have to look for an alternative

Tell me if you find a solid alternative. I've tried pCloud and Icedrive but I trust none of them enough (yet) to manage and sync my files properly. MEGA is a bit slow and rigid sometimes but at least it never fucked up my synced files badly for now. It's really hard to switch to another cloud provider because I'm scared their sync won't be on par with MEGA... (Well except for popular services like Google Drive, Dropbox, Sync or Box but their prices aren't as attractive and concerning privacy they aren't the best either).

zDEFz commented 3 years ago

Also MEGA supports file versioning. My reason to stay with MEGA.

asheroto commented 3 years ago

Scanning does seem to take a bit.

Consider the application Everything - it scans my entire hard drive in less than 20 seconds. Whereas megasync will take a few minutes to scan just 6 sync folders.

I suppose it is using multiple threads to achieve this. Might be something you guys look into using.

zDEFz commented 3 years ago

Scanning does seem to take a bit.

Consider the application Everything - it scans my entire hard drive in less than 20 seconds. Whereas megasync will take a few minutes to scan just 6 sync folders.

I suppose it is using multiple threads to achieve this. Might be something you guys look into using.

You are wrong. Scanning takes long because it checks serverside for file revisions + the weird encryption. You can have a LOT of those.

mattw-mega commented 1 year ago

Hi @alepulver , you may like to know we have a big update coming for MEGAsync which should be much faster starting up, especially when the sync is already up to date. You can read about it and perhaps help test it here: https://github.com/meganz/MEGAsync/issues/770 thanks

DanielJoseDolz commented 1 year ago

I have hundres of thousand of file and the scanning never stops. It never uploads. It is a shame Mega do not warn this. "up to 8TB and XXXX files". If there is no solution I have to confront the manager and tell her we wasted money and the design schema will not work

zDEFz commented 1 year ago

I have hundres of thousand of file and the scanning never stops. It never uploads. It is a shame Mega do not warn this. "up to 8TB and XXXX files". If there is no solution I have to confront the manager and tell her we wasted money and the design schema will not work

MEGA is not suitable for folders with many files / nesting. The support told me that one should have instead small amount of big files instead to fill the storage. You just don't use it properly I guess. Store locally, and then package into large archive files.

DanielJoseDolz commented 1 year ago

MEGA is not suitable for folders with many files / nesting. The support told me that one should have instead small amount of big files instead to fill the storage. You just don't use it properly I guess. Store locally, and then package into large archive files.

This warning in the pricing/features section would be nice.

Still working on it. Apparently the "backup" option could do it, but here is no versioning (it is a must for us because we are doing this out of ransomware fear). So I am having mixed feelings here. If I were on linux, I would have solved this with a simple rsync. This is frustrating

mattw-mega commented 1 year ago

Hi @DanielJoseDolz , @zDEFz , we have a much improved version of MEGAsync coming up which is fine with extremely large numbers of files and folders, and with nesting, especially once the files and folders are in sync. In particular, on windows it uses a much better Win32 function to retrieve the directory listings. On the first scan, it has to fingerprint the files too, which does take a while. But on second and subsequent scans, provided all other attributes are unchanged, it does not re-fingerprint the files. You can try it out at #770 if you like. For an up to date sync, you should find it very quick to start up and scan the existing syncs for any changes, and be back to idle. Especially if the OS has cached the directory information. thanks

mattw-mega commented 1 year ago

Hi @DanielJoseDolz , about the versioning in Backups. Actually backups are very similar to syncs, and versioning works the same way. If you have versioning turned on in your account, Backups will make versions. However the webclient does not show those under the Backup area. If you stop the Backup and move the folder tree to your main account area, you will see the versions are there. This came up internally recently, and I'm sure that will be fixed in due course. thanks

biggestsonicfan commented 7 months ago

Did anything ever happen with this? Been "scanning" for about six hours now. I paused a transfer and rebooted and I lost my entire queue somehow. So....

rodrihalo117 commented 7 months ago

Did anything ever happen with this? Been "scanning" for about six hours now. I paused a transfer and rebooted and I lost my entire queue somehow. So....

Same here, waited all the day and it's still "scanning"

rautamiekka commented 7 months ago

Did anything ever happen with this? Been "scanning" for about six hours now. I paused a transfer and rebooted and I lost my entire queue somehow. So....

Did anything ever happen with this? Been "scanning" for about six hours now. I paused a transfer and rebooted and I lost my entire queue somehow. So....

Same here, waited all the day and it's still "scanning"

Did you keep an eye on the task manager ? Writing the SQLite databases & the pure-text log files ain't exactly fast, especially if the to-be-scanned files are on the same drive as your home (user profile) folder.

biggestsonicfan commented 7 months ago

I didn't. I just literally assumed those would have been propagated when it had them all in the queue. I am unsure why I had to restart from the beginning. I've now reached a point where a bunch of 0 byte files serve as placeholders and resuming works.

LuisMCunha commented 5 months ago

Same here, it's been scanning for multiple days now with the PC always on. This for 26GB worth of files, it's kinda ridiculous.

johentsch commented 4 months ago

Same here, something must be wrong, definitely: image It just keeps scanning new files. 37 million??? Both ncdu and find .//. ! -name . -print | grep -c // tell me the folder recursively includes 340600 files only! Seems like an infinite loop to me...

EDITED: This is megasync 5.2.1-5 running on Arch Linux.

rautamiekka commented 4 months ago

Same here, something must be wrong, definitely: image It just keeps scanning new files. 37 million??? Both ncdu and find .//. ! -name . -print | grep -c // tell me the folder recursively includes 340600 files only! Seems like an infinite loop to me...

This is megasync 2.5.1-5 running on Arch Linux.

That's ancient ! Hell, way older than OP's. You can literally sum the numbers together and it'd still be far less than the current newest, which has at least mostly fixed the problem (I haven't tried to sync a tree of millions of files since then cuz that takes far more than 2TB, though).

johentsch commented 4 months ago

Hahaha, I'm sorry, I swapped the first digits, version is 5.2.1-5.