Scanning for 5 minutes at startup

Gittyperson commented 1 year ago

I did not make significant changes to my setup or files, but MEGAsync is now taking about 5 minutes to complete its initial "scanning" phase after the PC startup, with increased disk activity. After that, it works fine.

Its usual scanning time used to be about 10 seconds or less. Perhaps a recent MEGASync update did this, a switch from the 32 to the 64-bit MEGASync version or a Windows update, but I can't be sure.

Don't know if security software is related. There is increased CPU/disk activity from such software during this MEGAsync "scanning" phase. However, I tried both Microsoft's Defender and a 3rd party one (Emsisoft). Also tried disabling them, and adding exclusions (megasync.exe + the MEGAsync/Mega Limited folders). No difference.

Also tried reverting to various previous MEGAsync versions without success. At some point I think I got the previous behaviour back (very fast scanning) but I cannot do it again. Any suggestions?

Thank you MEGASync 4.6.8 64-bit, Windows 10 21H2 19044.1826

jx-mega commented 1 year ago

We would like to investigate more about your account status, could you email us at support@mega.nz from your MEGA account email address, please?

Gittyperson commented 1 year ago

Email sent.

Eldaw commented 1 year ago

Was there any resolution to this? I see the same problem, and I'm also using v4.6.8 64-bit on Windows 10. Starting MEGAsync causes several minutes of "Scanning" every time.

Gittyperson commented 1 year ago

It seems related to syncs with large amounts of data. I disabled one such sync and the scanning time went down significantly.

Not sure if it's an issue or normal behaviour. I'm quite confident this did not happen until recently (with the same syncs enabled), but perhaps I'm wrong.

Eldaw commented 1 year ago

The problem is that MEGAsync is doing a full scan at every startup. Of course if you reduce the amount of data to sync, the full scan will complete faster. But the MEGAsync client is not supposed to do a full scan at every startup.

Eldaw commented 1 year ago

LOL, I had reported the issue a few days ago via the app. Today I got a reply from the MEGA support team basically just giving instructions on how to uninstall and reinstall the client application. A template reply basically.

How will the bug ever get fixed if nobody from MEGA investigates why this issue is happening? Is there a configuration setting stored somewhere on my desktop PC that's causing the client to do a full scan at every startup? What troubleshooting can be done to figure out why it's happening, so that it can be properly fixed?

I hope somebody from MEGA can provide a helpful reply that moves things forward.

mattw-mega commented 1 year ago

MEGAsync does have to scan on startup, otherwise it won't be able to find out if anything changed in the local filesystem (and then apply those changes to the cloud side of the sync). There are improvements in the speed of that coming soon, using a more suitable windows API function for that.

Eldaw commented 1 year ago

@mattw-mega I think you are misunderstanding. There are 2 types of scan that MEGAsync performs: a quick scan and a full scan. Normally when starting up, MEGAsync does a quick scan.

The GUI of MEGAsync has a button that allows you to force a full scan if you want, which takes a very long time (presumably because this performs a full re-hashing of all the local files that are configured to sync).

I think you've confused the full scan with the quick scan, even though this discussion is clearly talking about a problem involving the full scan running when it shouldn't.

I have several computers syncing folders to MEGA in the cloud. On all of the computers except one, when MEGAsync starts it says 'Downloading File List...' for about 5 seconds and then it's ready. In other words, they all just do the normal scan at startup.

On just one computer, when MEGAsync starts it says 'Scanning' and that takes almost 10 minutes. Every single time it starts, even if nothing has changed. Clearly, it is doing the full scan every time for some reason.

I trust that helps clarify the problem.

I hope somebody from MEGA can provide a helpful reply that moves things forward.

Gittyperson commented 1 year ago

I was away for a while - still talking with support about this. I'll post if anything conclusive comes up.

I trust that helps clarify the problem.

I hope somebody from MEGA can provide a helpful reply that moves things forward.

Your post certainly makes a lot of sense. If this is indeed the issue here, I hope it gets fixed.

Gittyperson commented 1 year ago

@Eldaw if you still encounter this issue, try changing your MEGA password (via Web, not MEGAsync).

ramification commented 1 year ago

I had the same problem and then I realized that every _nodemodules of my projects was in sync :scream:

So... I've add the exclusion pattern node_modules and then it was way better !

Gittyperson commented 1 year ago

The issue was solved right after I changed my MEGA password via Web. MEGAsync now completes its initial scan almost instantly, with the exact same Syncs enabled.

MEGA also suggested that using the Keepass2Android app could be part of the problem (I was using it to access files in MEGA), although I could not confirm a direct relation to the issue.

Eldaw commented 1 year ago

I tried changing my MEGA password via the Web, and it has made no difference whatsoever. Starting MEGAsync, it still spends at least 10 minutes 'Scanning' every time, even though no changes have been made in any of my synced folders.

Eldaw commented 1 year ago

@mattw-mega I created a new issue with some more information concerning the issue, including some lines from the MEGAsync.log file. Please see #734

I hope somebody from MEGA can help resolve it.

Gittyperson commented 1 year ago

Re-opening this, since the problem reappeared right after I clean re-installed Windows 10 (for other reasons) on the same PC and of course MEGAsync (now at 4.7.0).

None of my previous troubleshooting steps work (logging out of all sessions, changing password).

Eldaw commented 1 year ago

@Gittyperson have you taken a look at the issue I logged, #734 ? In that issue I describe some of the entries I see in the log file called MEGAsync.log. I wonder if you see similar entries in your MEGAsync.log ? i.e. entries indicating that the MEGAsync client thinks it is in a crashed state (isCrashed true) at every startup which then causes it to delete everything from its local database and do a full scan from scratch. Take a look at that issue and let me know. I'm struggling to get a useful response from MEGA.

09/28-18:38:56.925215 37524 WARN Force reloading (isCrashed true)
09/28-18:38:56.941192 37524 WARN Deleting local cache: C:/Users/username/AppData/Local/Mega Limited/MEGAsync/megaclient_statecache12_b0VYZmZlMDlnMUkskFTxmfC6UncEBKdfALGa.db
09/28-18:38:56.942695 37524 WARN Deleting local cache: C:/Users/username/AppData/Local/Mega Limited/MEGAsync/megaclient_statecache12_rCsAAAAAAQDIzFdMHBsAAKBF333tPYNS.db
09/28-18:38:56.943721 37524 WARN Deleting local cache: C:/Users/username/AppData/Local/Mega Limited/MEGAsync/megaclient_statecache12_status_b0VYZmZlMDlnMUkskFTxmfC6UncEBKdfALGa.db

Gittyperson commented 1 year ago

Thanks for the info @Eldaw - yes, I've looked into your issue. I'm in the process of starting over and re-creating my syncs. I'm not yet to the point where the issue would usually appear. When/if it does I'll report back.

Good to see the issue has been identified. If I run into the issue, I'll try mattw-mega's suggestion.

rl-mega commented 1 year ago

Hi @Gittyperson, the issue causing a full scan at each start is fixed in v4.7.1, which will be released very soon. See my comment on @Eldaw 's issue.

Gittyperson commented 1 year ago

Thank you @rl-mega. I looked at my logs when the issue was occurring and didn't find any "isCrashed" entries.

I will soon re-enable my largest sync that was potentially the cause of the problem and post back.

Gittyperson commented 1 year ago

Haven't enabled the large sync yet - took another look at the current log file anyway. No search results with "iscrash" or "error", but "warn" gives me these:

Hundreds of lines like this pair:

09/30-23:29:06.238355 3672 DBG  Unable to process media file: nh:kAxymLrY [gfx.cpp:207]
09/30-23:29:06.238360 3672 WARN Media file processing failed for existing Node [gfx.cpp:214]

Then, near the end of MEGAsync.log:

09/30-23:30:03.170963 3672 WARN ~MegaClientAsyncQueue() joining threads [utils.cpp:2732]
09/30-23:30:03.171239 3672 WARN ~MegaClientAsyncQueue() ends [utils.cpp:2737]
09/30-23:30:03.172572 1696 INFO Request (DELETE) starting [megaapi_impl.cpp:16147]

...some DBG entries here...

09/30-23:30:03.177810 1696 WARN ~MegaClientAsyncQueue() joining threads [utils.cpp:2732]
09/30-23:30:03.177959 1696 WARN ~MegaClientAsyncQueue() ends [utils.cpp:2737]

Update: restarted MEGAsync and now I got isCrashed:

09/30-23:31:03.371056 3372 WARN Force reloading (isCrashed true)
09/30-23:31:03.381861 3372 WARN Deleting local cache: C:/Users/...MEGAsync.../megaclient_statecache12_.....Tj.db
...followed by more like this above...

09/30-23:31:04.006014 3372 DBG  Setting isCrashed true: !root || !inbox || !rubbish (account details callback)

rl-mega commented 1 year ago

@Gittyperson Thanks for reporting back. Could you please make sure that you are using the correct version (according to your timestamps v4.7.1 shouldn't have been available yet)?. To do that, either search for INFO MEGAsync is starting. Version string: in the log, or do right click/About on the tray icon.

In case you were still with v4.7.0: Please update to v4.7.1, which has now been published here: https://mega.nz/desktop (v4.7.1 is not available yet with auto-update).

In case you were using v4.7.1, or you still have problems: We need to investigate more, I'll give you instructions to contact support and we'll follow-up from there.

Notes:

after you install the new version and restart, if you were in the isCrashed loop, MEGASync will still do a full scan (but that should be the last time).
for now, changing the exclusions will still force a full scan. We are working to improve that.

Thanks for your patience.

rl-mega commented 1 year ago

@Gittyperson about the log extracts, I son't see anything of concern:

The first (from gfx.cpp) is about generating thumbnails and preview for media files (images, music files, videos...). Sometimes this fails, most probably because the libraries we use to process media files does not support it (because of format, codec...). When doing a full scan, MEGASync will try to create these previews for files that don't have them. Note: the web interface uses a different library, and in some cases will succeed in generating them where MEGASync fails (when you access the folder containing them through the browser).
The second is an extract of the nominal shutdown sequence (you closed MEGASync while it was scanning, I believe). The log level, WARN, is maybe misleading and could be INFO instead.

Gittyperson commented 1 year ago

Looks like 4.7.1 solves it, always scanning for 1-2 seconds at most. I'll reopen if it occurs again, but it's looking very good.

Thanks for your efforts and @Eldaw for persevering.

rl-mega commented 1 year ago

Happy to read that! Thanks for reporting :)

meganz / MEGAsync

Scanning for 5 minutes at startup #722