Call for ALPHA testers for hugely updated MEGAsync 5.0

[mattw-mega]

Hi MEGAsync technical users,

We have been updating the sync algorithm in conjunction with overcoming some internal limitations in the MEGA SDK library. The old sync algorithm and those internal limitations were co-dependent on each other, so it was not possible to update one or the other independently in smaller increments.

At the same time, we have been paying attention to the various issues reported for MEGAsync, and the updates to the sync algorithm include fixes for most of those problems, and enable fixes for the rest.

Because this is such a big update, we would like to invite keen people to try out the updated version before we release it, and report any situations for which the updated version doesn't work for them, or any bugs that they can find. The ways in which MEGAsync can be used is vast and we can't test every combination of all factors ourselves. Consider all the possible OS, filesystem, intentions for initial syncing and ongoing updates, interactions with other changes going on in the filesystem and the MEGA account simultaneously, and so on. Additionally we are introducing the long-promised update for exclusions, via the .megaignore file.

These are the main updates:

• .megaignore file for exclusions.
• Sync logic is now based on what changed since the last time a file was synced.
• This fixes the old problem where you can overwrite a file on the local side of your sync with an older version deliberately, but the sync would download the newer one from the cloud and overwrite that.
• Sync decisions are always based on comparisions of Cloud, Local, and Last synced state, rather than applying lists of changes from cloud and local.
• This enables the sync state to be kept offline, even while the MEGA account changes, which is important for the Suspended state mentioned below.
• Sync problems that are impossible for the sync algorithm to decide by itself are reported to the user in the new Stall Issues window.
• This means that you will now be alerted to things like multiple filenames that would clash when synced, so only one could be synced, or where both local and remote files changed since the sync last ran, and we don't know which is the one you want to keep.
• Sync logic runs on a separate thread, which should make the GUI much more responsive.
• Filesystem scanning and fingerprinting for syncs also runs on a separate thread, should be much quicker, especially on startup with already synced syncs.
• Fetching changes to your account since the last time MEGAsync ran should also be much faster.
• Instead of just Active and Disabled states, we now have Running, Paused, Suspended, Disabled.
• Suspended state is the most significant one. In this one, the last known sync state is kept on the the local disk in a database. Resuming that sync will continue it from where it left off. This is now the default for syncs that are not ticked in the Settings dialog. Previously those were in Disabled state (in which the sync database was deleted).
• This means that the old problems with download of files that were locally deleted while the sync was unticked are solved, when using Suspended state.
• Disabled state is still available for backward compatibility and to be able to easily discard the sync state, make some changes in the Cloud or Locally, and start the sync again from scratch.

We will post installers for Windows and Mac in this thread. They will not be officially signed, as we want to avoid anyone thinking they are anything other than ALPHA versions still under development. Please only use installers posted by myself. Additionally, there is plenty of new text that has not been translated yet. Please only test in ways that do not put your data at risk. As before, the syncs will only move your files and folders to the Rubbish folders when it thinks they should be removed from the sync, so nothing too bad can happen. (One important exception though, is syncs that are set up in an incoming share, where it is not possible to put those Nodes in the Node owner's Rubbish folder).

Details of how .megaignore rules work were posted in this other issue. Please refer there for details, and report questions or bugs specific to .megaignore in that thread: https://github.com/meganz/MEGAsync/issues/206#issuecomment-1143136772

Please also post feedback on any other issues in this thread, and we can iterate until any discovered issues are resolved. thanks

[mattw-mega]

Here are the win/mac installers (not signed) for 5.0.3: (EDIT: I've removed the link because 5.0.6 is posted further down) EDIT: I'll put the latest installers here in the first comment so they are easy to find.

EDIT: New latest version 5.0.19 here: https://mega.nz/folder/g6pUXJ7K#7XP5Ax1iQFIoasS3XyZpVw

EDIT: Prior version: 5.0.18 here: https://mega.nz/folder/8vYH0DSb#A4zdgo8oJ-Imp-5e3KLqkA

EDIT: Prior version: 5.0.17 here: https://mega.nz/folder/ZiRg0QpJ#JnP7wGhhuT2uPDlL6qYRCw

EDIT: Prior version: 5.0.15 here: https://mega.nz/folder/0vBhxKyZ#gVGybntAWNC9_Ph2RIzfLA

EDIT: Prior version: 5.0.10 here: https://mega.nz/folder/A7BhmLLR#KBaxQsPetJRQ-mV_NnaogA

EDIT: Prior version: 5.0.9 here: https://mega.nz/folder/lixl2IAA#CKzP-h7g_bGGhlA8flsgvA

EDIT: Prior version: 5.0.8 here: https://mega.nz/folder/d2ZxmS7L#V2VzWHROlmAHWN4d02uoxQ

EDIT: Prior version 5.0.7 is here: https://mega.nz/folder/srIABJ4Y#xf09858CQIzL4-oBSrR3dA

[curz46]

Looks good. Super stoked about this update. I've been dissatisfied with the sync client options from all the major cloud storage providers for years. Hoping this update can change that.

Any word on a Linux build? No worries if you are focusing on major platforms for now.

[mattw-mega]

Hi @curz46 , thanks for that. Which Linux distro are you on? I'll try to include that one in the next update.

[curz46]

@mattw-mega I'm on archlinux. I'd very much appreciate that! Again, no worries if it's a bother.

[acristoffers]

On macOS the GUI is not more responsive. I connected to a new account and let the client create the folder and then quit it. Then I added 32GB of files to it (with lots of small files) and opened MegaSync. The window is unresponsive. After some time it becomes responsive again, but it take many minutes. Also, there's the "sync issues" windows which doesn't really help much, as I cannot really do anything with the problems presented (no options) and can't even dismiss the issues.

[mattw-mega]

Hi @acristoffers , thanks very much for trying it out. I have been performance testing today, with a 1 million node up-sync on windows, and a 200k-node up-sync on Mac. It's ok for me and so I would like to know more about your case - if we can reproduce the problem we can probably fix it. I've made a short video so you can see the mac 200k node upsync (all 100 byte files for testing). This is on a 2018 mac mini, with SSD drive, and as you can see the GUI is responsive. Perhaps you could let us know your setup, a bit more about the machine you're using and maybe make a similar video (this is made in OBS Studio) - send me the link or any other info by PM please. Oh and perhaps note the CPU usage, is it maxxed out for your case? thanks https://mega.nz/file/AnInwbwa#dg-p8N9hArQLDb3d-T1mOGC6mygS4h0AtMB-660asKk

[acristoffers]

Well, whatever happened fixed itself. I was having lots of problems yesterday, but today when I sat down to record, the issue was gone. Maybe restarting the computer fixed it? Go figure. If it happens again I'll try to make a video as you said and to pin down the issue better.

[mattw-mega]

Here are installers (not signed) for 5.0.6 for those who would like to try (this is ALPHA software so please be careful). @curz46 I've included ArchLinux, please let me know if that installs ok. @acristoffers you may find that the Stall Issues are a bit easier to use now, with Refresh behaving a bit better. And there are various GUI improvements and other bugs fixed too. thanks

EDIT: removed link, as a later version is available: 5.0.7. I'll keep the latest in in the first comment, since github seems to allow ongoing editing of those. thanks

[webdev69420]

@mattw-mega that's awesome, I'll definitely try the Alpha! Do you know when this version will be production ready and released to the general public? Or even just an estimate (in 1 month? 2 months? or more like 6 months?).

Thanks again for sharing the Alpha build with us! 🥰

[sawft99]

Fixed a lot of sync issues I didn't even know I had. However, I have 3 folders with a "Hard link or Reparse Point detected" error and I am not sure what to do about it.

[mattw-mega]

Hi @sawft99 , you can add those names to your .megaignore file, so that the sync does not try to sync them anymore. Those are old Windows 98 type shortcuts to your actual Music/Pictures/Videos folders, so you should find that those continue to be synced anyway. The 'Ignore' button on the right will add those to your .megaignore file for you, if you prefer not to edit the file yourself. thanks

[sawft99]

Hi @sawft99 , you can add those names to your .megaignore file, so that the sync does not try to sync them anymore. Those are old Windows 98 type shortcuts to your actual Music/Pictures/Videos folders, so you should find that those continue to be synced anyway. The 'Ignore' button on the right will add those to your .megaignore file for you, if you prefer not to edit the file yourself. thanks

Just to be clear, this is ignoring an old Windows link of some kind but will still sync my pictures, video, music, etc folders?

[mattw-mega]

Just to be clear, this is ignoring an old Windows link of some kind but will still sync my pictures, video, music, etc folders?

Yes, that is correct. I just tried it out to double check, and those three are excluded for me, and Videos/Music/Pictures are still syncing (as they are separate actual folders). thanks

[sawft99]

Doesn't seem to stick

https://user-images.githubusercontent.com/81699231/213589431-44cc37b2-80f8-40e2-a16a-7a70d8f2b893.mp4

[mattw-mega]

Hi @sawft99 , depending on how big your sync is, you may need to wait a while for the change in the .megaignore file to be noticed and applied by the sync code that runs in the background in a separate thread. I have just tried this same exercise in a small sync, and it quickly resolved those. Clicking Refresh just reports what the sync code knows since the last time it finished a pass over the tree - and when .megaignore changes, it has to revisit the whole tree which can take a while if it's very large. Just in case that doesn't resolve after a while, you can also check the .megaignore file contents, you should see these lines at the end if you open it in a text editor:

-:Documents/My Music -:Documents/My Pictures -:Documents/My Videos

And, another option to resolve those is to delete the MyMusic/MyPictures/MyVideos filesystem entries. On my machine, I can't even get into them in Explorer, it gives some sort of access-denied error.

Great background picture, btw.

[sawft99]

@mattw-mega So I removed Mega fully and rebooted then reinstalled just to say I did that. I still had the same issue. I noticed when I hit the ignore button it isn't modifying the documents folder. It is modifying the music, video, etc folders accordingly instead. This is the before and after of the ignore file in music/videos/pictures:

Looks like it just added

-:.

at the end. The documents folder ignore file appears to have not been modified in any way.

~Once I manually went into the ignore file in my documents folder and put:~

-:/My Videos
-:/My Music
-:/My Pictures

~It began working fine. So it seems like a disconnect in the GUI ignore button doing something else other than that.~

Ok, never mind. Seems like there is some kind of delay. Here's my steps:

https://user-images.githubusercontent.com/81699231/213610502-ff1ee768-c041-44cd-8d42-c66b36843ecb.mp4

It will say it's good then it isn't.

Great background picture, btw.

Thanks!

Also are ignore files supposed to automatically create themselves?

[mattw-mega]

Hi @sawft99 , glad that one worked at last. By the way, I see you had installed 5.0.3 and that one did have a bug in the Refresh of the Sync Issues, which you ran into there and it is fixed in 5.0.6, a link for that is posted above, a bit further down from the first. Apologies for the confusion there, I thought you were using 5.0.6. I've removed the older link so others don't have the same problem.

Yes the .megaignore file is automatically created with default rules if it doesn't exist yet. If it's a pre-existing sync from before 5.0, the old rules from the Settings tab will be migrated into it.

Thanks very much for the reports, we will look into that one where the Ignore button added the wrong rule.

And, another great backdrop, nice one.

[mfm-mega]

Hi @sawft99, could you please provide us with the "My Music" file to test with this kind of Win98 compatibility shortcuts? Thank you!

[sawft99]

@mattw-mega It seems to keep resetting. If I exit and open again the error reappears. I will update though and see how it goes.

And, another great backdrop, nice one.

Thank you. 😁

@mauferreiro

My media ignore file? And before or after I update?

UPDATE:

Here is the ignore file before I updated and once I hit the ignore button: https://mega.nz/file/Ayp1iQST#w6j5kqPYDBAeib218XMCRUbccngrHqvUu7riAMzVrVg

UPDATE2:

Updating is no different. The ignore button acts the same way and the errors asking for the same folders keep appearing. Editing the documents folder ignore file gets rid of the issue but then once I restart Mega it comes right back. The upper left ignore file is from my documents folder.

https://user-images.githubusercontent.com/81699231/213784138-f9d41f29-a6e1-486a-9014-9eeb1bf65eab.mp4

[mattw-mega]

Hi @sawft99 , I've posted new installers in the first/top comment above, 5.0.7. Your case above is fixed, thanks very much for the thorough demo. I was able to replicate the problem you had there, with the stall issues being resolved after .megaignore edit, but coming back on restart, and that is fixed along with a few related things. The ignore button will add the right subpath for those shortcut folders now too, in your case you only need the plain "My Music" ones, not the ones beginning / or the ones beginning Documents/ which I had suggested when assuming you were syncing the whole user home dir. Since those are in the same folder, as .megaignore, just the name alone will match. You should find it reliable now for all those cases. thanks again

[vaoan]

Hello! I was wondering if for this build, the option to choose remote or local to all conflilcts was a thing. Everytime I change versions (I'v used the official one, 5.06 and 5.07) I have to go through the entire list of conflicts and click one by one the remote or local button. I'd rather just let the issue be solved in an easier way Thank you for this huge improvement. I've been looking forward to the .megaignore functionality

[sawft99]

@mattw-mega Awesome. Everything works as it should! Should this spot be for feature suggestions as well, somewhere else, or as a new isue? If here is good then:

• [ ] The ability to select conflicting files en masse to resolve all at once.

I had to click "Keep local copy" probably a hundred times for my old issues and ones from updating to Alpha. App and service still great so far 👍

[mattw-mega]

Hi @sawft99 and @GunB , I see you have the same feedback there, about bulk selecting local/remote for large numbers of stall issues. That makes sense, and I'll bring it up with the team and see how the GUI might work. It'll take some discussion etc so may take a while but I think we'll certainly add something to help with that. And yes, happy to take suggestions (relating specifically to the updated sync algorithm and related GUI features) in this thread. thanks

[mfm-mega]

Hi @sawft99, I'm glad to hear that with the latest version you can ignore such files (My Music, My Videos...etc). If it already works, I don't need any files from you. Thanks.

[cihe13375]

Hi! I just installed the 5.0.7 version, noticing that one of my sync folders is disabled after the upgrade. Manually enable it raises the error: "Operation on sync \<foldername> failed. Reason: Unable to retrieve sync root FSID". The old version (4.8.1) works fine. Is the support for exFAT completely dropped in 5.0? Thanks!

[mattw-mega]

Hi @cihe13375 , thanks very much for trying these early test versions. exFAT is still supported, though all the FAT filesystems have problems that make syncing difficult (primarily that they don't really have IDs for each file/folder), so not recommended. However it should be working, and I've just tested locally and set up a sync in 5.0.7 on an exFAT USB drive, no problem. Reviewing the code that leads to the report you're seeing, it could be (a) the folder on exFAT was a Reparse Point or other special type of folder (though I though exFAT did not support those), or possibly the actual folder name has different uppercase/lowercase letters than the path recorded in the sync configuration (or maybe special symbols outside normal ASCII?) Or possibly the filesystem reported an error when we tried to open that path. If you could check the logs, that should reveal this case and hopefully we can fix it. Please

• click 5 times on the account type icon in the Settings dialog, Account tab. That will turn on detailed logging and start saving it to MEGAsync.log in your desktop
• try to enable the sync again
• turn off the extra logging with the same 5 clicks (you'll see a Toast notification both times)
• send me the log by DM, or post a line or two here from it that reveal the problem.
  thanks very much

[cihe13375]

@mattw-mega Thanks for the quick response. I think the related logs are the following lines:

01/25-21:37:39.302495 14716 INFO Running sync (Two-way) "C:\Users\my_username\Download" to "/Download"
01/25-21:37:39.302572 5140 INFO Request (UNKNOWN) starting [megaapi_impl.cpp:15948]
01/25-21:37:39.303589 984 WARN You are syncing a local folder formatted with a FAT filesystem. That filesystem has deficiencies managing big files and modification times that can cause synchronization problems (e.g. when daylight saving changes), so it's strongly recommended that you only sync folders formatted with more reliable filesystems like NTFS (more information at https://help.mega.nz/megasync/syncing.html#can-i-sync-fat-fat32-partitions-under-windows. [fs.cpp:1611]
01/25-21:37:39.303693 984 DBG  Filesystem type: exFAT
01/25-21:37:39.303882 984 DBG  Adding sync: C:\Users\my_username\Download vs /Download [megaclient.cpp:13896]
**** some other stuff ****
01/25-21:37:39.304985 984 INFO Filesystem type: 0000008F7D3FD8A0 [fs.cpp:1230]
01/25-21:37:39.304988 984 INFO Filesystem IDs are stable: 0 [sync.cpp:665]
01/25-21:37:39.305039 984 DBG  Excluded: C:\Users\my_username\Download   Attributes: 1041 [fs.cpp:611]
01/25-21:37:39.305040 984 ERR  Could not open sync root folder: C:\Users\my_username\Download [sync.cpp:703]
01/25-21:37:39.305041 984 ERR  Could not determine fsid of sync root folder. [sync.cpp:3398]
**** some other stuff ****
01/25-21:37:39.390624 14716 ERR  Sync "Download" Path: C:\Users\my_username\Download disabled: 0

The C:\Users\my_username\Download here is an NTFS folder where an exFAT partition is mounted in (using the Disk Management tool in Windows). The exFAT partition has Bitlocker to Go enabled, if that's relevant (it's already unlocked before trying to sync).

[mattw-mega]

Hi @cihe13375, thanks very much for checking and finding that result in the log. And for explaining the exFAT mounted in NTFS situation. That is indeed the issue here. The line about Excluded and Attributes:1041 show that the path is a Reparse Point which makes sense with the drive mounted there. Reparse points can be a huge variety of other things as well though, and MEGA's code base usually avoids opening those as you can see at WinFileAccess::skipattributes in our public code. However, historically we did allow the sync to start, even though the attempt to open the folder itself and get its ID at that path fails. A side effect of that is that the sync database was never created either, so that scenario would have a problem with things like forgetting the state between runs, and perhaps downloading files that had been locally deleted in the meantime.

We should support it though since the user is warranting that path as suitable for opening, despite it being a Reparse Point. And because historically this scenario would run the sync. I've updated our ALPHA code to let the sync continue in this circumstance, bypassing the attribute check (just for the sync root) so we can in fact get the ID of that folder. That allows the ID to be double checked on the next run, for example, to prevent accidents. It also opens the sync database now, so you'll have proper state recorded inbetween the times that MEGAsync runs. There may be other minor bug fixes and changes as this is also based off our internal develop branches, so please use with caution.

I've posted it above in the first comment, installer 5.0.8. Please let me know how you get on thanks

[cihe13375]

@mattw-mega Thanks for the update! The syncing is working now. A suggestion I came up when trying to resolve conflicts: I think the information provided to user for resolving conflicts is too little. Currently only the filename and size are given for file version conflicts (server and client have different file content in initial syncing), and only filenames are given for filename conflicts. Sometimes it's hard to decide "which one I want to save" with such limited information (e.g. should I save "10.jpg", or "10.jpg" (filename conflict with the same name on appearance), or both?). I think the following features would be helpful, in descending order of importance: (1) display the last modified time of conflicting files; (2) for filename conflicts, display an extra hint if both files have exactly same content (same hash in practice maybe); (3) allowing preview (i.e. download to a temp folder and open it) of conflicting files With that saying, I think the current logic of file conflict resolve is already much better than older versions and some other file sync services. Thanks for all the hard work!

[Perkolator]

Yes the .megaignore file is automatically created with default rules if it doesn't exist yet. If it's a pre-existing sync from before 5.0, the old rules from the Settings tab will be migrated into it.

Is the default ignore rules in settings going to be kept or is that feature going away? Hopefully not. I'd like to set default ignore rules, and then, when needed, create specific custom ignore rules with .megaignore file.

And why is the default ignore rules imported to a .megaignore file ONLY if the the sync is from an older version? I don't understand. AND, why the need to import global ignore rules at all (if that feature is not going away) to a .megaignore file? Sounds confusing.

[mattw-mega]

Hi @cihe13375, thanks again for trying it, alerting us to this case, and for the feedback. I'll bring those things up with the GUI team, I think probably (1) and (2) at least are easily doable. It's not very obvious in the stall items, but if you open them up then you can click on the local or remote paths to open those in MEGA or in the local file browser (eg Explorer) so you can have a look at all the remote and local state that way. I do agree about making it even more convenient for the common cases too. thanks again.

[mattw-mega]

Hi @Perkolator, yes, those exclusion rules will no longer be present in the Settings dialog. There are several reasons for removing that GUI element:

• They were not defaults, but the rules applying to all syncs. If we leave them there, many/most people will try to keep using them that way, but changes will not affect their existing syncs.
• Keeping those as active rules and also supporting the .megaignore files (running both sets of rules effectively) was considered but was just too complicated in the end.
• We hope that the default .megaignore file (which are the rules you would see in Settings on a fresh install) are quite good. If you have suggestions for other default rules we would be keen to hear those and consider them?
• If you prefer to keep your own custom set of default rules, you can easily keep those in a .megaignore file outside of a sync, and copy that file into the root of a new sync before you configure it. In future we may offer something in the GUI to remind you or track a preferred default .megaignore file, something like that. But for a first release, we're trying to not make too many GUI changes.
• About only importing the old Settings rules for pre-existing syncs - this is just about the upgrade to 5.0 step, backward compatibility and keeping the pre-existing syncs working the same way they did before. For new syncs, you can copy the .megaignore from an existing sync to give it the same set of rules.

I hope that helps to explain the plans, and that your use cases are still covered by copying and/or editing .megaignore files. I think you'll find the new system is more flexible and powerful than before. thanks

[Perkolator]

For me it sounds that the ignore system is getting a lot more complicated to the users. 1. Only using ignore files, 2. and every sync needs a new file, 3. no global GUI setting to ignore for all syncs. And the ignore files don't seem to support creating an ignore file per folder if needed (one ignore file per sync in root folder of the sync, that's the new system, right?). For me it really sounds more complicated to the users, and the new feature is not even designed to have more power (only letting different rules per sync).

For me the best system would be:

1) Global ignore in GUI that affects all syncs. 2) .megaignore file that would have options: 1) An only_this_folder setting/line, which obviously would mean that rules in this file apply only to the same folder, not to subfolders. 2) Without only_this_folder setting/line in the file, the rules would apply to the same folder & its subfolders. 3) An ignore_global_rules setting/line would only use the rules set in the file. 4) Without the ignore_global_rules setting/line, global ignore rules would also apply, but rules in the file would trump global ones if in conflict.

EDIT: And that a .megaignore file could be placed in any subfolder of a sync, and different folders could have different ignore files.

That way there would be an easy way to add global rules in GUI for normal (/all) users, and a way to control the ignores more closely, if needed, for power users.

[mattw-mega]

Hi @Perkolator,

thanks for the suggestions, we certainly want to support all our users. You will be pleased to know that (i) and (ii) and putting extra .megaignore files in subfolders are already supported. Please have a read of this ticket which explains the .megaignore file format, and how you can set rules in one folder, and override them in particular subfolders, and have rules that only apply within the folder, or which also apply to subfolders (except where those are in turn overridden in sub-subfolders). https://github.com/meganz/MEGAsync/issues/206#issuecomment-1143136772

Another consideration about the global rules on one particular machine, that I forgot to mention, is that many users sync a MEGA folder to multiple machines. We have had many requests for a way to copy the rules from one machine and make them the same on another. The .megaignore file achieves that, because it is synced to the other machine too, just the same as all the other files in the sync. If we have a separate set of global rules on each machine, then we still have the problem where the rules on each machine may not be the same for different syncs of the same folder, resulting in confusion, support requests and so on. If there is just one system, it's much easier to explain and check.

I would like to understand more about your situation though, in case I'm missing something. Would you mind sharing a bit about why you want to be able to make exclusion changes across all syncs at once? Is there a simple example you can give to make it obvious, please? I know from the various complaints that have come in over the years, some people have set up many syncs where they would prefer to have set up just one, specifically because the old global exclusion rules were not very flexible. Perhaps you are in a similar situation. It should be much easier (or at least possible) now to set up a single larger sync, and control which subfolders are included or excluded from that.

thanks

[cihe13375]

After using for two days I noticed another issue. If a sync folder becomes temporarily unavailable (e.g. an external drive is pulled out), that folder automatically becomes "suspended" (which is good). The problem is that, the sync is not automatically re-activated after the folder becomes available again. This is an issue because some drives (TF card in my case) are inherently unstable, and it's common for them to, for example, get offline for 2 seconds and then get online again. From my experience, this is especially likely to happen when the system is recovering from sleep/hibernate states, and I have to manually re-enable the sync every time it happens.

This issue also exists in 4.8.1, but at least it shows a notification when such thing happens. The 5.0.8 version doesn't show such notification, and I have to manually right click MegaSync icon --- settings --- sync to see if I have to re-enable them. Anyway I hope this issue could be fully resolved as stated above (instead of only throwing notifications). It worked pretty well when I was using another file sync service.

[Perkolator]

@mattw-mega

Thanks for the clarification. Why I want to use global rules? I use linux and I have to create several different syncs because obviously I can't select my /home/username/ folder for to sync, it contains so much more than just my personal files. The size of my home folder is ~75GB, which for example has VirtualBox images & snapshots under .config/ folder, while my personal files that I want to sync take only ~3GB.

When a sync is started, a default .megaignore file will be added in the sync root folder, if one does not exist yet.

Is there going to be a GUI for these "default" rules? What if some update to the mega client changes these default rules, will they be propagated to the already set .megaignore files or only after one is automatically created after a new sync is set up?

---

If not already done, make the sync engine find .megaignore files & sync them always first (+apply the rules) before anything else gets synced?

---

If a sync folder becomes temporarily unavailable (e.g. an external drive is pulled out), that folder automatically becomes "suspended" (which is good). The problem is that, the sync is not automatically re-activated after the folder becomes available again. This is an issue because some drives (TF card in my case) are inherently unstable, and it's common for them to, for example, get offline for 2 seconds and then get online again.

The same problem probably happens with SMB/CIFS network shares. I haven't tried the test version, but with the current stable it became quickly clear that it's practically impossible to sync anything from a network share. They were always put to unsynced state when not available and when again available to system/mega client, they were never automatically enabled. It was tiring to manually enable them and then waiting for hours to full rescan them (large syncs).

I understand that with the old system there was no other option. But with the new client engine those network shares could be put to "suspended" (or "paused"?) state automatically and once they are available again, automatically enabled again (without full rescan). Real world scenario example; laptop user, home network NAS shares, laptop taken frequently to other location where the local NAS shares are not available.

Note that the network shares in linux can be mounted anywhere in the system, for example I have mounted them under /home/username/.smb/ folder. (in case you think of trying to "identify" share folders somehow based on where they are)

[curz46]

@Perkolator I'm a bit confused about your use case, and why it requires a global ruleset. Why couldn't you use the .megaignore file to exclude/include folders from your home directory as required? I believe it has a wildcard function, so you could just ignore everything and add folders back as needed. I do something similar for my dotfiles repository. If you really want several different syncs, you could create one ~/.megaignore (out of sync) and hard link every root ignore file to it.

I think it could be within scope for the megasync client to handle when synced mount points that may/may not be present. Of course, on Linux you could just restart the megasync client whenever a drive reconnects with a simple udev rule. It should hardly be rescanning them though!

I would certainly hope updates to the .megaignore file defaults do not get propagated to existing ignore files, in the same way I hope MEGA made no adjustments to my existing sync rules when they were in the settings menu. Minor updates should not change behaviour without explicit consent from the user. I don't suppose these defaults should change often anyway.

--

@mattw-mega sorry, it's taken me ages to get to trying this out! Could you build another archlinux image for the latest version? Thanks!

[Perkolator]

Why couldn't you use the .megaignore file to exclude/include folders from your home directory as required? I believe it has a wildcard function, so you could just ignore everything and add folders back as needed.

I don't want to put trust to that. I rather set all syncs per folder and then edit all the .megaignore files.. if there's no other global rule system with the new client. Safer. Currently I'm adding a .~lock.* rule (LibreOffice creates these when a file is open in it) and deleting the .* rule.

I think it could be within scope for the megasync client to handle when synced mount points that may/may not be present. Of course, on Linux you could just restart the megasync client whenever a drive reconnects with a simple udev rule. It should hardly be rescanning them though!

This goes beyond my knowledge of linux. Still somewhat new & learning. Although the "restarting client automatically" doesn't sound so good at first thought, especially if that starts a rescan. What do you mean by "drive reconnects"? There's no drives in linux. I just would like to see that mega client would detect when a synced folder is not available and put it in suspend (or paused?), and then when it detects (10 min check interval? or something like that) that it is accessible again, then it would automatically enable it again, without activating a rescan. I think same would work for USB drives/memory cards/etc. too?

[sawft99]

Why couldn't you use the .megaignore file to exclude/include folders from your home directory as required? I believe it has a wildcard function, so you could just ignore everything and add folders back as needed.

I don't want to put trust to that. I rather set all syncs per folder and then edit all the .megaignore files.. if there's no other global rule system with the new client. Safer. Currently I'm adding a .~lock.* rule (LibreOffice creates these when a file is open in it) and deleting the .* rule.

I think it could be within scope for the megasync client to handle when synced mount points that may/may not be present. Of course, on Linux you could just restart the megasync client whenever a drive reconnects with a simple udev rule. It should hardly be rescanning them though!

This goes beyond my knowledge of linux. Still somewhat new & learning. Although the "restarting client automatically" doesn't sound so good at first thought, especially if that starts a rescan. What do you mean by "drive reconnects"? There's no drives in linux. I just would like to see that mega client would detect when a synced folder is not available and put it in suspend (or paused?), and then when it detects (10 min check interval? or something like that) that it is accessible again, then it would automatically enable it again, without activating a rescan. I think same would work for USB drives/memory cards/etc. too?

Even though it is a "global" file, which really only means the root of the folder you're syncing, you could specify a specific folder in that couldn't you? Unless I am misunderstanding. But if you have a particular extension/pattern in a folder you can specify it specifically in the mega ignore file. So if you wanted .lock files to work everywhere else except your documents folder you could make a rule in the ignore file for something like "mystuff/docs/*.lock"

[Perkolator]

@sawft99 I'm sorry but I don't understand what you mean. The bottom line here is that I'd like to see a way to apply global ignore rules to all syncs. That's all. If that doesn't get implemented, then I have to create/edit ignore files (in the root folders) for all my syncs. In no circumstance I won't be adding my whole linux home folder as a single sync and then adding relevant rules to the ignore file to add only certain folders to be synced.. that will not happen. I've learned over the years to try to lessen the impact in case something goes wrong, and something could go wrong with mega client, or with the ignore file(s).

[curz46]

@Perkolator The standard Linux way to prevent a worst case scenario is permissions. You'd run megaclient as a separate user, call it mega, and only provide it access to certain folders like chgrp -R mega ~/abc && chmod -R g+rwx ~/abc. That said I think it'd be harmless to introduce a global ruleset... maybe it could work something like this:

• /etc/mega/ignore or ~/.config/mega/ignore on Linux, and equivalent sensible locations on other OSes. Exact same syntax and semantics as the .megaignore file.
• Interpreted as if all root sync directories had another ruleset, but the global one is always evaluated first. Root sync directories can override with inclusion rules.

It's really just a convenience feature but it would address your use case. No GUI necessary, which the mega team seem keen to avoid. cc: @mattw-mega another idea to go in your backlog. 😅

--

Edit in light of @mattw-mega's message - an obvious or not-so-obvious problem with that proposal is that it's non-intuitive how it would sync between devices. Likely, it wouldn't. So in retrospect it's probably not a great idea. It seems like there is a trade off here between keeping only the .megaignore file - ditching the GUI, and conveniently providing a way to ignore across multiple sync roots. I think I would favour the simplicity and universal nature of the .megaignore file and call @Perkolator's situation an edge case (I wasn't even aware you could sync folders that aren't in the same directory.)

[mattw-mega]

Hi @cihe13375 , @Perkolator, @curz46, @sawft99, thanks for all the testing and comments. I'm on vacation this week which is why I haven't replied yet - I'll be back next week and get back into it properly. Thanks for your patience. Oh, and @curz46 is correct, one of the motivations for the .megaignore system is that it can be used in both MEGAsync and MEGAcmd. If they are syncing the same MEGA folders to different devices, they will still be using the same rules on both devices for that sync. Anyway, back soon. thanks

[mattw-mega]

Hi @cihe13375 , thanks for your comments about re-enabling syncs automatically if the path they are synced at went away and re-appeared. This is something that is properly possible now that we have the Suspended state, whereas previously starting such a sync was like starting it from the beginning with no state, which often produced results that the users didn't expect or want. Now we have the state properly recorded in Suspended state, and it can figure out the net changes vs MEGA and vs the local filesystem, and what changes to apply on re-enablement. I've been working on that automatic re-enablement today, and it's looking promising. This will also address another common complaint, about syncs on drives that are mounted late in the machine startup process.

Hi @Perkolator , thanks for your comment about making a single change to apply an exclusion change to all syncs. If you had syncs on multiple machines, would you want them all to be affected across all machines? And about the default rules for a new sync - these would be the same defaults that have been in MEGAsync for many years, without any changes. Once copied into a sync though, we would not modify those as presumably the user is happy with how the sync is working already, and we don't want to break anything. Yes .megaignore files are downloaded first, if there is one in the remote folder and there isn't one in the local folder. This ensures the rules are in place before we sync any further down. While a .megaignore file is present locally, those are the rules that apply until either the user changes it locally or an updated version is downloaded by the sync itself. Thanks for the comments about network drives that cause syncs to become disabled - the changes mentioned above for cihe13375's case will help with that. And thanks for the example of the .~lock. rule, that makes sense and we'll consider how something like that might be managed. The . rule does avoid many of those special case files. Would you consider keeping the .* exclusion but then adding an include rule for a subset of those files?

Hi @curz46 , thanks for the discussion points about how .megaignore can be used. Rest assured, MEGAsync will not edit your already existing .megaignore files, as that may change what it does in a way that the user doesn't want. If it's working ok then we shouldn't change it. I'll try to include an Arch Linux build in the next one.

thanks All

[Perkolator]

If you had syncs on multiple machines, would you want them all to be affected across all machines?

I don't so I don't have a real world experience of what I would do, though quickly thinking, the most logical thing would be to have same rules.

And thanks for the example of the .~lock. rule, that makes sense and we'll consider how something like that might be managed. The . rule does avoid many of those special case files. Would you consider keeping the .* exclusion but then adding an include rule for a subset of those files?

I've to say that I don't understand what you're saying here. Why would you want to separately manage a .~lock.* exclude rule? And include what rule for a subset of "those files"? I can't keep the default .* exclude rule, I've files/folders that start with a dot that needs to be synced.

[mattw-mega]

Hi @cihe13375 , @Perkolator , and others who maybe interested - the link to new version 5.0.9 is posted now in the first comment above. It will auto-resume syncs that were Suspended due to their local path disappearing. That should take care of networked drives that are late to mount on startup, or USBs that are removed and re-inserted (however, there is a complication for that case on Mac Ventura at least, which is now reporting FAT filesystems as "lifs", and the file IDs (inode equivalent) change unpredictably, even within a single session. MEGAsync will refuse to try to sync those as it will think that files/folders are moved when they are not, and cause a mess). @curz46 - sorry but I don't have an Arch Linux build for that one yet, hopefully next week. thanks

[Perkolator]

That should take care of networked drives that are late to mount on startup

Is this the only use case where the auto-resume for network shares will work? Is it going to work on a situation like this: laptop (linux), at home network shares (on a NAS) available, laptop put to sleep or hibernate (no reboot!), laptop taken to work/etc., woken from sleep/hibernate, network shares are not available (this is where mega would put the network share syncs on suspend), laptop again put to sleep/hibernate, back at home, laptop wakes from sleep/hibernate, network shares become available again (here mega would auto-resume the syncs of the network shares).

You say "drives", it's just an expression for network shares, right? I mean, in linux there are no drives, only a network share mapped to some folder (I guess I'm just worried that you're just looking for a "network drive", and not some folder that is not accessible).

[mattw-mega]

Hi @Perkolator , yes that case should be covered too. The general mechanism is that if a sync was Suspended due to its local synced path disappearing, and that path comes back, then the sync can be auto-resumed. Please let me know if you see any problems with it. thanks

[Perkolator]

Please let me know if you see any problems with it.

I'll post here or open another issue. After the stable has been released, can't test these alphas unfortunately.

[mattw-mega]

Here are the answers to some questions submitted via our support team, which may be useful for others to know also:

• Yes the .megaignore is meant to be seen in transfers. This is how rules are shared between syncs of the same MEGA folder on different machines, and those people who use syncs that way will want to see and confirm those.
• If you want no exclusion rules at all for the sync, leave an empty .megaignore file. If one is not present, the sync will add a default one the next time it starts.
• GUI editors for rules like this are notoriously difficult to design, code, and use. For a first release at least, we are relying on text editing mostly. Some stall items can auto-add particular paths to it at the press of a button.
• Exclusion by size is still supported, please check this issue for the syntax: https://github.com/meganz/MEGAsync/issues/206#issuecomment-1143136772
• Force a full scan is still present, but on the Syncs tab (and does not require restarting the app anymore). Quick scan just checks directory listings and metadata such as file size and modfied date. Deep scan goes a bit further, checking the actual file data by fingerprinting it again, which is considerably slower (but more accurate).

[mattw-mega]

Hi @curz46 , a colleague has built this version for Arch Linux, and the .pkg.tar.zst file is in the 5.0.9 link above. Please test with caution as I have not tried this one myself. thank you