search

meganz / MEGAsync

Easy automated syncing between your computers and your MEGA Cloud Drive
Other
1.61k stars 278 forks source link

Symlinks #159

Open hxss opened 6 years ago

hxss commented 6 years ago

Do you guys planning to add support of symlinks?

dreamcat4 commented 6 years ago

Hi, I think it's very important to backup just the symlinks themselves. As normal files, if need be. This can be done with readlink (2) or man 2 readlink.

https://stackoverflow.com/questions/4572805/is-there-a-way-to-see-the-actual-contents-of-a-symlink

You don't have to follow the symlink or anything fancy, just store and save / then restore it's information. Where it points to. Without that I have incomplete backups. There are many unix programs which store user data as symlinks. And they expect them to be there.

mforbes commented 6 years ago

With Dropbox dropping all linux support except for ext4, you will probably have a lot of new users... some fraction will be turned away by lack of symlink support as @dreamcat4 describes. Dropbox is one of the few alternatives that (semi-)supported symlinks. (Google Drive does not for example). Might be a good time to implement this.

dreamcat4 commented 6 years ago

I would also like to point out:

If you make the symlink a regular file, like a text file. + some way to flag it as a symlink just to your megasync linux/mac program internally. Then if the folder appears on a windows machine. For merging / conflict resolution - it should not be a problem.

Because then the merging / conflict resolution can take place, very much just like it already does with normal regular files.

It should not be that hard to implement this way. But if you still have reservations then please try to explain. Otherwise it's only going to frustrate users even more, not being given any explanation why you cannot do this.

KOT040188 commented 6 years ago

I subscribe to the request. in mega support for symbolic links is very necessary. Now I'm using the yandex-disk service, but I'd like to switch to mega. Because of the lack of support for symbolic links, I can not do this.

R3D9477 commented 5 years ago

+1 for this issue. Please, add symlinks support @mforbes 100% true

Tcll commented 5 years ago

and here I was looking for a way to reduce how many copy/paste edits I had to do... guess I can't just make edits to 1 folder and have those edits be applied in separate locations.

if we had symlink support, this would be possible...

if it were possible on Mega, it would cut my drive storage use by multiples, as I wouldn't be uploading the same file to multiple folders..

MurzNN commented 5 years ago

Yandex Disk and Syncthing have full support of symlinks, so Mega developers can lookup the implementation in them and do it in same way in Mega

juanfernandoe commented 5 years ago

+1

dreamcat4 commented 5 years ago

Thing is that:

Without fixing this issue, and by ignoring symlinks as megasync does. Then this software is not working right on linux. Because it's overwriting symlink files on the local folder, with real files in the cloud, which are not symlinks.

Now I have lost my symlinks in my local folder (which I rely upon). And there is not much really to say, without trawling through the logs, to find out which files were overwritten. Because there is no conflict file generated. The local symlink is just wiped out.

So to 'better support symlinks'. Could be implemented in multiple stages. The first one(s) simply being: "co-exist better with symlinks". Rather than just destroy them without any kind of warning whatsoever beforehand.

Then later on, you might think to have some pathway, to better implement the saving and conflict management of symlinks, with files in the cloud. And improve the symlink support a bit more later on.

DoctorSubtilis commented 5 years ago

+1 for this issue. Please, add symlinks support. I tried Yandex but it seems not support really symlinks, and Syncthing is only for local devices. There is a megafuse update that says to work well, but it requires a lot of packages and code...

jakubgs commented 5 years ago

I just found this issue and I'm devastated(well not really, just a bit angry). I use git-annex to manage a lot of my files, and it leverages symlinks. Without support for syncing symlinks this I cannot backup any of the files in my git-annex repos, which almost all of my important files.

If this is not fixed I will not pay another year with MEGA.

polmr commented 5 years ago

The support of symlinks is intended to be added at some point in the future. Please, stay tunned for further news :-)

KOT040188 commented 5 years ago

That's good news!

beppe9000 commented 5 years ago

Don't forget folder junctions in Windows! Those should be followed and instead nothing happens...

mekineer commented 5 years ago

Don't forget folder junctions in Windows! Those should be followed and instead nothing happens...

+1

johnmwu commented 5 years ago

+1 on this issue

eliasrg commented 4 years ago

+1. This is really important for me. I also regularly lose symlinks and find myself cleaning them up manually. If it does not get fixed, I will have to find another syncing solution.

polmr commented 4 years ago

This has been included in our roadmap. It will be included soon

jakubgs commented 4 years ago

Any news on this? My plan expires in 3 days and this issue decides whether I buy another year. Will this feature be delivered in 2020?

KOT040188 commented 4 years ago

This has long been implemented.

jakubgs commented 4 years ago

Well, it's a good practice to close issues once they are resolved...

beppe9000 commented 4 years ago

Can someone link to the commit?

beppe9000 commented 4 years ago

Support is needed for the possible link types in ~Windows~ NTFS !

image

phoenix1887 commented 4 years ago

Edit: Symlinks appear to be working as I would like for at least newer versions of Linux... Faith 50% restored in Mega... However not syncing symlink folders for older versions like Ubuntu 14.04 (even though for Dropbox they did...) Would love a fix for this if possible (considering Dropbox had it working) Also there's the second bug I point out with creating a sync folder pair from an outside folder inside of an already synced folder. You should be allowed to do this.... You can program the software to see the sync folder pairs that exist and not do an endless loop etc. This would at least allow me to do this workaround since following of symlinks is not working on these servers. Anyway. You can sense the passion and investment I have in this software and this problem... :-) Original frustrated post below (written before I knew syncing symlinks worked at least in newer versions of linux)...

I'm very frustrated that Mega doesn't support symlinks to backup my files as I need, the way Dropbox backed them up for YEARS until they JUST recently made a change in the last 6 months to ruin it (which is why I left Dropbox).

MEGA absolutely HAS to get this fixed. I want Mega to follow the symlinks and backup my files to Mega. As it is, my files are NOT getting backed up. If mega isn't supporting this it indicates bigger problems internally, and I'm going elsewhere.

It seems as though people are having different positions on the way Mega should handle symlinks. Well them abstract the choice to settings and allow the users to make the choice!

  1. Ignore symlinks (current default).
  2. Follow Symlinks.
  3. Treat symlinks as a file. (symbolic link file)

Personally I need to have #2 and nothing else is an option for me. I'm not going to be forced to restructure my files to accommodate the syncing platform. The files are on servers and the structure can't change. The sync client needs to accommodate the file structure that exists on the user's machine, not REQUIRE them to change their file structure to accommodate the sync client.

And just to illustrate the scenario I have:

I have a folder that is synced with Mega. It's also shared with other servers that sync with that folder as well. Inside the synced folder on a particular server, there is a symbolic link that points to a directory located on another folder somewhere else on the server.

Currently Mega syncs NOTHING in regards to symbolic link which points to that external directory. It only syncs the other real files in the directory that are not symbolic links...

Mega suggests the workaround of creating multiple sync pairs to sync these folders. What a terrible solution! So now I have to go and setup separate sync pairs for EVERY SINGLE one of my symlink folders I want to sync. What a nightmare hassle, error prone procedure! I have dozens!

But wait... it doesn't even work! When I try to add a sync pair for the folder I have located somewhere else that I have the symlink for inside of the already synced directory.. it gives me an error and tells me that I already have the folder synced. That's bullshit! It's probably because I'm trying to sync a folder to a location inside a an already synced folder.

So now that's 2 bugs with Mega! It should allow this also! There's no reason it shouldn't allow me to sync an outside folder to a folder within an already established sync... And don't give me a BULLSHIT reason based on some "what if" "circular syncing" edge case bullshit.... You should make the software accommodate the user don't make the user accommodate the software! Program a solution that detects for this edge case or freaking make it available for paid users only, even. Don't make the option simply unavailable because of some theoretical edge case situation that you're just to lazy to program a clever solution to detect and stop... Don't let edgecase misuse scenarios dictate the abilities of the sofware... That's what DROPBOX does.... DROPBOX: The cloudshare platform designed for CATTLE. Don't be like them!!!! MEGA is supposed to be different!

So back to my case, not only would I have to create tons of sync folder pairs for all these symlink folders in order to back them up (not going to happen) but I would also have to restructure my files my moving the folders OUTSIDE of the already synced directory I want to have them in just so your syncing agent will allow me to create the folder sync pairs. Do you see how you're requiring me to go to extreme measures to accommodate you to backup my files? RUINS MY DAY. Not going to happen with the number of folders I have to sync and how I do things. I'll sooner move to a platform that supports it correctly as DROPBOX did for over 10 years and only change it recently because they're brainless.

I'm praying that the software team at Mega hasn't lost it's head... I have a glimmer of hope remaining... Please confirm...

Mattimus

KOT040188 commented 4 years ago

This is already released!

phoenix1887 commented 4 years ago

This is already released!

Where? It's not working on any of my servers.

KOT040188 commented 4 years ago

I have ubuntu 18.04 installed. Mega has been supporting symbolic links for a long time. Version 4.2.5

phoenix1887 commented 4 years ago

Incorrect. Please spam elsewhere @KOT040188 ... This is a real issue @polmr please update... :-(

KOT040188 commented 4 years ago

Incorrect. Please spam elsewhere @KOT040188 ... This is a real issue @polmr please update... :-(

I do not spam. I describe the real situation. Support for symbolic links appeared last year. In autumn or even in summer. I do not remember exactly.

phoenix1887 commented 4 years ago

Sorry @KOT040188 It usually helps to provide a little more information or substance in a post otherwise it doesn't offer much in terms of progress and just clutters the situation...

But indeed you're correct I did just try this on an Ubuntu 18.04 machine and it is in fact working as I would like... I feel at least 50% better... Faith 50% restored... in Mega...

However it's still definitely NOT working on my servers that have Ubuntu 14.04. When I was syncing with Dropbox on these servers it was definitely working correctly and following symlinks correctly... Any reason why Mega isn't syncing symlink folders correctly on Ubuntu 14.04 but Dropbox was...?

Unfortunately upgrading isn't going to be an option anytime soon... Sigh.

beppe9000 commented 4 years ago

Incorrect. Please spam elsewhere @KOT040188 ... This is a real issue @polmr please update... :-(

I do not spam. I describe the real situation. Support for symbolic links appeared last year. In autumn or even in summer. I do not remember exactly.

Real situation: not working on all supported platforms (or this issue would be closed), so needs improvement.

Even if you confirmed symlinks are followed properly in ubuntu then this issue is still about the remaining platforms supported by MEGASync lacking such an important feature.

KOT040188 commented 4 years ago

Even if you confirmed symlinks are followed properly in ubuntu then this issue is still about the remaining platforms supported by MEGASync lacking such an important feature.

Platforms are most likely outdated. This is confirmed by the commentator above, on which 14.04

beppe9000 commented 4 years ago

Yes. So far only ubuntu is confirmed to work. What about windows and mac ?

akvadrako commented 4 years ago

Where is MEGA support for symlinks documented? Does it follow them or sync them as links?

vholer commented 4 years ago

I have evaluated MEGA last week after a year again with client in version megasync-4.2.5-1.3.x86_64 (on EL7 and similarly on Fedora 31), symlinks didn't work and were simply ignored. (As a side note, also executable modes on file level are still not supported, but it's a different issue #201 ).

FYI: A data synchronizatoin service which supports symlinks is Tresorit (but, they resolve them same as Dropbox did, not preserve). And, they also don't support file modes, but at least they don't break it on every file change.

beppe9000 commented 4 years ago

By the way ― for the courageous ones ― the code that does the file reading is in the sdk repo:

And this is the actual code that explicitly skips symlinks from synchronization (search for mIsSymLink):

polmr commented 4 years ago

Hi there. Let's see if I can clarify something: We currently don't support symlinks. They are ignored by our sync engine. There might have been some issues in past versions with symlinks partially followed (e.g: Ubuntu's 14.04 version, which don't receive upgrades anymore. We will probably issue a new version for Ubuntu 14.04 for the next release, but an OS update is recommended).

You can, however, use a folder symlink to set the local folder when you establish a synchronization (it will sync the target folder).

Supporting POSIX symlinks is in the roadmap. The support we are aiming at (at least initially) is to treat them as symlinks. Don't following them. If you syncup a symlink, when synced down it'll be a symlink, pointing to the same target. Thus, you can recreate your local structure.

phoenix1887 commented 4 years ago

@polmr

So it sounds like you do support backing up the target folder for symlinked folders now but not for files? @KOT040188 pointed out and I tested also in ubuntu 18.04 it's correctly backing up the files for my symlinked folder which is what I want. But it's broken for ubuntu 14.04 and not doing anything...

For your next point, I absolutely disagree with the behavior of just backing up the symlink as a symlink. That does ABSOLUTELY NOTHING for me me. If I have a symlink in a sync folder it's because I want that file backed up on mega, not because I want you to save a copy of my symlink for me, that's literally pointless considering you can generate a symlink with 1 command. Mega is a file backup service. Backing up symlinks does nothing for for us as symlinks is an OS tool for a user to use on their own local machine. It's a pathway to reach the actual physical file, so we're providing Mega with a pathway to reach the physical file we want you to backup....

If I want to backup a file on my server on Mega, how else am I supposed to back it up If I can't just drop a symlink to it in one of my synced folders??? I literally don't know how to even do that??? There's no way. Not with out moving the physical file to the sync folder or copying it which is absurd, that's not a valid solution.

If you seriously can't decide which way makes sense of either backing up the symlink or backing up the target file, then make a setting and ALLOW THE USER TO SELECT which way they wish for it to behave. Then you can satisfy both people! I literally can't be satisfied unless it follows symbolic links and syncs with the targets.

To this day I'm going on week 3 of a paid subscription of Mega and it wont sync my files from my local machine to my server, because it's not following the symlink directory on the server (because of the ubuntu 14.04 bug) and you're telling me you're planning to update so it wont even follow symlinks on newer versions of the OS? How do you expect me to backup my files? I'm using a modern sync client and I still have to manually scp the files from my local machine to the directory on the server to sync what a pain in the ass.

beppe9000 commented 4 years ago

@polmr what is the plan for ntfs/windows?

vassilisw commented 4 years ago

Mac OSX links not working (e.g. .framework files do not sync appropriately) Image

beppe9000 commented 4 years ago

reminder of reasonable course of action: a) allow the user to opt-in for symlink support in settings. ignore all if off. b) sync content (default for external targets) c) replicate the link itself (maybe default to this if the target is synchronized already) d) allow to set per-folder behavior

voxsoftware commented 4 years ago

Hi there. Let's see if I can clarify something: We currently don't support symlinks. They are ignored by our sync engine. There might have been some issues in past versions with symlinks partially followed (e.g: Ubuntu's 14.04 version, which don't receive upgrades anymore. We will probably issue a new version for Ubuntu 14.04 for the next release, but an OS update is recommended).

You can, however, use a folder symlink to set the local folder when you establish a synchronization (it will sync the target folder).

Supporting POSIX symlinks is in the roadmap. The support we are aiming at (at least initially) is to treat them as symlinks. Don't following them. If you syncup a symlink, when synced down it'll be a symlink, pointing to the same target. Thus, you can recreate your local structure.

I want this aproach. This is the way. Don't follow, sync just as it is: Symlink

LaurentOngaro commented 4 years ago

The issue is the following for me: I've got linux symlinks in a sub folder located into a full synchronized folders structure. When Megasync refreshes the content of this structure on my WINDOWS machine, the state of the all the symlinks is changed Then when I come back to Linux, Megasync downloads the modified files and replaces the original symlinks by binary files, that are not seen by Linux as symlinks.

Is there a way to avoid this ? Or to ignore the just the symlinks ?

mattw-mega commented 4 years ago

Hi @LaurentOngaro , which versions are you running in your Windows and Linux machines (and if not the latest, please check if the problem still occurs with the latest - 4.3.3). If it is still a problem, what change is occurring on the Windows machine that causes it to change those cloud nodes? thanks

LaurentOngaro commented 4 years ago

I use the latest version (4.3.3) on both Linux and Windows I use Windows 10 (latest) and Arch linux (up to date)

All my synchronized files are stored in the SAME folder of an NTFS formatted drive, accessible in Linux or Windows. Some of that files are linux symlinks that point to MARKDOWN files, I can create and use them on Linux without issues When I boot into Windows, all the file are automatically uploaded to the server by the megasync app, even If none has been modified. I noticed that ,now, there are shown has their pointed file format (MARKDOWN) instead of links or even junctions. Then when I boot into Linux again, all the file are automatically downloaded from the server by the megasync app. And they are not seen has symlinks anymore, but as a regular file (MARKDOWN) and their content is not a text but a binary format.

for instance, this is the content I see when opening one of these files with Sublime Text in Linux:

496e 7478 4c4e 4b01 2f00 6d00 6e00 7400
2f00 4400 2f00 5700 6500 6200 5f00 4400
6500 7600 2f00 6800 7400 6400 6f00 6300
7300 2f00 7700 7700 7700 2f00 6700 6100
6d00 6500 6300 6f00 6400 6500 7200 6200
6c00 6f00 6700 2f00 7500 7300 6500 7200
2f00 7000 6100 6700 6500 7300 2f00 3000
3500 2e00 6f00 7400 6800 6500 7200 7300
2f00 6100 6f00 7a00 2d00 6d00 6500 6d00
6500 6e00 7400 6f00 2f00 6900 7400 6500
6d00 2e00 6600 7200 2e00 6d00 6400
beppe9000 commented 4 years ago

it might be a reparse point data ?

LaurentOngaro commented 4 years ago

I think it's a kind of binary data displayed as hex values by sublime text

ragnarov commented 2 years ago

Too bad that mega doesn't support this. It would be nice to have an option that sets what to do every time. So that one could tell mega before beginning an upload, how they want mega to handle symlinks. I am of-course talking about Linux, mainly Arch Linux.

Andrew15-5 commented 1 year ago

I just started using MEGA and thought that I can synchronize a directory between 2 cloud storage — Yandex.Disk and MEGA. I was wrong. It turns out that the symbolic link of a dir that I wanted to upload to MEGA is simply ignored. If I instead copy the directory itself to a place where MEGA stores a local copy of the cloud, then the automatic synchronization kicks in just like that. I use Pop!_OS 22.04.

I opened a ticket but then discovered the links to the source code and after that — this issue.

mattw-mega commented 1 year ago

Yes, there are a lot of complications with symlinks