symlink support for Linux

makomi commented 11 years ago

seafile server v1.7.0.1
client v1.7
found no log entries related to the issue on my system

Issue Creating a symlink with "ln -s" creates a symlink on my local machine, but everybody I am sharing the link with gets a copy of the actual file instead of a symlink. I think this is related to issue #130.

Is following symlinks the wanted behavior, i.e. is this a feature and not a bug?

Why I need this feature In many cases I find it difficult to structure my data sufficiently just by sorting it into folders, because e.g. it would be easier to browse my data if pictures in one folder would also be accessible in another folder.

xarinatan commented 7 years ago

My main reason to want symlink 'support' is to be able to make backups of my servers without having to create extensive seafile-ignore.txts that may also include configs I do want to backup.

Just syncing the symlinks 'carelessly' with the original path inside them is completely fine with me, I'll happily take that any day over the current "hang and use 100% CPU but it's ok it's backwards compatible" solution.

ikcalB commented 7 years ago

@amalaiteko I'm afraid, I have not looked into the seafile sources. Yet I feel able to argue that way, having worked with inotfy, sync algorithms and related stuff.

As I understand it, the seafile team does not see an easy all-in-one-perfect solution - which is not achieveable imho. One cannot consider any corner case in backing up & restoring.

Additional to the fact that only a minority of people actually depend on that feature (devs are probably only a fraction of all seafile users), I sense the (non)response as unwillingness to deal with the topic itself, rather than implementing a reasonable solution (which means compromise, as I tried to explain)

ikcalB commented 7 years ago

@amalaiteko when using rsync for backup, seafile is pointless. Really.

@eHanseJoerg Renaming symlinks might be viable to you - I would never ever let any software force me into that. Regardless of the fact, that countless software packages depend on symlinks and of course, their correct (unrenamed) filename

jcrben commented 7 years ago

@killing you might look at how other syncing systems handle symlinks - e.g. https://docs.syncthing.net/users/faq.html?highlight=symbolic#what-things-are-synced

amalaiteko commented 7 years ago

@ikcalB I agree with you to everything but the rsync <-> seafile statement. I use seafile not only as a backup solution, but also as a cloud service. I use rsync only for files that I don't use (often) or for files that are not so important (it was some work to set up filters to distinguish between files that are important to me and files that aren't). For my files and folders that are important to me that contain symlinks I kinda sync them carelessly as they hardly ever reach 1MB in size (or I restructured my documents so it won't sync the same files twice).

I agree with you that this solution surely is not beautiful. It works for me. It would be a big enhancement though, if I could rely on seafile for everything.

ikcalB commented 7 years ago

@amalaiteko I did not inted to criticise your setup - glad it works for you!

Not everyone is able, to accept the "degradation" of symlinks to duplicate files - i.e. symlinks are often used to choose between alternatives, the same script may be symlinked to different names,...

What's more: If I feel the need to back something up, I want this to happen automatically - which perfectly works with seafile. And because this works so well, I backup everythin that's worth it with that software. Just, that seafile currently isn't able to handle symlinks properly is a dealbreaker for me, and some others

killing commented 7 years ago

Thanks for all your feedbacks. We understand the algorithms very well. But the main difficulty of adding support for symlinks is that the old clients don't recognize symlinks in Seafile's internal metadata structure. Since Seafile clients will create metadata objects and upload them to the server, adding a metadata type that old clients don't recognize is very hard. That would cause a lot of instability. The difficulty is in implementation and architecture.

jcrben commented 7 years ago

Allow for people to break the backwards compatibility in a new version, perhaps? Figuring out how to migrate the customers old data sounds tricky, but can't people opt to wipe their history clean and import the same as someone who is starting fresh if they want?

amalaiteko commented 7 years ago

I'd even set up my seafile server and client architecture from scratch again to make this happen. For those who can't afford this, they can just stick to their setup, but without the symlink support. I like the idea.

ikcalB commented 7 years ago

@killing Thanks for clarifying!! I did not know, that is the problem you are facing.

I'd like to continue on @jcrben 's idea: This feature is not of interest for the majority of users. As soon as it is tested & working, what about:

new seafile-server installations: activate by default
upgraded seafile-server installations: do NOT activate this by default. activating this means, that all clients have to be updated to a special version.

When having this feature on the server side:

Then you could either automatically replace all standard files with symlinks on the server, so one does not lose his history.
If implementing this is too much effort, I'd happily accept, to dump my history and resync to a new library!!
like @amalaiteko said: If only a fresh install of the seafile-server is a viable solution, I'd happily accept that too!!!

HINT: making this option configurable, seems like a good idea anyhow, imho. So ppl can opt out, if they do not want to confuse their users (windows-media-only backups for non-devs come to my mind)

tiramiseb commented 7 years ago

A workaround that would please me is to replace symlinks with files which contain the symlink target as text, or even completely ignore them ; it would be a selectable option, of course. It would need modifications only on the client side and would not break backward compatibility.

ikcalB commented 7 years ago

@tiramiseb only as the last, very last resort ...!!

zhanghai commented 7 years ago

I'd like to (have an option to) handle symbolic links the same way as in git, i.e. similar to regular files.

ikcalB commented 7 years ago

@DreaminginCodeZH which is "option 2." as I called it a few posts above.

if seafile decided to implement symlink handling in any fashion, simply substituting the library path (my "option 3.") is not much of additional effort - where possible - yet plenty of convenience!

MurzNN commented 7 years ago

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

@killing, as Seafile based on Git for store data, we can use Git version of symlinks in the cloud side: http://stackoverflow.com/a/15465497 - store symlink as regular file, and mark it type as symlink in server side storage.

As I write earlier https://github.com/haiwen/seafile/issues/288#issuecomment-133414998 - all OS have native symlink support, even Windows, so there are no problem with creating and managing it on client side.

The problem is only on client side with older clients and it's libraries - we must not broke/change it sync process. Best solution is add option "Follow symlinks" to library like --copy-links option for rsync, and turn it on by default on all old libraries.

So users that want to use symlinks will can turn off this option and work normally with symlinks.

P.S. Many linux users actively use symlinks in their works, and broken symlink support in Seafile will be big "surprize" for linux users :(

pboley commented 7 years ago

Just set up Seafile today and discovered this unexpected issue for myself... I know it's not very constructive, but I would like to add my voice to those very strongly wishing for proper symbolic link support in Seafile. I understand the issues with backwards compatibility for clients, but isn't it reasonable that certain server versions (e.g. >=6) will only work with certain (newer) client versions? This unfortunately is a glaring shortcoming in an otherwise exemplary personal cloud solution.

KarlZeilhofer commented 7 years ago

we will use these lines in our seafile-ignore.txt

*/dontsync*
*/dontsync*/

*/symlink*
*/symlink*/

so every folder or file which starts with symlink or dontsync will be ignored by the seafile client.

lbr88 commented 6 years ago

I need this too. Incompatibility with older clients is to be expected when upgrading major version so what is the real issue here?

xarinatan commented 6 years ago

IMHO it's kind of crazy we've been debating symlinks for so long. Surely not treating symlinks as symlinks with a program that touts synchronisation as its primary feature counts as broken? And hanging and using 100% CPU until the enduser kills the program is absolutely insane. Recursive symlinks are part of literally every windows, mac and linux installation, and this broken behavior will cause more people issues than it will benefit others.

My solution for the problem would be simply implementing symlinks as symlinks, breaking backwards compatibility with a major version if needed (although I don't see why, older clients will just be incompatible with symlinks and cause the old, already broken behavior), and letting people who require symlinks to work as data aggregation use hard links, which the client will have to detect as normal files as that's what they essentially are to the filesystem. In my opinion, hard links are what one's supposed to use in this scenario, not letting a bug sit in the code that breaks workflow for the majority of people but allowing certain people to use it as backup 'feature' even though there's other, better ways to do that. https://imgs.xkcd.com/comics/workflow.png

Just my two cents. I'd still love but still cannot use Seafile as the backup client I was hoping it would be. I consider this is a defect in Seafile that I'm waiting for a fix on.

ikcalB commented 6 years ago

@xarinatan xD +10 for the hilarious comic!

dreua commented 6 years ago

@blubberdiblub Could you tell me your alternative via E-Mail: seafile_pls_make_symlinks@posteo.de (I just created this alias to prevent spam)

@all: Have you upvoted the first comment?

@killing : So you are saying you are not going to change anything that needs to change Seafile's internal metadata structure for ever? I guess it's hard, but I think it's worth it! Can we help in any way to make this happen anytime soon?

Simple client-only implementation proposal (opt-in)

Instead of a Symlink, just send a Text file named "seafilesymlink_not_supportedname_of_symlink" with the destination (and if needed other metadata and more comment) as contents to the server. If the client receives such a file and is capable of symlinks it can recreate the symlink (maybe with an alert to activate the symlink feature in the settings first), if it doesn't the user is informed about possible problems by the file's name and content.

amalaiteko commented 6 years ago

Since nothing changed for quite a while I finally gave up on seafile and dropped it. I'll probably go with owncloud/nextcloud for stuff that I need to have access to from different devices and different locations. Everything else will be synced with rsync. This is not quite as nice as seafile would be with the mentioned option, but imho nothing happened for so long with so much requests that I finally came to this decision.

I will still have this issue subscribed in the hope, that it will change one day. Then I can reconsider to use it again or not.

IllyaMoskvin commented 6 years ago

I have some good news! I wrote some scripts to automate symlink management:

https://github.com/IllyaMoskvin/seafile-symlink

There's one for PowerShell, and one for bash. They should be functionally identical. They will find symlinks, add them to seafile-ignore.txt, store their data either in syncable placeholder files or in a text database (configurable), and restore them therefrom.

Obviously not as good as the real thing, but I hope this helps in the meantime.

One thing I noticed while writing this tool is that the Seafile docs might be wrong about how to ignore symbolic links to directories:

If the pattern ends with a slash, it would only match a folder. In other words, "foo/" will match a folder "foo" and paths underneath it, but will not match a regular file or a symbolic link "foo".

If a pattern doesn't end with a slash or a wildcard, it would not match a folder. For example, "foo" can only match regular file "foo" or a symbolic link; while "foo/" and "foo*" match a folder and paths under it.

It appears that Seafile follows symbolic links, seeing them as normal files and directories. So if you want to ignore a symbolic link to a directory, you do need to end it with a forward-slash.

At least – that appears to be the case on Windows. I suspect that due to the whole "everything is a file" thing on nix systems, you may need to add two seafile-ignore.txt entries for directory symlinks: one with a forward-slash for Windows, and one without, for nix.

I can confirm that the tool seems to work 100% as intended on Windows, but I haven't had a chance to see if it blocks symlink syncing on other systems. I'm operating on the assumption that the Seafile Client is functionally identical for all systems in regards to ignore rules. I'm a bit burned out from writing it, so if anyone feels like testing that on Linux or macOS, please do, and let me know of any bugs you find.

benosman commented 5 years ago

@blubberdiblub - I'd be interested to find out what sync tool you are using with symlink support.

To save you from getting more requests, would you mind sharing the name of the product on here?

Already other products have been named such as dropbox, syncthing, nextcloud so another one in the mix wouldn't harm.

MurzNN commented 5 years ago

Yandex Disk have good support for symlinks.

blubberdiblub commented 5 years ago

@benosman Well, I'm using Syncthing now. Unfortunately, they removed some Symlink support under Windows some time after I started using it for reasons that I consider unjustified, but at least it still supports Symlinks unter Linux.

MurzNN commented 5 years ago

@blubberdiblub can you share link to issue about removing Symlink support under Windows?

benosman commented 5 years ago

@blubberdiblub - thank you, I ended up using Syncthing too, along with Seafile for other libraries. I do use Windows too but the symlinks are less important there.

makomi commented 5 years ago

Please discuss alternatives to Seafile via a different communication channel. This issue only addresses the missing symlink support in Seafile for Linux.

xarinatan commented 5 years ago

@makomi Syncthing is an overlay, you're supposed to use it with sync programs like Seafile (dropbox, aws, ftp, etc), so it's not offtopic here, if anything it's just a workaround for this gross flaw in Seafile.

Also tbh, considering the developer seems to not care about this huge flaw, perhaps discussing alternatives is the right thing to do here :| (Yes, I'm still very salty about it)

blubberdiblub commented 5 years ago

@MurzNN https://github.com/syncthing/syncthing/releases/tag/v0.14.26

akvadrako commented 5 years ago

This is really a necessary feature. Also I need it for macOS, not just Linux.

danwilliams commented 5 years ago

Having just installed Seafile Pro yesterday, and synced a large folder from Machine A to the brand-new Seafile server, and then to Machine B (which previously had no copy of the files), I then naturally ran a couple of checks to ensure that the data matched.

It didn't.

After investigation I of course found that the symlinks were derefenced - so all the linked-to material was copied. Ouch.

Both clients, and the server, are on Linux.

A quick search found this issue - six years old. I am incredulous that this fundamental and critical feature was not there from the start - and that it has been ignored for so long. A symlink is a file. A special file, but still a file. If you copy something else, you are in error. If I sync a folder and I have a symlink pointing to a library for instance, it may very well be correct that the library is different versions on each machine. That is not your job to resolve - if you copy the symlink, that's perfect.

On Windows, you can then either not restore the symlink, or, create a zero-length file or something.

Ultimately, without symlink support, Seafile is broken. I am uninstalling, deleting the server I created, and cancelling the Pro account. You are not having my money for this.

Unbelievable.

j0hannes commented 5 years ago

+1

xarinatan commented 5 years ago

I'm not the best at C++ but I've taken a look at the code and what could be done to fix the issue. It seems that the server does not care about what it gets sent, it treats the files as binary and stores it as blocks, so it needs to be fixed on the client. Looking at the client code, I don't see any place that specifically decides what kind of file should receive what kind of treatment, closest I can find is https://github.com/haiwen/seafile-client/blob/master/src/filebrowser/tasks.cpp which defines the functions to upload the files. Perhaps somewhere in there, or perhaps somewhere else, there could be a check added if something is a symbolic link, and if it is, craft a symlink and send it to the server, if it isn't, upload as usual. Alternatively perhaps there are more low level ways to interact with the filesystem which return an actual symlink file when referring to one, symlinks should be recognizable as normal (link) files on all platforms (windows, mac, linux, freebsd, etc), unlike hardlinks (which is basically the same as a physical file referring to an inode on disk), so it should be possible to simply copy the symlink files without any edits.

Hope this helps, I don't like being snarky and mean but man I really expected more from a well funded organization that's been pointed to a severe deficiency the better part of a decade ago.. I REALLY like seafile, I really really do, and I still use it for my photo backup of my phone and stuff alike, but there's so much other things that I can't use it for right now, like backing up my servers with versioning and deduplication (how awesome would that be with seafile!), versioning my complexer code projects that have recursive symlinks, storing temporary backups of machines that I have to fix, etc etc etc, that's all NOT POSSIBLE RIGHT NOW BECAUSE OF THIS BUG.

xarinatan commented 5 years ago

Just to summarize since I'm still running into this issue daily and I'd really like to be able to use symlinks as symlinks, please downvote/comment if you disagree, upvote if you agree:

Seafile currently treats symbolic links as hard links, causing symlinks to break/disappear when transferred and the source of the link to be transferred instead (imagine what this would do if windows .lnk files were treated like this and you'd try sync your desktop .. that's how mac/linux users feel.)
The majority of people in this thread would prefer link files to be transferred without interpretation or changes.
Some people may use it as a data aggregation feature. However this could be done with hard links instead and should not necessarily need symlinks or a specific fix to allow what they want to do (whereas treating symlinks as not-symlinks causes impassable problems).
There are some mentions from the developers about issues with the backend metadata and older clients, however people seem to agree that it's acceptable to break backwards compatibility on a major version release.
How much work it is I don't know, but I'd happily chip in dozens on a bugbounty for this. I need this to actually function for my daily work, right now I can only do some parts of it, and I'm sure I'm not the only one.

Am I missing anything? Is there anything that's holding this issue back? Is there anything I could do to get this picked up on?

xarinatan commented 5 years ago

Ohey, there DOES seem to exist a bountysource page for this, now I'm not sure if this is entirely official (there's no members on the team page) but I've gone right ahead and set a 50$ bounty on this for Seafile/haiwen/anyone who's willing to fix it to claim, I am not joking when I said I'd pay for this bugfix, and I encourage others to chip in as well. PLEASE make symlinks work as symlinks, it's by far my biggest annoyance with Seafile and constantly makes me consider moving to an alternative. https://www.bountysource.com/issues/1186791-symlink-support-for-linux :pray: :pray: :pray:

ibeardslee commented 5 years ago

I have given up on Seafile because of this. It had been looking like a tool that could do things very well, but this issue has been a deal breaker for me. I'll also unsubscribe myself from the issue so I don't continue to get bitter and twisted ;)

xarinatan commented 5 years ago

@killing @caixiangyue @cuihaikuo @freeplant @imwhatiam @LeoSirius @lins05 @llj @poetwang any one of you want to make 50$ fixing a bug? And save seafile from being discarded by lots of potential customers? It's right there up for grabs! :( :pray:

amalaiteko commented 5 years ago

@ibeardslee I already have been uninstalling seafile because of this issue like you. I also will unsubscribe now, because Seafile does not seem to care about their community (at least from this point of view). There were several ideas and compromises from the community to come towards the developers as well. None of it seems to be neither good enough nor important enough for Seafile to take care of. Good Bye!

makomi commented 5 years ago

@ibeardslee @xarinatan @amalaiteko I understand your frustration, but please stick to the topic, i.e. focus on what contributes to a solution. Announcing your disappointment does little to solve the issue while making it more difficult to find helpful information in this thread. I am convinced the Seafile devs are aware of this request and the number of people that are interested in a solution. Stay positive and use something else.

xarinatan commented 5 years ago

@makomi I understand you want to stay polite and political, and I'm trying to stay polite as well, but it's been many years, and I've invested 50$ into it when I'm the kind of person who turns dimes in the grocery store to save up on the long term, I've attempted to go through the code to attempt to point at places that would need changing, and I've spend a lot of time on working around this just so I could keep using Seafile, where others have also implemented workarounds and ways to deal with it. I feel like I've invested significant resources into this, and it'd be nice if the developers would do what they said they were going to do 6 years ago. I'm sorry if that's rude or not contributing but I find the way it's been handled rather unprofessional. :/ I'd happily give them apologies for being angry if I'd think it helps but right now I think they're just trying to ignore this ticket entirely, and I think they need to be shown that people are really caring about this issue..

makomi commented 5 years ago

@xarinatan I try to keep the prespective. Unless we pay them for their time and they accept to work on this issue, they have no obligation to fix it and we should keep this in mind during our discussion. Maybe their paying customers care more about other issues. Even as a customer, I ultimately have the option to take my attention and money elsewhere.

michaelcadilhac commented 5 years ago

I needed this feature in a recent application, so I tainted the code to reach a (hopefully) temporary solution. This only modifies the daemon and works seamlessly on my Linux machines. (Obviously, no support for Windows.)

In this fork, symlinks are never followed and are sent as regular files that contain the target (in plain text) to the server. The "flag" that says that a file is in fact a link is that its modifier (i.e. the last account that modified the file) starts with a "@". The flag is then added on sending, and removed on receiving.

Unpatched daemons should work with the repo, they would simply see the symlink as a normal file, but as long as it's not modified (that is, as long as the "modifier" is not changed), patched daemons will correctly have them as symlinks.

Have a look, let me know if it breaks something for you: https://github.com/michaelcadilhac/seafile/tree/symlink-modifier

xarinatan commented 5 years ago

@michaelcadilhac that looks really nice! Thanks for sharing that! I've skimmed over the code, it seems to just edit the daemon which is on both the server and the client end, so I thought I could get away with just recompiling it for a single client, but it seems not to work, do I have to use it on the server as well?

to be honest, while it may seem hacky, I've seen worse hacks really, it'd only clash if a client would specifically make a file with that exact naming scheme, so you could just block those kinda files or rename them for the user, they're already doing so for some other filenames AFAIK.. At very least this would enable linux (and mac?) users to keep their precious symlinks from making a huge mess on every occasion, for windows people it seems less relevant in general, although I don't see why with some more work it would work for that platform as well, would just have to use a different way of creating those links again. Although more stable solutions that use the metadata and such to properly implement it, that'd be even more amazing of course.

michaelcadilhac commented 5 years ago

@xarinatan I don't believe the daemon is on the server side. Can you give more details on what doesn't work? How did you test it?

xarinatan commented 5 years ago

@michaelcadilhac oh, I thought it was also on the server end, my bad. In that case I've probably done something wrong. I noticed afterwards that I was running an older version of the server (6.2.x), which I've upgraded to 7.0.3 as well, perhaps that changed things, so I've tried again, however I still can't get it to work.

I've installed the updated Seafile from your repo as following:

clone your git repo and enter it
run the following: ./autogen.sh ./configure make make install
run seafile client.

I'm trying again right now while first removing the copy of seafile already present though, that will probably also affect things. [Edit] Yea that's changed things, of course the GUI client used a different binary, I'm stupid. Also github's stupid for sending my comment while I wasn't done yet thanks to an accidental ctrl+enter :stuck_out_tongue_closed_eyes: still trying to get that made version to run now, I think it installed into a path that I hadn't included in my bash env yet [Edit2] Actually it's included in my path alright, just it seems to be unable to find some part and exits. Do I need to install the seafile client separately? or do I need to use different build instructions for the client to work? I'm getting this error:

[alex@Thorium seafile]$ seaf-cli 
  File "/usr/local/bin/seaf-cli", line 127
    print "%s not found in PATH. Have you installed seafile?" % prog
                                                            ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("%s not found in PATH. Have you installed seafile?" % prog)?

Thanks for your time! It'd be super awesome to have this bug squashed at least for some of the clients as my server backup system, it'd make my life a lot easier! :tada: [Edit 3] Actually staring at that some more, that looks like a plain old python syntax error, I'll open up the file and see if I can fix that myself.. [Edit 4] I changed ..env python to env python2 as it appears to be old style python. However now it seems to be unable to find the module seafile, I probably have to change it across the whole project to work or something. Perhaps they used some setup script to alias python to python2? I'm gonna try mess a bit with it and see if I can get it to work. [Edit 5] I used sed to change every instance of env python to env python2 but no luck, still the same ImportError: No module named seafile. Trying to think of other things to try, but I feel like I probably have to install some dependency or other, but to avoid installing the wrong/masking things, I'd like to ask what kinda things did you have installed alongside with it?

michaelcadilhac commented 5 years ago

@xarinatan Right, so you don't need to use a custom built seaf-cli, if that's any help: Only seaf-daemon is needed, and that's a binary, so it should be easier to run. If you insist on using a custom sea-cli, you may want to check that the python files are installed in a path python2 knows about—I'd tend to believe that your ./configure will default to python3. But the simpler option is to use your distribution's seaf-cli, and maybe remove /usr/bin/seaf-daemon so that /usr/local/bin/seaf-daemon is used instead. Good luck!

xarinatan commented 5 years ago

hey @michaelcadilhac I think I did it right this time with the help of your instructions, but unfortunately it's still not working for me :( if I do ps ax | grep seaf-daemon I see 17406 ? Sl 0:02 /usr/local/bin/seaf-daemon -c /home/alex/.ccnet -d /home/alex/Seafile/.seafile-data -w /home/alex/Seafile, so it does seem to be using the right daemon now, however if I do ln -s /etc/localtime ~/Desktop/symlinkspls, it still appears to be uploading the target file rather than making a link. At least if I check in the webinterface it still seems to generate a 3kb file named 'symlinkspls'. So unless the serverside is secretly grabbing the /etc/localtime from the server, it doesn't seem to work. But considering that the resulting uploaded file was 'last edited 12 days ago', which corresponds only to the actual filestamp of /usr/share/zoneinfo/Europe/Amsterdam, I think it's the actual target it uploaded instead of the symlink. I'll try recompile the repo again and doublecheck the source on my end when I wake up again, but first I must get some sleep after a long nightshift, so I thought I'd pass on what I've found so far. Thanks again!

michaelcadilhac commented 5 years ago

@xarinatan Sorry, that's not enough info to help you out. You could try to put some debug info in my patch and see if the conditionals ("if link") are taken. —and of course, making sure you are compiling the right branch.

haiwen / seafile

symlink support for Linux #288

Simple client-only implementation proposal (opt-in)