symlink support for Linux

[makomi]

seafile server v1.7.0.1
client v1.7
found no log entries related to the issue on my system

Issue Creating a symlink with "ln -s" creates a symlink on my local machine, but everybody I am sharing the link with gets a copy of the actual file instead of a symlink. I think this is related to issue #130.

Is following symlinks the wanted behavior, i.e. is this a feature and not a bug?

Why I need this feature In many cases I find it difficult to structure my data sufficiently just by sorting it into folders, because e.g. it would be easier to browse my data if pictures in one folder would also be accessible in another folder.

[killing]

Yes this is the expected behavior. Syncing the symlink (including its target) is not possible since it may not be possible to create the target folder on another client.

[makomi]

Do you think, it would be a good idea to detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library?

[blablup]

For me it would be enough if the symlinks would be synced without the files behind it. If the Target maschine do not have the files, the symlink would be dead (That is totaly ok for me). Otherwise I would have Problems in some situations where I have files wich are out of date for me.

Example: /home/me/seafile_synced_dir/

• fileA
• fileB -> /mnt/some_mount_on_a_fileserver/ProjectDirA/fileB

Now if I change my Computer and sync my "seafile_synced_dir" to the new Computer there would be no link to the file-servers file. If I change "fileB" It wouldn't be changed on the file-server and If the file on a file-server changes I wouldn't recognize it. Also I have links to dirs on servers which shouldn't be synced (In Example VM templates or ISO Files. It would be TB of Data). Or even sensitive Informations which should never be synced to any mobile device.

And at the moment I'm not shore what happens, If I change fileB on another PC (which was synced by seafile, no symlink) on my first PC. Is the symlink replaced? Would the change be written to file-server?

[makomi]

I propose the following behaviour:

Detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library i.e. external (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library respectively create a potentially broken symlink, if the user chose that option in the Seafile client settings.

[bobbaluba]

I agree with blablup. Please let the symlinks be symlinks, that's the least ambiguous solution. I have tons of projects with relative symlinks, this would cause duplication and/or infinite recursion when synced.

[blakerohde]

I too would like to see symbolic links blindly kept, regardless if the files exist or not on the other client.

[killing]

OK. Since many people are requesting this feature, we'll add an option in the next release.

[ghost]

I actually ran into this symlink problem (infinite loop)

May I advice you to :

• treat symlink as symlink and/or
• use the content of the exclude/ignore file and do not try to follow any link/folder that is excluded

Actually the daemon seems to check all the files/dirs including the ignored ones... (as the seafile.log tells about wt-monitor-linux.c(102))

[dschoepe]

Has there been any progress on this issue? common/fs-mgr.h mentions a metadata type for links (SEAF_METADATA_TYPE_LINK), but that doesn't seem to be used anywhere yet.

[ibeardslee]

I'm against following symlinks by default. I'll give an example of something that just blew up for me.

On my Ubuntu desktop (~/Desktop) I have links to things like Dropbox (840MB), Downloads (800MB) our internal network shares, my Music (18GB, on another partition and as a separate Seafile Library) and a link to ~/Seafile where a number of other Libraries are located. I also have just under 200MB of in-progress files and directories sitting on my Desktop that I thought would be good to have sync'd into Seafile.

I ran out of diskspace (started with 16GB free) in /home as Seafile tried to follow the links on the desktop, including the link to ~/Seafile which has the index cache.

Seafile shouldn't be following the symlinks as it syncs.

[johanwilfer]

Any updates on this? I would also expect a symlink to remain a symlink and not to be followed. I installed the linux clients version 4.0.6 and server 4.0.5 and they are syncing the target, not the link.

[ibeardslee]

We have our own dev looking at adding some functonality around symlinks as options in the client .. see https://github.com/haiwen/seafile/issues/130

[MurzNN]

Is there any progress about symlink support in next seafile version? Will be good to have per-library option how to handle symlink files:

• copy target file (current behavior)
• copy as symlink (new feature)

[funkyfuture]

my two cents: i rather enjoy this as a feature to aggregate folders that are in differrents paths outside a library into one library. this makes it easy to work on same fs-objects in different contexts.

[ikcalB]

@funkyfuture then, a syncing problem comes up - when linking to the same file from two or more libraries: which library to sync to or from? (the same reason, why seafile does not allow creation of libraries inside other libraries)

@killing what about the symlink support now? (Pls read above) I'd suggest to add an option (maybe even on a per library-basis) to:

• always copy files (this is the current behavior)
• copy files if the link points outside the library and preserve library-internal symlinks
• preserve symlinks in any case

[funkyfuture]

@ikcalB i don't get your argument. it will sync from and to any library that links the object in question.

[MurzNN]

Seems that Windows also have symlink support http://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/ So we can do support not only in Linux, but on Windows, Mac, Android too.

[ikcalB]

@funkyfuture I re-read your two cents and am unfortunately not able to get what you mean.

@MurzNN nice, thanks for sharing! @freeplant As MurzNN pointed out, symlinks are indeed available on all supported platforms. People who sync from linux, should be well aware of symlinks. (Or should at least be informed by the creators of the library)

What do you think about configurable options like:

• always copy files (this is the current behavior)
• copy files if the link points outside the library and preserve library-internal symlinks
• preserve symlinks in any case

[vin-c]

Hello, I've again made a test concerning this problem. In my (particular) environment, I use seafile to sync some folders of my home_dir, some of them are wine environments with symlinks to the root_fs (/). This does lead to infinite loops of symlinks as shown below. I need to sync the environments but not the root_fs.

Test case:

• create a library inside a test_dir folder
• add a symlink inside that point just one or two level up (the sync process will be in a loop) : lrwxrwxrwx test_dir/library_name/link_to_top_dir -> ../../test_dir
• add a seafile-ignore.txt excluding "link_to_top_dir/"
• sync to server

the logs are showing : repo-mgr.c(1211): Failed to stat /(...)/test_dir/library_name/link_to_top_dir/library_name/link_to_top_dir/library_name/link_to_top_dir(...) Too many levels of symbolic links

I don't care about the "right" behaviour against symlinks (follow them or not) but please make sure that at least seafile-ignore.txt work with them !

Sure if users have options for each library or globally, it will help prevent this behaviour.

Regards

Server : 4.4.6 Client : 4.3.4

Will try with client v5

[freeplant]

What client version do you use? v5.0 should have fixed the ignore problem.

[vin-c]

@freeplant : Updated client to v5.0.0, same behaviour on client side

For the record, I cleared all configuration and .seafile-data, starting from scratch with v5.0.0 and followed the test case explained before.

The client's logs are still showing the error "Too many levels of symbolic links", but now, the server library does not contain the linked folder at all. Before (client v4.x), the directory structure was created on the server and I could follow the many directories created (library_name/link_to_top_dir/library_name/link_to_top_dir/...)

This is better but I will not try with a root_fs link... as I don't know where the error is thrown and don't want to mess up my system now...

[zhanghai]

Please add a symlink-as-file option, just as how git handles this accross platforms.

See http://stackoverflow.com/a/954575/2420519

git just stores the contents of the link (i.e. the path of the file system object that it links to) in a 'blob' just like it would for a normal file.

[ikcalB]

@killing more than 2 years ago, you said that you were implementing a solution - still I cannot spot anything in the roadmap.

will you implement configurable symlink support, finally!?

1. decide if the target is inside -> A or outside -> B of that library
2. and for either case A or B add a seperate option to:
   • recreate that symbolic link, regardless whether the target exists
   • copy the contents (if possible)
   • ignore that symlink
   • leave some kind of explanatory file behind

(admittedly, the last 2 are not optimal, and might need rethinking)

[blubberdiblub]

I'm also waiting for non-broken symlink support for quite some time. Obviously, syncing and creating symlinks as actual symlinks (not the files/directories they point to) is the most preferable solution in the majority of cases. Windows can do symlinks, too. They would even be interchangable between operating systems if they are relative and you normalize them (regarding the path separators, for instance) before putting them into a seafile Library.

[killing]

Sorry we didn't have time to change the code in the past :-( I did some investigation on Dropbox's behavior. It seems Dropbox follows symlinks by default. The rationale is to support syncing a folder outside of the Dropbox folder. However, since Seafile supports syncing any folder in the system, this use case is not for Seafile. So I think it's good to at least provide an option to let the client keep the symlink. We'll still keep following symlink the default behavior, for compatibility.

[CrazyPandar]

please add keep symlink option in client settings. make it default to false for compatibility.

[deragon]

I add my voice to those who would like symlinks to be preserved. Though an global option should exist, the user should be able to override it on a per library basis. Options should be:

• Follow symlinks always (copy data) - Default
• Preserve symlinks at all cost - Symlinks or nothing; no data if platform does not support it.
• Preserve symlinks whenever possible. Follow symlinks (copy data) if platform does not support it.

In a more advance feature, if symlinks could be converted to Windows shortcuts and vice-versa, that would be great. Though, it might not be necessary... Apparently, symlinks exist in Windows (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx). I am not a Windows guy, pure Linux here. I cannot tell if these Windows symlinks are any good.

[blubberdiblub]

@deragon As far as I can tell, Windows symlinks work fine from a pure file system abstraction point of view. However, Windows Explorer was not able to cope with them very well. I'm not sure if this is still the case on Windows 8 or 10, as I haven't tried them there.

[amalaiteko]

The following paragraph is for information only as for me not everything was so clear: Seafile...

• ... will follow symlinks, even if they point at an address outside of the synced path and sync that content.
• ... has no options to choose otherwise, but adding each symlink separately into the seafile-ignore.txt file (or via script). One should add the whole path, for avoiding to not sync the original folder or file (e.g. not just "test1" but "~/MyFiles/test2/test1").
• ... will create duplicates and it will cost additional space on your disk when symlinks are followed and not ignored, no matter, if the content was already snyced in that library (by another symlink or the original path) or not.

I'd really like an option to choose what happens with symlinks, as others have already been pointing out several times already. Following symlinks can still be the default behavior, if I can change it somehow (client level, library level, or even seafile-ignore.txt-level). I like the idea that links are being copied but not followed.

Update: I updated this comment. It now has only information character as I figured out my problem.

[denisnone]

I prefer that Seafile would follow first level directory symlinks and not all other deeper symlinks.

This will give us ability to: 1) make an aggregation library that will gather some unimportant data from all over the disk without cluttering library list in our web/desktop client interface. 2) preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to break source control systems like git. It'll go crazy complaining about source changes all around.

That's why preserving symlinks is absolutely necessary.

[CrazyPandar]

at least add an option when creating library。we linux developers heavily need this feature。

On Tue, Oct 18, 2016, 23:20 denisnone notifications@github.com wrote:

I prefer that Seafile would follow first level directory symlinks and not all other deeper symlinks.

This will give us ability to: 1) make an aggregation library that will gather some unimportant data from all over the disk without cluttering library list in our web/desktop client interface. 2) preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to break source control systems like git. It'll go crazy complaining about source changes all around.

That's why preserving symlinks is absolutely necessary.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/haiwen/seafile/issues/288#issuecomment-254541479, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJevrtk3RnYJQsd-x9ePrAzUOIBN_hHks5q1OPGgaJpZM4AxyAY .

[blubberdiblub]

Not only developers are affected. All kinds of people really. Even if you don't create symlinks yourself, many programs still create symlinks on their own in their work or data directories under your $HOME path. Just think of Steam or Wine. If you want to sync those directories between 2 computers, you screw everything up because Seafile follows symlinks unconditionally.

[jcrben]

This is my number one request (and also has the second most comments).

[xarinatan]

I too would really like to see symlinks synced as symlinks. I tried to make a backup of /var and /etc on a test machine, but that ended up with infinite recursion. I'd love to make Seafile an easy backup solution for desktops and servers, but with current behavior I'll probably have to make archives to a temporary location first, which is a costly solution since it's multiple terabytes in total size. It also doesn't have the benefit of syncing individual changed files anymore since it'd be stored in an intermediate format...

My two cents on people wanting aggregate libraries, is hard links. Hard links have other implications however, so I don't know if everyone would agree with this.

[tiramiseb]

Same problem than vin-c here : I had a symlink loop inside a library that made seaf-deamon take 100% of the CPU. It took days before I found out this was because of this loop...

[blubberdiblub]

Well, I've given up on Seafile now. The issue has been here for years and nothing has been done about it. I found another synchronization solution which syncs symlinks as symlinks .

[MurzNN]

@blubberdiblub, can you post link to this solution? At now I must use rsync for syncing symlinks :(

[blubberdiblub]

I don't believe it's OK to post it here, but I can send you a mail.

[jcrben]

For those concerned, please put a thumbs on reaction on the first comment so that this shows up higher when sorting based on reactions. Right now it's only got 3 and many more commenters...

[ikcalB]

@blubberdiblub i'd want that mail too.

Even if i think, you should mention it here, since: it's a workaround

[blubberdiblub]

@ikcalB well, you don't have your mail address exposed on your github profile page. Can I use the one in your ethercat-master repository?

I'm not mentioning it here since it's not just a workaround, but a completely different product. Unless the developers tell me it's ok to mention it, I'm going to assume that my comments will be removed if I do.

[eHanseJoerg]

@blubberdiblub I would be interested in that mail, too. info (at) ehanse.de

@killing , it would be nice to hear a comment on the implementation status of this.

[CrazyPandar]

I am going to give up too. lol

Jörg R Schumacher notifications@github.com于2017年1月28日周六 上午6:01写道：

blubberdiblub, I would be interested in that mail, too. info (at) ehanse.de

@killing https://github.com/killing , it would be nice to hear a comment on the implementation status of this.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/haiwen/seafile/issues/288#issuecomment-275787028, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJevi6i61h3KK6egHA9JpG-hzo3Mhekks5rWmk7gaJpZM4AxyAY .

[xifi-kif]

I won't give up for now. Still waiting for a Good solution from the developer

[killing]

I think deeper into this feature. And I found that it requires a lot of changes to Seafile to support symbolic links natively, with similar reasons as for Dropbox: http://stackoverflow.com/questions/15254860/how-to-let-dropbox-treat-symbolic-link-as-it-is

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

[ikcalB]

@killing IMHO, this is only an excuse!!

only in this thread (not to mention the others, regarding symlinks) many possible solutions have been brought up. Let me summarise the options for you:

Let the current behavior, namely dereferencing files and having them multiple times, be option 1:

1. de-reference symlinks

-> sync to multiple, identical files

In case where symlinks HAVE to be preserved, because everything else is simply unacceptable:

2. just keep the symlinks without any checking

-> relative symlinks like ../../somefile will work, as long as the point inside the library -> relative symlinks pointing outside the library will most likely break

-> absolute symlinks will very likely break

3. keep symlinks and substitute library path

That means, if I have the top library folder in /path/to/my/lib,
and an absolute symlink is /path/to/my/lib/secondfolder/thirdfolder,
then the symlink is actually a reference within my library to ./secondfolder/thirdfolder.

On the second machine the path where the library is to be synced: /machine2/my/lib
and shall be restored to /machine2/my/lib/secondfolder/thirdfolder. Quite easy, imho

-> relative symlinks like ../../somefile will work, as long as the point inside the library -> relative symlinks pointing outside the library will most likely break

-> absolute symlinks pointing inside the library will work -> absolute symlinks pointing outside the library will very likely break

Proposed Final Solution

• Implement option 3
• The first one already exists
• (Option 2 is optional imho)
• additionally let the user choose, what shall happen with likely broken symlinks (every example above, that will likely break - you cannot check if it actually would!)
  a. keep the symlinks anyway (all above scenarios) b. fall back to Option 1 (which would dereference the symlinks, making sure files are synced - even if symlinks are not preserved then)

REMARK: That is all you can do, with reasonable effort. No magic, nothing complicated. The rest is up to the user, to ensure he understands the implications of using symlinks with seafile

EDIT: avoid confusion by splitting additional information:
HINT: 3.b) would allow, to include "library-external" directories for syncing - i.e. creating one accumulative library, instead of having dozen seperate ones.

[ikcalB]

@blubberdiblub please use the one in ethercat-master. many thanks!

[eHanseJoerg]

Here is a workaround that is ok for me: all my symlinks have the name "Link to xyz" . I simply add 'Link to *" to my seafile-ignore.txt in the folder, thats it. Not beautiful, bit I am not willing to give up on Seafile. Yet.

[amalaiteko]

Correct me if I'm wrong, but @killing didn't say that the solutions proposed aren't helpful or need to be rethought or anything. IMHO he says that the IMPLEMENTATION, the programming (of whatever solution you want regarding symbolic links), doesn't seem to be possible without changing core files and/or functions of seafile. So the proposed solution he can provide without having to change too much within seafile is to provide an option that syncs "symbolic links as files, with the destination path as contents". This solution, it seems, is, from a realistic point of view, possible, in contrary to the solutions many other people want or suggested before.

There is no need to give this post (talking about the last post from @killing ) a thumb down. I'd rather give it a thumb up for giving informations to understand why this issue hasn't been already addressed a long time ago (which I would have appreciated... a long time ago, but better late then never).

@ikcalB Not sure how "deep" your knowledge about the seafile source code is. I hope that the seafile devs already thought about several approaches before as this issues has been addressed from a lot of people already. I don't understand either why this is so hard to implement, as many approaches seem to be easy to implement, or at least within reason.

[amalaiteko]

An implementation as many people stated above is overdue, imho. Sad that seafile loses users, because they kind of missed to communicate (for a long time, maybe starting now?), why the implementation is such a big thing, but still understandable that people leave.

My workaround: I created a folder and created symlinks to every folder I want to sync. Helpful especially if you want to sync your home folder but can't sync it, because seafile-data folder is stored there. You can try to create a symlink to your home folder and use a seafile-ignore file for preventing to sync the seafile folder and other folders or symlinks. If that doensn't work, create a symlink to every folder you want to sync. For data that you don't need in your cloud but want to back it up: You can use rsync to sync those folders. rsync by default just copies the symlink and does not follow them (as expected imho). rsync as well has strong features like ssh support, you can provide an ignore file, similar to the seafile-ignore file and more.

Of course this uses another tool to bypass the "wrong" symlink behavior from seafile. By just selecting the folders you want to sync you can try to avoid folders that have a lot of symlinks that you don't need anyway in your cloud.

As always this is a personal workaround and it might not fit for you, but maybe you have some ideas about how your own workaround might look like, at least if it doesn’t include changing to another software.