Syncing symbolic links as reference

EorlBruder commented 6 years ago

Currently when I add a symlink inside my Nextcloud folder it doesn't sync because "Symbolic links are not supported in syncing". I am trying to use Nextcloud as a home for my data, to have it synced between my PCs. Now I'd also like to be able to access my git-repositories (mostly) code via the directory-structure in Nextcloud but I don't want to sync the git-repositories via Nextcloud. This is why it would be really nice for me to be able syncing only the link to the repository which is on a fixed position on any system. eg: ~/Nextcloud/current_projects/fizzbuzz would point to ~/repositories/fizzbuzz. If I happen to be on a system where this git-repository isn't present I can just clone the repository there. This option could be optional (like the syncing of hidden files).

There's also a issue about this for the owncloud-client here: https://github.com/owncloud/client/issues/1440

camilasan commented 6 years ago

I see the use case, but we also would have to deal with windows links on mac, mac links on Linux... you get the drill. So between confusion and technical difficulty, and adding that most users don't even use symlinks all that much, this mostly seems a low-priority thing for me. Of course, anyone is free to work on it but he/she will have to do a full (cross-platform) solution, and that won't be easy.

camilasan commented 6 years ago

For reference: https://help.nextcloud.com/t/symbolic-link-support/220/27

EorlBruder commented 6 years ago

Would the proposed implementation in point three of this post https://help.nextcloud.com/t/symbolic-link-support/220/18?u=callegar be a viable way to solve the problem?

ferdiga commented 6 years ago

I agree with the arguments mentioned in the link above. a symbolic link is a very useful way (especially on Linux) to extend local storage capacity using other partitions and / or disk. at least on Linux the nextcloud client shouldn't exclude links as these are transparent to all Linux programs

nefelin commented 5 years ago

I understand the difficult involved but I would love to see a solution to this. Or at least an option to sync links across compatible platforms...

I use symlinks to sort a very large image collection that I use for drawing practice into smaller collections, which lets me have image sets exist in multiple collections, it is very useful and intuitive but would be even better if it synced.

m1cm1c commented 5 years ago

I see the use case, but we also would have to deal with windows links on mac, mac links on Linux... you get the drill.

Are you saying that if Nextcloud were to support symlinks, it would have to be able to convert them to *.lnk files on Windows? Converting between symlinks and *.lnk files is impossible in either direction, simply because you can't find a generic way of converting between paths on Linux and paths of Windows in either direction. Why not simply only create symlinks on the other system if it's also a Unix-like system? You can't unify Windows with the rest. They chose to be fundamentally incompatible with Linux / Mac / FreeBSD / OpenBSD / Solaris / whatever.

On the windows client, you can choose between:

doing nothing
dumping a file there that states where the link leads on a Unix-like system

I suppose you currently just dump Windows' *.lnk files as what they are on Unix-like systems, even if they are of no use there (well, they hardly are of any use on Windows too because they can't be used in the middle of paths, but that's a different topic).

fwsGonzo commented 5 years ago

No one is asking for conversion between symlinks and "windows links". There is no one-to-one comparison anyway. The dream solution here is to just not care that you found a symlink on linux and just treat it normally. If its a folder, its just a folder - extending storage. I can`t understand why the nextcloud client is hindering this to begin with, as you would have to actively disallow it.

EDIT: This is actually making it hard to use this cloud for me at all. I absolutely needed this feature to work exactly like Dropbox (and others).

piranhaphish commented 5 years ago

I agree that this is needed. I would go as far as to say that ignoring symlinks is a bug. Like was mentioned above, a symlinked file/folder can simply be treated as a normal file/folder; you have to intentionally go out of your way to avoid doing so (by using lstat() instead of stat(), for instance).

I don't think anybody is asking for the symlink itself to be represented on the cloud-side, moreso just simply recognized by the client side.

EorlBruder commented 5 years ago

I actually was asking for symlinks to be synced as "symlinks" and not to sync the content. The main reason for me is that I want to work out of my nextcloud-directory by default (I have an organization system in place there) but I do not want to sync git-repositories via nextcloud.

AncalagonTheBlack commented 5 years ago

Such a Feature would be awesome. So many times it would come handy and today again.

alexeymuranov commented 5 years ago

I've noticed some seemingly random behaviour of my nextcloud-client 2.5.0 on Ubuntu (installed from nextcloud-devs PPA) with respect to symlinked folders inside a synchronised folder.

I first noticed that files in some local folder A which was not synced itself, but was symlinked from inside a synced folder S_local seemed to have disappeared by themselves. I looked into it and found that in the local synced folder S_local i had a symlink A to the "external" folder A, but in the remote synced folder S_remote there was an actual non-empty subfolder A (with the same name) in that place. When trying to sync, the nextcloud client was simply wiping the contents in my local folder A (which was just symlinked from inside S_local), and it was not downloading anywhere the contents of the remote actual folder A.

So far i didn't manage to reproduce this behaviour with a fresh folder (but i can reproduce it in the A folder). However, i observed some other strange behaviour. For example, i have currently a symlink to a folder locally, but a real folder with the same name remotely, and their contents is only partially synchronising.

Is this related to any known bug? I am not ready to report it myself, because i cannot yet reproduce the file-eating behaviour from scratch.

alexeymuranov commented 5 years ago

After some more investigation, i decided to report the problems i observed, because when synchronisation client removes files for no reason, it is a serious problem IMO: #899.

fkbreitl commented 5 years ago

Like many others I also need to sync my folders via symlinks and request this important feature. From what I read above there are no good reasons not to support it and other cloud clients like Dropbox and tools like rsync support it, too. It is also not necessary to support all platforms at once. You can start with one platform and later others. One just needs to start and it is a rather simple thing to do. So what are the plans?

sphakka commented 5 years ago

Some ideas might come from this recent update to rclone

basos9 commented 5 years ago

EDIT 2023-10-22: This node is actually for Symlink dereferencing https://github.com/nextcloud/desktop/issues/3335 removed comment see referred issue

jcklpe commented 5 years ago

I'm actually having this problem with the client right now. It has for a long time complained to me about symlinks but now it's refusing to sync them at all, which is causing problems for me. Any suggestions on how to fix this in the short term please let me know!

rickdoesdev commented 5 years ago

On the point that is commonly being raised as a counter re crossplatform; it's worth noting that Windows/NFTS -does- support symlinks (and hardlinks, can we please preserve hardlinks while we're at it) which are read and followed same as linux. It's had this support for years and years but hardly anyone uses them. Lately MS has been desparatly trying to claw back a developer user base that was migrating away so they've made the symlinks more obvious; and seem to have removed the previously required privilege escalation required to use them, so sym and hardlinks are a 1st class citizen on windows. .lnk files should not factor into this feature request at all.

It's been many years since I've been on a mac though, so I'm not sure what proprietary nonsense apple is doing, but osx is bsd, and last I checked it also still used sym and hardlinks under the covers (ln / ln -s command at least, I assume the fundamental node created on the HDD is the same)

(Edit: for people wanting a nice explorer ui for handling links on windows; http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html is an amazing bit of software and a day one installation for me on any fresh windows setup)

jcklpe commented 5 years ago

I can confirm the moves by MS to include symlinks and hardlinks more and the link shell extension linked by rcuddy is cray invaluable. Highly recommend!

scholer commented 5 years ago

On the point that is commonly being raised as a counter re crossplatform; it's worth noting that Windows/NFTS -does- support symlinks (and hardlinks, can we please preserve hardlinks while we're at it) which are read and followed same as linux.

I really would like to emphasize this point by rcuddy. Symbolic links are super valuable as-is, which is why Microsoft has supported them for a long time. While symbolic links weren't originally supported in Windows XP, they have been supported since Vista. NTFS has full support for both symlinks and hardlinks (and junctions!). Microsoft initially made the mistake of requiring admin privileges to create symlinks, mostly out of fear of how Windows XP apps would react to symlinks. But Microsoft has now realized that symlinks has fantastic value when organizing the filesystem (just see how ubiquitous symlinks are Linux!). Microsoft has steadily been making it easier and easier for normal users to create symbolic links. (Last I checked, to allow normal users to create symlinks, you still either had to enable "developer mode", or use "Local Security Policies" to allow non-admin users to create them. But I imagine Microsoft will eventually remove this restriction - and until then, it wouldn't be too difficult to just ask NextCloud users to configure this if they want to synchronize symlinks.)

My recommendation is as follows:

Preserve symbolic links, as-is. Relative symbolic links are practically the same on all systems. That is, if I have a relative symlink pointing to ../very/deep/sub/directory/, that symlink can be preserved across all systems. Absolute symlinks are perhaps a bit trickier because of the NTFS drive-letter prefix (although Windows also support "absolute" paths without a drive letter, e.g. cd \Users to go to C:\Users if you are working on the C-drive, and D:\Users if you are on the D-drive). My recommendation would be to ignore symbolic links with a drive-letter specification, with an option to strip the drive-prefix (e.g. C:\Users is converted to /Users). Alternatively, absolute symlinks could be ignored entirely.

If people wants to "sync folders outside the NextCloud folder", NTFS junction points is the obvious solution: They are basically directory hardlinks*, but can be used to link to directories on other drives/partitions.

(*Junctions are not actually hardlinks, since making actual directory hardlinks opens a can of worms. Junction points uses NTFS "reparse points", which basically tells the software to stop and reparse the path in a specific way, depending on the type of reparse point.)

Of course, it would also be possible to just have client-side settings to configure the symlink sync behavior, e.g.

Relative symlink: ☒ Sync, as-is ☐ Sync, dereferencing link ☐ Ignore.
Absolute symlink: ☐ Sync, as-is ☐ Sync, dereferencing link ☒ Ignore.

PS: Windows' .lnk shortcut files have always been an inferior substitute for symbolic links: They must be referencing an absolute path, and they cannot be traversed/followed by e.g. the terminal/command prompt or search utilities.

Refs:

Adrien-Luxey commented 5 years ago

Hi there, just want to provide my use-case where I don't want NextCloud to follow the symlinks and synchronize their content:

I have this big "Code" directory (with 100MB+ CSVs and the like) that has been causing constant 100% CPU from NextCloud for a while. You might know that it is not trivial to exclude a folder from synchronization locally (to ask NextCloud to stop trying to upload a folder to the server). Following suggestions in the aforementioned Reddit thread, I just moved my "Code" folder outside of the synced directory, and created a symlink to its previous location. This solved my problem. So I'm glad symlinks are not followed by NextCloud, as I don't see any other easy solution to prevent uploading subfolders of the synced directory.

basos9 commented 5 years ago

Have you tried ignore files (Settings->General->Edit Ignored Files) ?

m1cm1c commented 5 years ago

@Adrien-Luxey If the directories have unique names, you can simply add their names to the ignore list @basos9 mentioned. If you have this use case several times, you can simply create an ending for these directories. For example, I sync my entire ~/Documents directory which happens to include quite a few git repos that I don't want to sync but want to exist uniquely on my different computers. So i simply created an extension .gitty for them and added *.gitty to the ignore list.

Adrien-Luxey commented 5 years ago

Thanks @basos9 and @m1cm1c for your answers. Your solution is quite swell, @m1cm1c, although it's as hacky as mine. It shows that the folder exclusion is really a missing feature (e.g. why not accept full paths in the exclusion list?). Anyway, when you end up merging #1067, it won't hurt people using symlinks to exclude folders, as it will only cause more synchronization and not content destruction. I'll switch to @m1cm1c's suggestion then.

jcklpe commented 5 years ago

@Adrien-Luxey I recently had some problems getting the ignore list set up correctly but I finally got specific folders excluded like this:

**/the-name-of-the-folder-you-want-to-exclude/**
**/if-you-need/you-can-use-multiple-folders-to/specify/**

RafalSkolasinski commented 5 years ago

Ideal solution would be to not treat symlinks in any special way and just sync them. The best example is how git handles those.

m1cm1c commented 5 years ago

@RafalSkolasinski One could argue that not treating symlinks in any special way would be how Dropbox handles them, not like git handles them: To sync the contents of whatever they are linking to.

alexeymuranov commented 5 years ago

IMO, rm and cp are not treating symlinks in any special way.

m1cm1c commented 5 years ago

@alexeymuranov I agree. However, I'd argue that the user's notion of what rm and cp are doing is quite different from what a cloud service is doing.

I think that:

The user's expectation of what rm is doing is to remove a subtree of a file system.
The user's expectation of what cp is doing is to duplicate a subtree of a file system to a different location.
The user's expectation of what a cloud service is doing is to keep in sync on different computers the data referred to in subtrees of file systems.

On the more practical side (potentially breaking the expectation of the majority of users), I see that both ways of treating symlinks have their use cases. Continuing not to sync them in any way is the worst way to proceed.

I think that the ideal solution would allow to user to specify the cloud service's behavior on a per-symlink basis. This could be done in files named .nextcloud_attributes* (the wildcard allowing for different ones of these files iterated over in asciibetical order to accumulate the total configuration). These files could also specify whether they themselves will be synced, allowing for a lot of use-case-specific flexibility.

RafalSkolasinski commented 5 years ago

How about allowing person administrating the Nextcloud (or each user) specify this config it the the web interface?

m1cm1c commented 5 years ago

Can also easily be implemented in a way that allows per-element selection of whether it should be synced as a link or as a file/directory, so it sounds good to me. 👍

It could be done as an option on each element "from here on downwards, sync links as such/as files or directories" where options chosen further down the tree are dominant. These options should apply on the elements themselves already so that one can sync symlinks as such in a directory that itself is symlinked to on one if the machines it's synced to.

wernerflamme commented 5 years ago

As someone who just sits in front of a Client that refuses to sync symlinks: the Client state is "warning". I have no option to say "ignore this error now (and in eternity), just sync what you can" so that the status is set to OK again. I can't be sure what the problem is caused by, except looking at some hundred symlink warnings.

nilsbecker commented 4 years ago

I can't be sure what the problem is caused by, except looking at some hundred symlink warnings.

Yes! This a UX bug. I was considering opening a new issue only for the current handling of this warning. Users cannot be expected to refrain from syncing directory trees that contain symlinks. In fact for many users, a large percentage of their top-level folders contains symlinks somewhere. For all those (me included) the current warning message is just noise and obscures real problems that need attention.

I have filed an issue #1363

biva commented 3 years ago

Hello, any news here? I think Nextcloud should not decide what to sync or not: users should be free to force the sync of any file. Including symbolic links. On 3.2.1 links on Windows are not synced.

At least, there should be an option in the ignored files list: by default, symbolic links could be not synced, and users could decide to remove the pattern .lnk.

I think it's super easy to implement and would solve the issue for most of the people here, without putting at risk the standard users. Wouldn't it?

[edit] I just realized that previous versions of the client allowed to sync .lst files: 3.2.1 doesn't allow it. Is it a bug or a limitation? I submitted a dedicated issue https://github.com/nextcloud/desktop/issues/3244

chylex commented 2 years ago

I'm trying to sync Windows shortcuts, which I assume are treated as symlink so it doesn't work.

The use case for me is to have something I can double-click to reach hard to remember folders on Windows, so I'd like nextcloud to sync them as-is because if it tried resolving them and syncing the contents instead of links, it would not only break this use case, but also blow up my storage...

EDIT: Not sure what's going on, I have restarted the client several times while trying to solve a problem with .gitignore files causing 500 Internal Server Error (solved by renaming them to something else and renaming them back??), and now Windows shortcuts seems to have synced and have a green checkmark next to them. Weird.

Ok, so the desktop client is complaining about symbolic links not being supported and listing some of my shortcuts, except I already had a bunch of them synced and they appear on the web. But if I copy a shortcut file in the same folder, then all other shortcut files in that folder suddenly get a yellow icon and the new one does not get synced. I don't know what's going on.

Found #3244 as the cause of this issue.

RokeJulianLockhart commented 2 years ago

@chylex, Windows "links" are merely .lnk files, but explorer.exe hides the extension. Invoke them within a text-editor via "Open With..." if you do not believe this. I expect that your problem has been caused by "http://github.com/nextcloud/desktop/issues/3244".

"http://github.com/nextcloud/desktop/pull/3057#issue-848588487" explains slightly why Nextcloud appears to believe that those are symbolic links: they are presented to software by the filesystem as symbolic links. I expect that this means that ".lnk"-files are automatically converted when accessed by anything except explorer.exe into symbolic links.

However, NTFS certainly supports symbolic links and another type of semi-symbolic link (junction points) so verify that you certainly do not refer to any of those before expecting that my judgement is correct.

recolic commented 2 years ago

I don't think we should discuss windows "links" in this issue, because windows "link" is not symbolic link at all. It's just a normal file containing a special filename and some content.

RokeJulianLockhart commented 2 years ago

@recolic, if you read what I have posted, I believe that you shall realize that @chylex's confusion is reasonable, and partially relevant, until "http://github.com/nextcloud/desktop/issues/3244" is remediated, due to how "http://github.com/nextcloud/desktop/pull/3057#issue-848588487" explains that .lnk files are treated by Nextcloud as symbolic links.

chylex commented 2 years ago

I don't know why the need to argue semantics, but .lnk files are not symlinks. They are treated as symlinks by QT, but that is not how Windows or NTFS treats them.

According to NTFS, this is a symlink:

PS F:\> $f = Get-Item testlink
PS F:\> $f.Attributes
Archive, ReparsePoint
PS F:\> $f.Linktype
SymbolicLink

This is a .lnk file:

PS F:\> $f = Get-Item shortcut.lnk
PS F:\> $f.Attributes
Archive
PS F:\> $f.Linktype
PS F:\>

I can also edit the shortcut file itself in a text editor:

obrazek

In fact, using file APIs, shortcut files are treated as regular files. Here is a difference between reading a shortcut file, and reading the file the shortcut actually points to:

> File.ReadAllBytes(@"F:\shortcut.lnk").Length
2294
> File.ReadAllBytes(@"C:\Users\Dan\Nextcloud\Portable\Audacity\AudacityPortable.exe").Length
197600

Regardless, as I already explained in the comments of whichever PR changed how Nextcloud treats .lnk files, I built a custom version of Nextcloud Desktop with that PR reverted to make .lnk syncing work again.

taminob commented 8 months ago

Hi, I think to implement this feature for all platforms at once with cross-platform synchronization is too big for one feature. Splitting it into more manageable pieces might help to make some progress.

I'd like to work on it and provide an implementation to handle Linux symlinks. I can also have a look at Windows (although I don't have any experience with the APIs there), but I don't have a MacOS installation.

I would implement Linux symlink synching without dereferencing (no server-side handling, sync as file; "git style") first since this should be in my opinion the most useful and easiest one. This feature would include an option in the Linux client (default off) to enable this behavior and otherwise keep the current behavior.

In the next steps, this can be also implemented for Windows and MacOS with another feature being then the cross-platform synchronization. Dereferencing of symlinks for each platform could then be additional three issues together with the configuration option to switch between the different behaviors.

Do you agree and would you accept such a contribution?

rico666 commented 8 months ago

I really don't get it. For over 5 years, this babble revolves around how/if to do SYMlinks or not. What do I care about Windows and MacOS?

Nextcloud runs on Linux. Period. Even if you use a "All-In-One" (https://nextcloud.com/blog/your-guide-to-the-nextcloud-all-in-one-on-windows-10-11/) install on Windows it's still Docker & WSL.

Therefore, the Nextcloud server has native support for symlinks. I have my files on Linux, I do a "ln -s" on my filesystem and see syncing action of the Nextcloud client, still the symlink does not appear even in the web frontend.

It's 2023. low hanging fruits. Minimum viable product. Show the symlink at least in the web frontend, allow clicking on it, show the destination directory - if accessible by permissions. Is that more than 1 hour of work?

basos9 commented 8 months ago

On the point that is commonly being raised as a counter re crossplatform; it's worth noting that Windows/NFTS -does- support symlinks (and hardlinks, can we please preserve hardlinks while we're at it) which are read and followed same as linux.

...

My recommendation is as follows:

Preserve symbolic links, as-is. Relative symbolic links are practically the same on all systems. That is, if I have a relative symlink pointing to ../very/deep/sub/directory/, that symlink can be preserved across all systems. Absolute symlinks are perhaps a bit trickier because of the NTFS drive-letter prefix (although Windows also support "absolute" paths without a drive letter, e.g. cd \Users to go to C:\Users if you are working on the C-drive, and D:\Users if you are on the D-drive). My recommendation would be to ignore symbolic links with a drive-letter specification, with an option to strip the drive-prefix (e.g. C:\Users is converted to /Users). Alternatively, absolute symlinks could be ignored entirely. ...

PS: Windows' .lnk shortcut files have always been an inferior substitute for symbolic links: They must be referencing an absolute path, and they cannot be traversed/followed by e.g. the terminal/command prompt or search utilities.

Refs:
* https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file

* https://www.tuxera.com/community/ntfs-3g-advanced/junction-points-and-symbolic-links/

* https://www.2brightsparks.com/resources/articles/NTFS-Hard-Links-Junctions-and-Symbolic-Links.pdf

Hello I quote the relevant reply from scholer which is a plan for this. I realize that my coment above is for the complementary feature (sync dereferenced symlink content, which is another issue https://github.com/nextcloud/desktop/issues/3335)

So focusing on the sync sym link as a reference case. There are these mentioned already.

Internal (to the sync root) links vs external ones ~~Absolute links vs relevant~~
Non unix os support
Web support

SO the MVP is kind of complex already. Since it involves changes in

the desktop client for *X oses
the server code for web (e.g. for relevant links that point outside of the sync dir, web should not resolve the link.
the desktop client for Windows (at least to ignore a symlink file until the implementation of windows links.

But I am with you ! At least I could help for the testing.

basos9 commented 8 months ago

Hi, I think to implement this feature for all platforms at once with cross-platform synchronization is too big for one feature. Splitting it into more manageable pieces might help to make some progress.

I'd like to work on it and provide an implementation to handle Linux symlinks. I can also have a look at Windows (although I don't have any experience with the APIs there), but I don't have a MacOS installation.

I would implement Linux symlink synching without dereferencing (no server-side handling, sync as file; "git style") first since this should be in my opinion the most useful and easiest one. This feature would include an option in the Linux client (default off) to enable this behavior and otherwise keep the current behavior.

In the next steps, this can be also implemented for Windows and MacOS with another feature being then the cross-platform synchronization. Dereferencing of symlinks for each platform could then be additional three issues together with the configuration option to switch between the different behaviors.

Do you agree and would you accept such a contribution?

In addition I relink the discussion to warm up This post is useful for clarification https://help.nextcloud.com/t/symbolic-link-support/220/22 and one comment of mine https://help.nextcloud.com/t/symbolic-link-support/220/32 that has some thoughts.

As I remember the sync links as is (no client dereferencing) has some things to consider. Regarding my first point above, it seems more relevant the distinction between internal (to the sync root) link targets and external ones (rather than absolute and relative).

And it seems that the simplest (minimum viable thing) would be to consider only internal links to be semantically sane (remember for many client machines, even for linux ones, absolute links may not have a meaning, or the same meaning)

Also, this option (to consider only internal links), by the way, simplifies the first implementation for windows, since we will not deal with drive letters.

taminob commented 8 months ago

@basos9 thanks for picking out the comments with possible challenges and solutions.

I think absolute links have to be handled with care, too, since the client's sync directory and the server's data directory won't match, so a conversion will be required.

SO the MVP is kind of complex already. Since it involves changes in

* the desktop client for *X oses

* the server code for web (e.g. for relevant links that point outside of the sync dir, web should not resolve the link.

* the desktop client for Windows (at least to ignore a symlink file until the implementation of windows links.

I will experiment a bit (didn't work with any nextcloud code so far), but could very well be that ignoring links that point outside of the sync dir would be easier for the MVP to remove your second bullet point. But I'll also have to check how the current web interface will react if symlinks start appearing in the data directory.

Since there seems to be no opposition, I'll start working on this issue and will open a PR with a first proposal soon.

basos9 commented 8 months ago

Good news. Lets focus on this functionality (sync links as is). I closed the issue for the client dereferencing issue to help on this direction (see the comments there for rationale).

Regarding the server, We will see but I think that he should be modified to support a new file type for symbolic links. Except if this will work out of the box, but I am not very optimistic based on the comments below.

Quoting some more relevant comments from here (4-5 years ago) to keep them as implementation reference https://help.nextcloud.com/t/symbolic-link-support/220/25

As a final clarification, Nextcloud currently implements (but disables by default for security reasons and due to the hash mismatches) the scenario you are talking about (aka client dereferencing) , but does not implement the one (never dereference) that I am talking about.

https://help.nextcloud.com/t/symbolic-link-support/220/15

symlinks have to point to other files inside your Nextcloud data dir (which to me means relative links only, and no use of ‘…’ to escape the data dir) - so the server would necessarily need to validate this when a new link is uploaded/modified on the server

https://help.nextcloud.com/t/symbolic-link-support/220/18

Recognize the existence of a symlink on the client host and store it in the server as a file. For instance, if I have a symbolic link, such as foo.txt being a symbolic link to ../bar.txt, I would be happy of seeing on the server a foo.txt file with content symlink -> ../bar.txt. Then, the other way round, when getting something that is on the server, but not on the client, recognize if the item to be synced is a file whose content matches the symlink magic, and (i) if so; and (ii) if on a platform supporting symlinks, change the file on the client into a symbolic link.

https://help.nextcloud.com/t/symbolic-link-support/220/32

Regarding type A (noderef), I can see use cases, but I think it should be limited to symbolic links targeting paths inside the sync root. To distribute links that link outside the root is not so practical, since:
The paths environment might be different from client to client (e.g. client a has folder /opt/data/, client b does not). If you need a client to modify a path outside the sync root, with type B (since you know what you are doing) you can add a symlink and only to the environments you want to (e.g. only on client a and not on b)
b. There is no cross platform meaning for e.g…" C:\data" vs “/home/data”
Also I consider this to be more difficult to implement, since you need to modify client and server to be aware of a new file type. In addition, client should consider all platforms specifically. Also you need to sanitize that links will be pointing inside the sync root.

RokeJulianLockhart commented 8 months ago

Apologies for this, but could someone summarise whether the current consensus in this issue is to synchronize symbolic (and hard?) links, junctions, and .url files without modifying them? If not, which shall be synchronized and which not? I'm a layman – I can't confidently parse some of the technical jargon utilized thus far. Thank you.

taminob commented 8 months ago

Apologies for this, but could someone summarise whether the current consensus in this issue is to synchronize symbolic (and hard?) links, junctions, and .url files without modifying them? If not, which shall be synchronized and which not? I'm a layman – I can't confidently parse some of the technical jargon utilized thus far. Thank you.

I'm currently working on the synchronization of symbolic links on Unix systems (developing on Linux, might need help to actually test it on Mac). These links will be synchronized as links to the server.

Hard links will be basically impossible to synchronize (except I miss some possibility, but they'd probably be a mess) - they are probably already getting synchronized via dereference? I don't know too much about junctions - synchronization in general should be possible, but I won't address them in the first implementation where I'll try to keep the diff as small as possible. .url files are probably already getting synchronized, aren't they (I think it's just a normal file with a special file ending)? But I won't address any Windows specific links for now since the Linux case is also the most basic one and all the different conversions to a Unix symlink (for storage in Nextcloud server) should be done in the according client as a separate feature.

In my opinion, each of these other features deserve their own issue to discuss details there - they'll probably be big enough to have a full separate PR each. Smaller PRs might help tracking down bugs and on the other side allow continuous progress step by step.

If someone else already wants to start working on something, feel free to contact me.

f1d094 commented 7 months ago

Upon review of the entire discussion chain, I would like to note that under any working circumstance the correct way to synchronize symlinks would be to not dereference them and copy them as symlinks. Under no circumstances should links be processed or followed by synchronization software. They are a basic filetype, where they point to should be irrelevant from the context of file synchronization. Does Nextcloud open .pdf documents and follow any links therein? No. Why would filetype symlink be different? If the link is a broken link, who cares? This is the user's problem and definitely not the problem of the file-sync software. If the target does not exist on "n" number of clients, the server, or even the source...this does not matter. It is a basic file. Please just sync it.

As far as the web/server-side, they should appear as defined by the web-server configuration. Under apache, for example, if "Options -FollowSymLinks" is set then symlinks do not appear/are not followed. Potential security exposures due to lax security of the administrator is a matter of training and/or proper configuration of the webserver. Again, the symlink is just a file and what it contains should not be of any concern of file-synchronization software.

The use case of wanting to easily "link-in" large amounts of content is not valid. De-duplication can be accomplished by the reverse. The content can live under the Nextcloud root and be linked to from the non-nextcloud environment without breaking a fundamental aspect of the filesystem relied on by 99.99967% of *nix users. Symlinks should never be dereferenced.

As far as systems that do not support filetype symlink, then they are not supported and should not be synced to those systems. The end. No conversion, no handling. If a windows user wants similar functionality, they can make a .lnk file of their own which will in turn come back and pollute the filesystem of the linux users but will have no impact on their functionality. Again, no processing of the actual files should be done by file synchronization software.

Files. They should be synchronized unmolested. A symlink is a filetype. Leave them alone and sync them as expected please.

De-referencing or not-sychronizing symlinks breaks the standard operation of *nix filesystems and should be treated as a bug, not a feature.

Handling of this filetype should be the simplest thing in the world. Don't over-complicate things.

@basos9

f1d094 commented 7 months ago

As an additional thought: Since Nextcloud's web folder structure isn't based directly off of the OS file structure, why can't symlinks that would be synchronized to the server simply be ignored by the web frontend and still synchronized to all clients?

This eliminates any "need to sanitize" anything.

@basos9

taminob commented 7 months ago

As an additional thought: Since Nextcloud's web folder structure isn't based directly off of the OS file structure, why can't symlinks that would be synchronized to the server simply be ignored by the web frontend and still synchronized to all clients?

This eliminates any "need to sanitize" anything.

@basos9

A good idea for the first MVP, I might fall back to this behavior if it's too much trouble to catch the invalid symlinks. For the final implementation, I think that having them also show up in the Web UI will be more user-friendly since they are otherwise hidden files on Windows and you have no way of deleting them (probably not too bad since they only require a couple of bytes, but still).

I did already some work on server and desktop and was able to upload a symlink using the single file upload (via PUT request), currently working on the bulk upload. I will create the Draft PRs (server and desktop) later today.

basos9 commented 7 months ago

Good news, I am preparing a developement environemnt to test your changes @taminob .

Regarding the comments of @f1d094 I think that all the concerns have been stated, but I will repeat some thoughts.

Yes this issue #250 is for the client to sync symbolic links as a reference. (And I agree that the client dereferencing should not be implemented (see comments in the relevant issue)).

Regarding the sanitization of link's target. There might be usability issues (and we should carefully consider security related ones, remember that nextcloud supports sharing, federation, external storages and is in general a complex system). Yes for the MVP we could skip this. But we should vote for a secure and user friendly final implementation after all. The issues:

links in folders that do not exist in different clients
absolute links that do not make sense in different oses or do not exist in different clients
We don't break all possible nextcloud deplyments
Any security issues (either in client or in server side)

As I am thinking it, if we put and opt-in (default off) switch for the desktop client (per client instance), to sync links (even with out sanitizing the link's target), this could be acceptable as a power user feature, which will not affect the majority of users. Also this would not affect existing configurations during the client upgrade for the version supporting symlinks.

Regarding the server, and more specifically for the web UI a secure representation is to present the link as a special file of type link (and not let the server's OS try to dereference the link and present it to the WEB UI) (say I upload a link afile->/etc/passwd, when I open the file in the webUI I should not see the file's content). Regarding the link's representation in the server, it could be a real link, or maybe a virtual file (only metatada) in the db. (I am curious what did the the test with the PUT request that @taminob made)

As a general note for symbolic links in *X systems (and Junctions (aka mklink) if Windows NTFS filesystems (and not .lnk files) ) are indeed special files from the OS side. This means they are not regular files (like pdf with html links inside, or windows .lnk files). This means that there are special system calls and C's std library functions to deal with them. Actually. the default for a system call open("/path/to/symlink) in a Linux system is to open the target flle (dereference the link). Witch means that we need another call (e.g. readlink) to read the link's target.

nextcloud / desktop

Syncing symbolic links as reference #250