detect and report symlinks from the server

[moscicki]

In our storage server we have symlinks. Users can create them directly on the storage (sometimes unwillingly). For example, someone started emacs which creates a lock file which is a symlink (which is broken by definition):

.#file.txt# -> user@host.pid

We need to have a mechanism to add information in PROPFIND response that this entry is a symlink. Client may then implement a suitable policy of handling this situation: ignore, mirror as symlink or follow. The detailed policy to be discussed and it should be aligned with how client-side symlinks are treated, as discussed in: #1887, #1440

Having such information would also allow the client to detect a situation of local conflict when locally a filesystem entry would be created with the same as the symlink.

This is a request to include symlink information in the sync protocol, document it and implement the default policy on the client (ignore) with local conflict detection.

[guruz]

FYI @evert for WebDAV input.. FYI @pmaier1 @PVince81 @DeepDiver1975 for general input..

[moscicki]

Hello,

Any progress on this, please?

It does not need full server implementation. All I am asking is this:

This is a request to include symlink information in the sync protocol, document it and implement the default policy on the client (ignore) with local conflict detection.

[evert]

WebDAV has two specifications to help with links.

1. BIND/UNBIND, which, to me, is more like hardlinks.
2. MKREDIRECTREF, which is more like a symlink.

While I don't have experience with these specs, I think #2 is more appropriate.

[moscicki]

Both specs (BIND and MKREDIRECTREF) have experimental status and I do not think are appropriate for what we need here.

What we need is to add symlink information to the response of PROPFIND. Since owncloud already uses custom attributes (e.g.: https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md#list-directory) it should be easy to add one more. It could be a new type inside the element or simply an additional attribute inside the element. The presence of this attribute would indicate that this is a symlink on the server and for the default client behaviour ("ignore") that should be completely sufficient.

That should be sufficient for now for this request.

Subsequent implementation of propagation of symlinks locally (if we really want it) would be more tricky. It would require to add the information about the symlink target (and actually would also require to map the target into local filesystem namespace). It would also require to add a method to create symlinks on the server to propagate them in the other direction. I would leave it for later as it requires more discussion.

[pmaier1]

Ok, what do we have here: 1. Add symlink information to the response of PROPFIND

• Server detects symlinks on (external) storages and has the ability to inform clients about this property
• Clients can then react wrt different policies (ignore, follow) -> Makes the system more robust and reliable -> needs to be done, @PVince81 @DeepDiver1975 for technical implications, please.

2. Local symlink propagation

• Clients get the ability to react with a third policy which is to mirror symlinks on the server to the local filesystem (mirror as symlink) -> Nice to have for feature completeness. Needs to be discussed if there are real world use cases to justify an implementation.

[PVince81]

So this is purely about telling the client about symlinks but not creating them ?

We'd still need a new column in oc_filecache (or a separate table) to store that kind of information for when the scanner finds such symlinks existing on external storage.

Does that info need to include the target of the symlinks or just a flag that it's a symlink ? (flag could be a Webdav property "oc:symlink")

[PVince81]

By the way, since this is about external storage this means that the storage API needs to be extended to support the new property, which also means that external storage implementations (Local, SFTP, etc) need to be adjusted accordingly to report symlinks.

[pmaier1]

So this is purely about telling the client about symlinks but not creating them ?

Well, as far as I understand it, 2. is about creation, right?

Does that info need to include the target of the symlinks or just a flag that it's a symlink ? (flag could be a Webdav property "oc:symlink")

1. needs the flag
2. needs the target in addition, right?

[moscicki]

I would actually foresee to specify the target in the protocol already (but that would not be needed for the default "ignore" action of the client). Existance of the property (with possibly some target) is good enough as a flag.

The target specification is not so simple to handle by the client if you consider that the client may synchronize only certain subfolders. A symlink may lead out of the local synchronization tree. For example, client synchronizing folder X on the server and the symlink X/x -> ../Y/y. Folder Y is not synchronized on the client. It does not make sense to propagate this symlink IMO (and may be even dangerous as it reaches out of the synchronization tree).

This may also happen definitely in case of more generalized storage such as EOS (a symlink may be absolute or relative and point into a completely different tree which is not synchronized).

[evert]

The MKREDIRECTREF spec does in fact have the means to notify a client that something is a redirect. In addition, it also has a mechanism that allows client to either inspect the 'symlink' or the target of the symlink. I also believe that Gnome's GVFS has some support for it, perhaps also for implementing symlinks.

[dragotin]

If the clients continue to ignore the symlinks and ownCloud does not allow to create a symlink through the web interface on the server, this becomes a very specific feature for storages that allow the creation of symlinks outside of ownCloud on the server storage. That exists, like on the one described by @moscicki or on smb which also has symlinks. But it will be rare.

Adding the information that it actually is a symlink on the server storage would be the same as the detection of the symlink on the local file system where we find that out by a stat() call and be useful information. With that, the files could be properly ignored and thus a proper conflict resolution would happen, that should already work I hope ;-)

Not storing the symlink target feels too half cooked for my developers heart. If it is about the filecache table, couldn't the information if it is a symlink be encoded into a higher bit of the permission field? Doing select distinct bin(permissions) from oc_filecache;, shows that the permissions only use the lowest fife bit on my installation, so the sixt bit could indicate that it is a symlink. Not nice, but doable. The symlink target could easily be stored either in a detail table linked via fileId, or even in a normal WebDAV property, to be retrieved as needed.