Allow specifying allowed/disallowed character patterns in a rclone mount

ghost commented 2 years ago

The associated forum post URL from `https://forum.rclone.org`

What is your current rclone version (output from `rclone version`)?

rclone v1.59.1

What problem are you are trying to solve?

Read this issue first: https://github.com/drakkan/sftpgo/issues/1005

Server-side blocking : character in filename confuses rclone mount. With --vfs-cache-mode writes, when I try to upload a file with the name test:test, rclone fails and retires uploading endlessly. Accidentally copying a file with remaining : character in filename, only to find out it didn't get uploaded later is not funny.

Sep 29 19:10:24 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:10:24 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied
Sep 29 19:10:33 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:10:33 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied
Sep 29 19:10:53 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:10:53 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #3, will retry in 40s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied
Sep 29 19:11:33 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:11:33 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied
Sep 29 19:12:53 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:12:53 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied
Sep 29 19:15:33 fedora sh[368464]: ERROR : /test:test: Failed to copy: Update Create failed: permission denied
Sep 29 19:15:33 fedora sh[368464]: ERROR : /test:test: vfs cache: failed to upload try #6, will retry in 5m0s: vfs cache: failed to transfer file from cache to remote: Update Create failed: permission denied

Mounting with --vfs-cache-mode off instead doesn't fix the issue and rather makes it worse, as the "illegally" named file tends to outright disappear and your data is lost.

How do you think rclone should be changed to solve that?

I want rclone mount to be able to disallow certain characters in filename, so when user tries to create or copy a file with that specific character in filename, it's blocked by rclone directly, rather than sending to the server only to get rejected.

Maybe a new flag like --filename-disallow-character-pattern "*:*" or --filename-allow-character-pattern "a-z, A-Z, 0-9" under rclone mount, which makes the corresponding FUSE mount to disallow the use of specific character patterns. Any existing files with disallowed characters on the remote should be invisible to the user.

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

ncw commented 2 years ago

The way to fix this would be to implement encoding in the sftp backend (I was surprised to see it wasn't there already).

It needs to be similar to the local backend encoding here: https://rclone.org/local/#restricted-characters

Then you could add Colon to the encoding and rclone will translate the : into a unicode lookalike and back.

ghost commented 2 years ago

Then you could add Colon to the encoding and rclone will translate the : into a unicode lookalike and back.

Would that actually work though? I thought Colon translates full-width colon filename under FUSE mount into half-width colon on remote and vice-verse, but not the other way around so half-width colon still can't be avoid on remote even if local encoding is implemented for sftp.

Quoting the documentation:

take a Windows system where there is a file with name Test：1.jpg, where ： is the Unicode fullwidth colon symbol. When using rclone to copy this to a remote which supports :, the regular (halfwidth) colon (such as Google Drive), you will notice that the file gets renamed to Test:1.jpg.

Or is there anything like a ReverseColon where half-width colon under FUSE mount is used to represent full-width colon?

ncw commented 2 years ago

I thought Colon translates full-width colon filename under FUSE mount into half-width colon on remote and vice-verse, but not the other way around so half-width colon still can't be avoid on remote even if local encoding is implemented for sftp.

The encoding lets you store file names that the remotes doesn't normally let you store, so if encoding was implemented for sftp then it would allow files with : in them to be stored as ： on the sftp server.

I understand the SFTP server is a windows box, so making that encoding the same as the normal windows encoding (as used by the local backend) would be perfect for Linux clients. For Windows clients you wouldn't have any encoding as all the file names are all already correct.

So linux box would store colon:file.txt which would be translated to colon：file.txt for storage on the Windows sftp server. This would appear as colon:file.txt on the linux client when read back.

A windows box (where you wouldn't set the encoding) would see this file as colon：file.txt.

Does that make sense?

ghost commented 2 years ago

Yes, this would work for me.

rclone / rclone