-L with lsf should display links to directory as a directory

laur89 commented 1 year ago

What is your current rclone version (output from `rclone version`)?

rclone v1.53.3-DEV

os/arch: linux/amd64
go version: go1.16.15

What problem are you are trying to solve?

I have following logic:

we list remote directory contents via rclone lsf
choose which nodes we want to copy and build relevant --filter '+ /file1' --filter '+ /dir1/**' flags
use the filter flags generated in previous step with rclone copy in order to transfer those remote nodes

This approach depends on lsf appending trailing slash (/) to the end of directory names.

How do you think rclone should be changed to solve that?

Believe directory symlinks, when -L flag is provided, should also be displayed with this trailing slash showing we're in fact dealing with a directory.

Animosity022 commented 1 year ago

Old version. No rclone.conf No actual full command. No log file showing an example of what you are asking about.

felix@gemini:~$ rclone version
rclone v1.60.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-52-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.2
- go/linking: static
- go/tags: none

Example command


felix@gemini:~$ rclone lsf GD:
crypt/
jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
linkdir/
test2/
test3/

That's a link:

Animosity022 commented 1 year ago

If you mean symbolic links on a local file system, same output:

felix@gemini:~$ rclone lsf /home/felix/test -L -vv
2022/10/25 13:25:19 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/10/25 13:25:19 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "lsf" "/home/felix/test" "-L" "-vv"]
2022/10/25 13:25:19 DEBUG : Creating backend with remote "/home/felix/test"
2022/10/25 13:25:19 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/10/25 13:25:19 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/10/25 13:25:19 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
linkdir/
2022/10/25 13:25:19 DEBUG : 2 go routines active

laur89 commented 1 year ago

Using FTPS remote:

~/.config/rclone/rclone.conf:

[remote]
type = ftp
host = remote.host.domain
user = laur89
port = 21
pass = <redacted>
tls = false
explicit_tls = true
no_check_certificate = true

Listing remote source dir:

$ rclone lsf -L -vv  remote:files/srcdir
2022/10/25 19:46:20 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "lsf" "-L" "-vv" "remote:files/srcdir"]
2022/10/25 19:46:20 DEBUG : Using config file from "/home/laur/.config/rclone/rclone.conf"
2022/10/25 19:46:20 DEBUG : Creating backend with remote "remote:files/srcdir"
2022/10/25 19:46:20 DEBUG : ftp://remote.host.domain:21/files/srcdir: Connecting to FTP server
dir
file
2022/10/25 19:46:23 DEBUG : 2 go routines active

Listing said directory on the remote host:

$ ls -lAt  ~/files/srcdir/
total 0
lrwxrwxrwx 1 laur89 laur89 7 Oct 25 19:14 dir -> ../dir/
lrwxrwxrwx 1 laur89 laur89 7 Oct 25 19:13 file -> ../file

Animosity022 commented 1 year ago

Can you please test on the latest version and share the output?

laur89 commented 1 year ago

$ ./rclone lsf -L -vv  remote:files/srcdir
<7>DEBUG : rclone: Version "v1.60.0" starting with parameters ["./rclone" "lsf" "-L" "-vv" "remote:files/srcdir"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "remote:files/srcdir"
<7>DEBUG : Using config file from "/home/laur/.config/rclone/rclone.conf"
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: Connecting to FTP server
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","remote.host.domain:21")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*fshttp.timeoutConn, err=<nil>
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","46.232.211.100:5842")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","46.232.211.100:7343")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*tls.Conn, err=<nil>
dir
file
<7>DEBUG : 2 go routines active
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: closing 1 unused connections

Note no trailing slash for dir. Remote is one of Ultra.cc servers.

Animosity022 commented 1 year ago

Those will be changes to the FTP library:

https://github.com/jlaffaye/ftp

as that's what's used.

@ncw - not sure how you log these or reflect them.

ncw commented 1 year ago

-L is a flag for the local backend only.

Rclone doesn't understand symlinks on FTP servers.

Can you run your test with --dump bodies so we can see what the server sends

./rclone lsf -L -vv --dump bodies remote:files/srcdir

laur89 commented 1 year ago

$ ./rclone lsf -L -vv --dump bodies remote:files/srcdir
<7>DEBUG : rclone: Version "v1.60.0" starting with parameters ["./rclone" "lsf" "-L" "-vv" "--dump" "bodies" "remote:files/srcdir"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "remote:files/srcdir"
<7>DEBUG : Using config file from "/home/laur/.config/rclone/rclone.conf"
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: Connecting to FTP server
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","remote.host.domain:21")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*fshttp.timeoutConn, err=<nil>
<7>DEBUG : FTP Rx: "220 (vsFTPd 3.0.3)"
<7>DEBUG : FTP Tx: "AUTH TLS"
<7>DEBUG : FTP Rx: "234 Proceed with negotiation."
<7>DEBUG : FTP Tx: "USER laur89"
<7>DEBUG : FTP Rx: "331 Please specify the password."
<7>DEBUG : FTP Tx: PASS *****
<7>DEBUG : FTP Rx: "230 Login successful."
<7>DEBUG : FTP Tx: "FEAT"
<7>DEBUG : FTP Rx: "211-Features:"
<7>DEBUG : FTP Rx: " AUTH TLS"
<7>DEBUG : FTP Rx: " UTF8"
<7>DEBUG : FTP Rx: " EPRT"
<7>DEBUG : FTP Rx: " EPSV"
<7>DEBUG : FTP Rx: " MDTM"
<7>DEBUG : FTP Rx: " PASV"
<7>DEBUG : FTP Rx: " PBSZ"
<7>DEBUG : FTP Rx: " PROT"
<7>DEBUG : FTP Rx: " REST STREAM"
<7>DEBUG : FTP Rx: " SIZE"
<7>DEBUG : FTP Rx: " TVFS"
<7>DEBUG : FTP Rx: "211 End"
<7>DEBUG : FTP Tx: "TYPE I"
<7>DEBUG : FTP Rx: "200 Switching to Binary mode."
<7>DEBUG : FTP Tx: "OPTS UTF8 ON"
<7>DEBUG : FTP Rx: "200 Always in UTF8 mode."
<7>DEBUG : FTP Tx: "PBSZ 0"
<7>DEBUG : FTP Rx: "200 PBSZ set to 0."
<7>DEBUG : FTP Tx: "PROT P"
<7>DEBUG : FTP Rx: "200 PROT now Private."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||5611|)"
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","46.232.211.100:5611")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "drwxr-xr-x   20 1235     1235        16384 Oct 25 10:17 unrelated-dir-1"
<7>DEBUG : FTP Rx: "drwxr-xr-x    5 1235     1235         4096 Oct 26 10:54 unrelated-dir-2"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Oct 25 19:14 dir"
<7>DEBUG : FTP Rx: "-rw-r--r--    1 1235     1235           14 Oct 25 19:13 file"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Oct 25 10:17 unrelated-dir-3"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Aug 01 15:00 unrelated-dir-4"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Oct 25 19:14 srcdir"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Oct 25 09:47 unrelated-dir-5"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 May 16 03:50 unrelated-dir-6"
<7>DEBUG : FTP Rx: "226 Directory send OK."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||9841|)"
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: dial("tcp","46.232.211.100:9841")
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files/srcdir"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "lrwxrwxrwx    1 1235     1235            7 Oct 25 19:14 dir -> ../dir/"
<7>DEBUG : FTP Rx: "lrwxrwxrwx    1 1235     1235            7 Oct 25 19:13 file -> ../file"
<7>DEBUG : FTP Rx: "226 Directory send OK."
dir
file
<7>DEBUG : 2 go routines active
<7>DEBUG : ftp://remote.host.domain:21/files/srcdir: closing 1 unused connections
<7>DEBUG : FTP Tx: "QUIT"

Important to note the trailing slash in link (dir -> ../dir/) is there only because it was provided to ln program at link creation, ie valid directory links may be missing that slash:

/--/
<7>DEBUG : FTP Tx: "LIST files/srcdir"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "lrwxrwxrwx    1 1235     1235            7 Oct 25 19:14 dir -> ../dir/"
<7>DEBUG : FTP Rx: "lrwxrwxrwx    1 1235     1235            6 Oct 26 14:30 dir-link-without-slash -> ../dir"
/--/

rclone / rclone