[Bug] mirror command does not work in Ubuntu 18.04

[pbtura]

I'm trying to use mirror to copy some files from a remote server to a local folder. I can connect to the ftp and run other commands successfully. When I try to do 'mirror -c' however, the command does not produce any results. None of the requested files are copied to local. I had another person on our team run the same command to make sure it wasn't a local configuration issue and they got the same results.

To see if I could locate the source of the problems, I ran an strace command: strace -f -yy -e trace=file -q -o lftp.strace . Looking at the results, I suspect the problem is related to timezone in some way. The command is repeatedly trying to stat /etc/localtime before timing out. I will include the stacktrace results below.

Environment: Ubuntu 18.04 LFTP 4.8.1

16397 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=16413, si_uid=1001, si_status=0, si_utime=0, si_stime=0} ---
16397 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 5</etc/nsswitch.conf>
16397 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 5</etc/ld.so.cache>
16397 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
16397 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_compat.so.2", O_RDONLY|O_CLOEXEC) = 5</lib/x86_64-linux-gnu/libnss_compat-2.27.so>
16397 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 5</etc/ld.so.cache>
16397 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
16397 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 5</lib/x86_64-linux-gnu/libnss_nis-2.27.so>
16397 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
16397 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnsl.so.1", O_RDONLY|O_CLOEXEC) = 5</lib/x86_64-linux-gnu/libnsl-2.27.so>
16397 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
16397 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 5</lib/x86_64-linux-gnu/libnss_files-2.27.so>
16397 openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 5</etc/passwd>
16397 getcwd("/home/pbuchheit/eyerep-files", 4096) = 29
16397 openat(AT_FDCWD, ".", O_RDONLY|O_DIRECTORY) = 5</home/pbuchheit/eyerep-files>
16397 getcwd("/home/pbuchheit/eyerep-files", 4096) = 29
16397 chdir("/home/pbuchheit/eyerep-files") = 0
16397 chdir("/home/pbuchheit/eyerep-files") = 0
16397 chdir("/home/pbuchheit/eyerep-files") = 0
16397 openat(AT_FDCWD, "/home/pbuchheit/eyerep-files", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 6</home/pbuchheit/eyerep-files>
16397 lstat("/home/pbuchheit/eyerep-files/.", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
16397 openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 6</etc/passwd>
16397 openat(AT_FDCWD, "/etc/group", O_RDONLY|O_CLOEXEC) = 6</etc/group>
16397 lstat("/home/pbuchheit/eyerep-files/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
16397 lstat("/home/pbuchheit/eyerep-files/cmd.log", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
16397 lstat("/home/pbuchheit/eyerep-files/lftp.strace", {st_mode=S_IFREG|0664, st_size=14409, ...}) = 0
16397 openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 5</usr/share/zoneinfo/America/Louisville>
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0
16397 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2781, ...}) = 0

(repeats several thousand times)

[pbtura]

Quick update. I also tried doing the command in Ubuntu 20.04 and got the same results.

[lavv17]

Please try cls command in the same directory to check if the file listing can be parsed successfully.

On Thu, 3 Feb 2022 at 18:21, pbtura @.***> wrote:

Quick update. I also tried doing the command in Ubuntu 20.04 and got the same results.

— Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/659#issuecomment-1029099000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHLWXB4VXHZFWUES2KLOH3UZKMONANCNFSM5NMK6WYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Alexander.

[pbtura]

Interesting results. Doing an 'ls' on that directory works fine but 'cls' runs into the same error as 'mirror'. I also tried a different directory with fewer files and got the same results.

[pbtura]

Please try cls command in the same directory to check if the file listing can be parsed successfully. On Thu, 3 Feb 2022 at 18:21, pbtura @.> wrote: Quick update. I also tried doing the command in Ubuntu 20.04 and got the same results. — Reply to this email directly, view it on GitHub <#659 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHLWXB4VXHZFWUES2KLOH3UZKMONANCNFSM5NMK6WYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.> -- Alexander.

I did some digging into the source code. From what I can tell, there is something off about the behaviour of the Ftp::ParseLongList() method of the FtpListInfo class. If I connect to a remote unix server, it will figure out the correct line parser and return the parsed data. However, if I try to connect to a non-unix server (MSLD in our case), the data will get parsed, but the code to store the current parser as the 'guessed_parser' never gets called. As a result, when it gets to the end of the method, the parser and output data have not been stored, so it returns an empty list.

Digging deeper I found some odd behaviour in ParseFtpLongList_UNIX(). Since the remote server is NOT unix based, in theory this parser should result in an error. However, MSLD returns file info strings formatted like "size=0;modify=20220118175102;create=20191127173656;perm=cpledf;type=dir; ." . When ParseFtpLongList_UNIX checks the line it calls

if(strchr("bcpsD",line[0])) // block, char, pipe, socket, Door.
      return 0;

Since the first character of the line is 's' it treats it as a socket and returns 0 rather than attempting the parse and returning an error. I commented out that line and re-ran my command and it printed the contents of the directory.