ftp mirror deletion bug with files prefixed by tilde (~)

[fragtion]

Hi all

I'm noticing a bug with mirror function where if a folder contains a file whose filename is prefixed by "~" (which is allowed in unix naming scheme), it syncs fine on the first run, but on the next run, the mirror function mistakenly deletes that file, as well as all subfolders of the same directory! (If you then run the mirror script for the third time, it downloads the files again.. run it a 4th time and it deletes them again, etc etc recursively)

Problem exists on latest stable release (4.7), and current master git repo (2018/11/13)

We are mirroring large repositories on a schedule, and this repetitive "delete" -"re-acquire" cycle is wasting a lot of bandwidth for nothing.

In my case, the culprit file was an Adobe InDesign lock file called "~2018 bluefox catal~wp))10.idlk"

Here is how the behaviour can be reproduced:

Assume I have the following directory structure on remote FTP:

/root/abc/123/file1
/root/abc/456/file2
/root/xyz/123/file1
/root/xyz/456/file2
/root/xyz/~2018 bluefox catal~wp))10.idlk

If I run mirror operation, on first run it shows something like:

root@server:/# ./test2.sh
source: Is a directory
Transferring file `./~2018 bluefox catal~wp))10.idlk'
Making directory `xyz'
Making directory `xyz/123'
Making directory `xyz/456'
Transferring file `xyz/456/file2'
Transferring file `xyz/123/file1'
Making directory `abc'
Making directory `abc/456'
Transferring file `abc/456/file2'
Making directory `abc/123'
Transferring file `abc/123/file1'

on second run (both source and destination directory trees both untouched), it shows:

root@server:/# ./test2.sh
source: Is a directory
Removing old file `./~2018 bluefox catal~wp))10.idlk'
Removing old directory `abc'
Removing old directory `xyz'

Seems I'm able to dodge the bug using this argument: --exclude-glob "*\~*"

Copy of my script attached for reference:

#!/bin/bash
HOST='localhost'
USER='myusername'
PASS='mypassword'
TARGETFOLDER='/usr/data/ftpsync/'
SOURCEFOLDER='root/'
LOGNAME="/tmp/$HOST-localhost_test-$(date +%F-%H-%M-%S).log"
PIDFILE="/tmp/mirror_localhost_test.pid"
if [ -e ${PIDFILE} ] && kill -0 `cat ${PIDFILE}`; then
   exit # already running
fi
# make sure the PIDFILE is removed when we exit and then claim it
trap "rm -f ${PIDFILE}; exit" INT TERM EXIT
echo $$ > ${PIDFILE}
lftp -f "
set ftp:passive-mode on
set ftp:ssl-allow yes
set ssl:verify-certificate no
open $HOST
user $USER $PASS
mirror --ignore-time --no-perms --no-umask --continue --delete --parallel=5 --log=$LOGNAME --verbose --use-cache '$SOURCEFOLDER' '$TARGETFOLDER'
bye
"
chmod -R 755 $TARGETFOLDER
rm -f ${PIDFILE}

Thanks

[lavv17]

Please test.

[fragtion]

@lavv17 I was hoping to test this but I'm seeing build issue on latest tree as follows

Making all in src make[2]: Entering directory '/usr/local/src/lftp/lftp-github/src' /bin/bash ../libtool --silent --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../lib -I../lib -I../trio -O2 -Wall -Wwrite-strings -Woverloaded-virtual -fno-exceptions -fno-rtti -fno-implement-inlines -MT ResMgr.lo -MD -MP -MF .deps/ResMgr.Tpo -c -o ResMgr.lo ResMgr.cc In file included from ResMgr.cc:20: ../lib/config.h:1664:5: warning: option ‘-Wmissing-prototypes’ is valid for C/ObjC but not for C++ [-Wpragmas] _Pragma ("GCC diagnostic ignored \"-Wmissing-prototypes\"") \ ^~~ ../lib/config.h:1664:5: warning: option ‘-Wmissing-prototypes’ is valid for C/ObjC but not for C++ [-Wpragmas] _Pragma ("GCC diagnostic ignored \"-Wmissing-prototypes\"") \ ^~~ In file included from ResMgr.cc:26: ../lib/math.h:2576:1: error: ‘int signbit(float)’ conflicts with a previous declaration _GL_MATH_CXX_REAL_FLOATING_DECL_2 (signbit) ^~~~~~~~~ In file included from /usr/include/c++/8/math.h:36, from ../lib/math.h:27, from ResMgr.cc:26: /usr/include/c++/8/cmath:661:3: note: previous declaration ‘constexpr bool std::signbit(float)’ signbit(float x) ^~~ In file included from ResMgr.cc:26: ../lib/math.h:2576:1: error: ‘int signbit(double)’ conflicts with a previous declaration _GL_MATH_CXX_REAL_FLOATING_DECL_2 (signbit) ^~~~~~~~~ In file included from /usr/include/c++/8/math.h:36, from ../lib/math.h:27, from ResMgr.cc:26: /usr/include/c++/8/cmath:665:3: note: previous declaration ‘constexpr bool std::signbit(double)’ signbit(double __x) ^~~ In file included from ResMgr.cc:26: ../lib/math.h:2576:1: error: ‘int signbit(long double)’ conflicts with a previous declaration _GL_MATH_CXX_REAL_FLOATING_DECL_2 (signbit) ^~~~~~~~~ In file included from /usr/include/c++/8/math.h:36, from ../lib/math.h:27, from ResMgr.cc:26: /usr/include/c++/8/cmath:669:3: note: previous declaration ‘constexpr bool std::signbit(long double)’ signbit(long double x) ^~~ make[2]: [Makefile:2205: ResMgr.lo] Error 1 make[2]: Leaving directory '/usr/local/src/lftp/lftp-github/src' make[1]: [Makefile:1613: all-recursive] Error 1 make[1]: Leaving directory '/usr/local/src/lftp/lftp-github' make: *** [Makefile:1558: all] Error 2

Running ubuntu 18.10 cosmic (although had this error on 18.04 bionic as well) gcc version 8.2.0 (Ubuntu 8.2.0-7ubuntu1

Should this be a separate issue? Tx