search

lavv17 / lftp

sophisticated command line file transfer program (ftp, http, sftp, fish, torrent)
http://lftp.yar.ru
GNU General Public License v3.0
1.11k stars 162 forks source link

Mirror sometimes hangs on completion with --loop #581

Open DuncanSands opened 4 years ago

DuncanSands commented 4 years ago

x86-64 Ubuntu 18.04.4 LFTP | Version 4.8.4 Libraries used: GnuTLS 3.6.5, idn2 2.0.4, Readline 8.0, zlib 1.2.11

.lftprc: debug

Connection method: sftp

Hanging mirror command (doesn't always hang): mirror -v --continue --delete --loop --parallel=2 --use-pget=4 --on-change=!"touch $TOUCHFILE" -p -X metadata -X .state -X Events -X FoundationCompanyPlusDaily Products mirror

I noticed that every time it hangs, there is "Retrying mirror" followed by download of an additional file. For example:

Transferring file CurrencyRates/CurrencyRatesChange20200318.zip' ... more files here... PriceEquityRecentHistoryIntraday/PriceEquityRecentHistoryIntradayChange20200318-02.zip' Total: 29 directories, 900 files, 0 symlinks New: 38 files, 0 symlinks 94670 bytes transferred in 101 seconds (939 B/s) Retrying mirror... Transferring file `GICRS_ISIN_Daily/GICRS_ISIN_DailyChange20200318210016-001.zip' ... hangs ...

I am trying without --loop to see if it helps, but it will take a few days to know.

Debug log: lftp.log

DuncanSands commented 4 years ago

I can confirm that without --loop there is no hang.

andrei-radulescu commented 3 years ago

i can confirm as well on 20.04.

It hangs with retrying the whole parent process somehow and goes into a infinite loop as this is in my case the bash script. If run manually inside the lftp promt it won't hang.

ProjectMountiainMan commented 2 years ago

I am stuck on Getting directory contents (0) [Waiting for response...] while trying to run reverse mirror with --loop. The files are transferred properly. Running manually inside lftp does not help. At first I thought it might be log files be added to the src directory, but putting them somewhere else does not help.

ProjectMountiainMan commented 2 years ago

Adding the -P flag (-P 5) seemed to fix the hanging behavior for me.

ESNewmanium commented 6 months ago

Can confirm that this is still present on lftp 4.9.2 that ships with Ubuntu 22.04. Any mirror command with --loop that finds a new file will hang. Mirror commands that don't find any new files on the first loop will complete as expected.

$ lftp --version
LFTP | Version 4.9.2 | Copyright (c) 1996-2020 Alexander V. Lukyanov

LFTP is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with LFTP.  If not, see <http://www.gnu.org/licenses/>.

Send bug reports and questions to the mailing list <lftp@uniyar.ac.ru>.

Libraries used: GnuTLS 3.7.3, idn2 2.3.2, Readline 8.1, zlib 1.2.11

EDIT: I'll additionally add that I compiled copies of 4.8.4 and 4.7.7 but still found the bug to be present. I had to go back to 4.5.6 for a version where mirror --loop didn't exhibit this bug.