lftp hangs on file upload (except very small files)

[wq9578]

lftp always worked fine for me, but for some time now upload fails except for very small files. Download still works well, including mirroring large directories. I'm using a current macOS system with lftp installed via homebrew.

Update It worked with cyberduck Protocol is FTPes (explicit FTP over TLS)

Here is the log, with a small file working (README, 308 bytes), and a larger file (COPYING, 18092 bytes) failing:

➜  dokuwiki lftp -e "debug; cd doc; put README; bye" server
Verzeichniswechsel OK, cwd=/                                 
---- CWD path to be sent is `/doc'
---> CWD /doc
<--- 250 OK. Current directory is /doc
Verzeichniswechsel OK, cwd=/doc
---> TYPE I
<--- 200 TYPE is now 8-bit binary             
---> PRET STOR README
<--- 200 Ready to proceed                     
---> PASV
<--- 227 Entering Passive Mode (xxx,xxx,xxx,xxx,157,109)
---- Verbinde Daten Socket mit (xxx.xxx.xxx.xxx) Port 40301
---- Datenverbindung hergestellt                       
---> STOR README
<--- 150 Accepted data connection             
Certificate verification: subjectAltName: »ftp.some-server.com« matched
---- Schließe den Daten Socket
<--- 226-17490 Kbytes used (0%) - authorized: 10485760 Kb
<--- 226-File successfully transferred
<--- 226 0.021 seconds (measured here), 14.26 Kbytes per second
---> MFMT 20220903171522 README
<--- 213 UTIME OK                                        
308 Bytes übertragen
---> QUIT
<--- 221-Goodbye. You uploaded 1 and downloaded 0 kbytes.
<--- 221 Logout.
---- Schließe den Kontroll - Socket

➜  dokuwiki lftp -e "debug; cd doc; put COPYING; bye" poly
Verzeichniswechsel OK, cwd=/                                 
---- CWD path to be sent is `/doc'
---> CWD /doc
<--- 250 OK. Current directory is /doc
Verzeichniswechsel OK, cwd=/doc
---> TYPE I
<--- 200 TYPE is now 8-bit binary              
---> PRET STOR COPYING
<--- 200 Ready to proceed                      
---> PASV
<--- 227 Entering Passive Mode (xxx,xxx,xxx,xxx,157,157)
---- Verbinde Daten Socket mit (xxx.xxx.xxx.xxx) Port 40349
---- Datenverbindung hergestellt                        
---> STOR COPYING
<--- 150 Accepted data connection              
Certificate verification: subjectAltName: »ftp.some-server.com« matched
---- Schließe den Daten Socket
»COPYING« bei 18092 (100%) [Warte auf Übertragungsende]                       

Then the upload is stalling, nothing happens anymore.

[lavv17]

It may be a network problem with MTU, for example. tcpdump can be helpful here.

On Thu, 2 Feb 2023 at 04:39, wq9578 @.***> wrote:

lftp always worked fine for me, but for some time now upload fails except for very small files. Download still works well, including mirroring large directories. I'm using a current macOS system with lftp installed via homebrew.

Here is the log, with a small file working (README, 308 bytes), and a larger file (COPYING, 18092 bytes) failing:

➜ dokuwiki lftp -e "debug; cd doc; put README; bye" server

Verzeichniswechsel OK, cwd=/

---- CWD path to be sent is `/doc'

---> CWD /doc

<--- 250 OK. Current directory is /doc

Verzeichniswechsel OK, cwd=/doc

---> TYPE I

<--- 200 TYPE is now 8-bit binary

---> PRET STOR README

<--- 200 Ready to proceed

---> PASV

<--- 227 Entering Passive Mode (xxx,xxx,xxx,xxx,157,109)

---- Verbinde Daten Socket mit (xxx.xxx.xxx.xxx) Port 40301

---- Datenverbindung hergestellt

---> STOR README

<--- 150 Accepted data connection

Certificate verification: subjectAltName: »ftp.some-server.com« matched

---- Schließe den Daten Socket

<--- 226-17490 Kbytes used (0%) - authorized: 10485760 Kb

<--- 226-File successfully transferred

<--- 226 0.021 seconds (measured here), 14.26 Kbytes per second

---> MFMT 20220903171522 README

<--- 213 UTIME OK

308 Bytes übertragen

---> QUIT

<--- 221-Goodbye. You uploaded 1 and downloaded 0 kbytes.

<--- 221 Logout.

---- Schließe den Kontroll - Socket

➜ dokuwiki lftp -e "debug; cd doc; put COPYING; bye" poly

Verzeichniswechsel OK, cwd=/

---- CWD path to be sent is `/doc'

---> CWD /doc

<--- 250 OK. Current directory is /doc

Verzeichniswechsel OK, cwd=/doc

---> TYPE I

<--- 200 TYPE is now 8-bit binary

---> PRET STOR COPYING

<--- 200 Ready to proceed

---> PASV

<--- 227 Entering Passive Mode (xxx,xxx,xxx,xxx,157,157)

---- Verbinde Daten Socket mit (xxx.xxx.xxx.xxx) Port 40349

---- Datenverbindung hergestellt

---> STOR COPYING

<--- 150 Accepted data connection

Certificate verification: subjectAltName: »ftp.some-server.com« matched

---- Schließe den Daten Socket

»COPYING« bei 18092 (100%) [Warte auf Übertragungsende]

Then the upload is stalling, nothing happens anymore.

— Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHLWXCLXL33RXB232GDWN3WVMF5VANCNFSM6AAAAAAUONTVTQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Alexander.

[FryDay]

I'm having this same issue on MacOS. Anything over about 10K fails.

[entonio]

For me the problem is between 15.5k ~ 16k. Sometimes it manages to transfer 16.000 bytes (with some 451 errors in between, but it ends up succeeding), then the next moment it will fail for 15.500 bytes. It always works without 451 errors for 15.000 bytes.

Here's trying to upload a file named '15500', with 15.500 bytes:

> put 15500
---> PRET STOR 15500
<--- 200 Ready to proceed
---> PASV
<--- 227 Entering Passive Mode (194,___,___,___,___,___)
---- Connecting data socket to (194.___.___.___) port 46061
---- Data connection established               
---> STOR 15500
<--- 150 Accepted data connection
---- Closing data socket
<--- 451-Error during read from data connection             
<--- 451 Transfer aborted
---> SIZE 15500
<--- 213 0                                         
---> PRET STOR 15500
copy: put rolled back to 0, seeking get accordingly
<--- 200 Ready to proceed
---> PASV
<--- 227 Entering Passive Mode (194,___,___,___,___,___)
---- Connecting data socket to (194.___.___.___) port 44828
---- Data connection established               
---> STOR 15500
<--- 150 Accepted data connection
---- Closing data socket
<--- 451-Error during read from data connection             
<--- 451 Transfer aborted
---> SIZE 15500
<--- 213 0
---> PRET STOR 15500
copy: put rolled back to 0, seeking get accordingly
<--- 200 Ready to proceed
---> PASV
<--- 227 Entering Passive Mode (194,___,___,___,___,___)
---- Connecting data socket to (194.___.___.___) port 41630
---- Data connection established               
---> STOR 15500
<--- 150 Accepted data connection
---- Closing data socket
<--- 451-Error during read from data connection             
<--- 451 Transfer aborted
---> SIZE 15500
<--- 213 0                                         
put: 15500: Fatal error: max-retries exceeded
---- Closing idle connection
---> QUIT
<--- 221-Goodbye. You uploaded 75 and downloaded 0 kbytes.
<--- 221 Logout.
---- Closing control socket

[entonio]

NB the problem disappeared for me by using --use-pget-n=10 (it was a reverse mirror). It's worked successfully with a file of 1000 MB (I haven't tested with more than that). I'm not sure if it's related to the original issue.

[entonio]

Aaaaaand the problem returned, just like that. So it's got nothing to do with use-pget. It's the same internet service provider, OS, I haven't even rebooted, so I find it strange that it would be the MTU. A couple of hours after the successful transfers, it's back to the same 451.

One thing I've noticed is that it always seems to occur with the last chunk of the file, e.g. the 1000 MB file will upload fine until the last KBs, and then enter the Sending data/TLS -> Waiting -> rollback-some-KB-and-retry loop.

This would look to me like a server-side issue, but Cyberduck never seems to have a problem.

[entonio]

I've installed a ubuntu VM in my mac, the same 4.9.2 LFTP and... it works. So it seems that 'my' issue is down to something macOS-related (it happens in Intel Mojave and in M2 Sonoma, that I've tried).