search

lavv17 / lftp

sophisticated command line file transfer program (ftp, http, sftp, fish, torrent)
http://lftp.yar.ru
GNU General Public License v3.0
1.11k stars 162 forks source link

mirror using fxp = stalling and incomplete #550

Open kornos opened 5 years ago

kornos commented 5 years ago

hello

I have lftp version 4.8.4 /cygwin I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Getting directory contents (0) [Receiving data]" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood (this error is not for all files) Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

lavv17 commented 5 years ago

Try "set ftp:use-stat off"

ср, 25 сент. 2019, 23:01 The Fscking Sultan notifications@github.com:

hello

I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Receiving data" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXBM6LH5QETREW35LVLQLO7RXA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNWM3GA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXBI6ZQNJ2E6BS4BRSLQLO7RXANCNFSM4I2RNRWQ .

lavv17 commented 5 years ago

Also enable debug in lftp to see what is going on (command debug)

чт, 26 сент. 2019, 6:32 Alexander Lukyanov lavv17f@gmail.com:

Try "set ftp:use-stat off"

ср, 25 сент. 2019, 23:01 The Fscking Sultan notifications@github.com:

hello

I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Receiving data" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXBM6LH5QETREW35LVLQLO7RXA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNWM3GA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXBI6ZQNJ2E6BS4BRSLQLO7RXANCNFSM4I2RNRWQ .

kornos commented 5 years ago

ok, enabled debug and "set ftp:use-stat off".

  1. Error below doesn't appear anymore mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood (this error is not for all files)

2.Other relevant things I am seeing in debug.log:

---- Data connection established ---> LIST <--- 150 Opening ASCII mode data connection for directory listing. <--- 226- [Ul: 0.0MB] [Dl: 0.0MB] [Speed: 22.40K/s] [Free: ....MB] Timeout - reconnecting ---- Closing data socket ---- Closing control socket ---> TYPE A <--- 200 Type set to A. ---> PASV <--- 227 Entering Passive Mode (...............) ---- Connecting data socket to (.........) port 30149 ---- Data connection established ---> LIST <--- 150 Opening ASCII mode data connection for directory listing. <--- 226- [Ul: 0.0MB] [Dl: 0.0MB] [Speed: 20.99K/s] [Free: ....MB] Timeout - reconnecting ---- Closing data socket ---- Closing control socket ---> TYPE A <--- 200 Type set to A. ---> PASV <--- 227 Entering Passive Mode (........) ---- Connecting data socket to (.........) port 30869 ---- Data connection established

Line quality is very good,tested with other fxp clients, I think that timeout is not related to that.

ml- commented 5 years ago

IIRC the following were recently defaulted to yes:

ftp:ssl-protect-data
ftp:ssl-protect-fxp
ftp:ssl-protect-list

I'm not sure if your set ftp:ssl-allow false setting disables all SSL. Might want to turn the protect settings off since it looks like you don't want SSL.

Your parallel connection seem quiet high, are you sure it works fine with the same settings on FTPRush?

kornos commented 5 years ago

OK , I've added those setting 3 settings above, nothing changed.

"Your parallel connection seem quiet high" -> it is not high for me , because I have a 10gbit/s line which needs to be saturated. For 10gbit/s lines, u need multiple connections(10-15) in order to saturate the line , this is what I use(successfully) for FtpRush.

I made tests with 10, 5 and 1 connection. In all cases it blocks with same error I posted initially + "Getting directory contents (0) [Receiving data]".

Weird issue is that this happens around after 20-25 files, not at the beginning. Just a reminder : this transfer is done in FXP mode.

ml- commented 5 years ago

Can you enable the most verbose debug (debug 9) and post all the logs? Don't forget to censor private data. Would be nice to know where it stops.

Also, do you control the FTP Server that is FXPd from/to? Anything helpful in the logs?

kornos commented 5 years ago

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in my second post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I wrote in my 2nd post. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

lavv17 commented 5 years ago

The server sends 226- which means another line is coming, but is never received.

сб, 28 сент. 2019, 5:03 The Fscking Sultan notifications@github.com:

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in the first post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I posted first. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXHNNYLFUCAJCZID4ATQL23QJA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72OJBA#issuecomment-536142980, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXEHDYW6QFNM3KMCURDQL23QJANCNFSM4I2RNRWQ .

lavv17 commented 5 years ago

You can try "set use-stat-for-list yes" to avoid LIST command.

сб, 28 сент. 2019, 20:07 Alexander Lukyanov lavv17f@gmail.com:

The server sends 226- which means another line is coming, but is never received.

сб, 28 сент. 2019, 5:03 The Fscking Sultan notifications@github.com:

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in the first post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I posted first. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXHNNYLFUCAJCZID4ATQL23QJA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72OJBA#issuecomment-536142980, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXEHDYW6QFNM3KMCURDQL23QJANCNFSM4I2RNRWQ .

kornos commented 5 years ago

"You can try "set use-stat-for-list yes" to avoid LIST command."

This will work considering you recommended me above to "set ftp:use-stat off" ?

lavv17 commented 5 years ago

Yes, these are different settings

сб, 28 сент. 2019, 21:50 The Fscking Sultan notifications@github.com:

"You can try "set use-stat-for-list yes" to avoid LIST command."

This will work cause you recommended me above to "set ftp:use-stat off" ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXCYJOWWLDRKO232ISTQL6RPNA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD73AEIQ#issuecomment-536216098, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXCIUJ6TNL4JWUY745TQL6RPNANCNFSM4I2RNRWQ .