FTP LIST timeout with latest Delphi 11 (and idFTP)

[markhumphreysjhc]

I am upgrading from Delphi 10.2 to Delphi 11, and the behaviour of the idFTP.InternalGet method seems to have changed.

It looks like this is a change that is in this repo, around the changes in handling RFC 959 Section 3.2 - lines 2100 in idFTP.

The behaviour is that the request for aLIST <filename> (which exists) timeouts.

The library code in 10.2 does not have this issue, and works as expected.

[markhumphreysjhc]

The commit that seems to have introduced the behaviour is https://github.com/IndySockets/Indy/pull/301

[markhumphreysjhc]

As an experiment I have taken the InternalGet method from the 10.2 code (from before this change), and recompiled the code and it works as previously, so there is a definite change in behaviour after the change.

[rlebeau]

The change for #301 was confirmed as working for multiple servers, so there has to be another factor at play here.

Also, note that the code you are referring to is only used during ACTIVE mode transfers, which is the default mode for the FTP protocol, however that mode very commonly DOES NOT work in modern networking environments due to firewall/router limitations, which has nothing to do with Indy. So, please confirm your network is actually setup to allow inbound connections to TIdFTP's listening ip/port(s) for ACTIVE mode transfers, otherwise try using PASSIVE mode transfers instead (set TIdFTP.Passive=true).

[markhumphreysjhc]

If I change to use TIdFTP.Passive=true, then the Delphi 11 version works. I've gone back and made the same changes to the 10.2 version, and the now the behaviour is the same (and works as I would expect). I will make the changes to our actual code base and see whether the issue is fixed.

[rlebeau]

Though using Passive mode is generally preferred, it would still be nice to know why Active mode transfers are not working for you. I'm pretty sure it is not a problem with TIdFTP itself, but just to make sure, I would suggest using a packet sniffer like Wireshark to verify whether or not inbound TCP connections are actually reaching TIdFTP's Active mode listening port. Odd that the 10.2 code would work and the 11.0 code does not, since it is the same TCP handling being used for the inbound port, just different signaling being used to know whether to read from the command socket or the data socket first.

[boramis]

We're running into this too, connecting to a ProFTPD server running on Ubuntu 18.04. I haven't had a chance to break out a packet sniffer yet, but it's trivially reproducible by using a clean Ubuntu 18.04 VM, "sudo apt install proftpd-basic", and then logging in as the current user. I'm seeing it with the attached sample under Windows 10 + Delphi 10.4 as well as Linux/macOS using Free Pascal. It fails with the current code and succeeds as soon as the change from #301 is reverted.

IndyActiveFtpBug.dpr.txt

[michaelJustin]

The same has been reported in the German forum "Delphipraxis.net", the FTP server was said to be ServU system. After setting Passive to True, the timeout did no longer occur. Unfortunately I don't have Delphi 11 to reproduce it. Link to the Delphipraxis post: https://www.delphipraxis.net/1497486-post7.html

[michaelJustin]

I can reproduce the time out (EIdAcceptTimeout, line 2128 in unit IdFTP) with Free Pascal 3.2.0 and test server test.rebex.net in active mode. In passive mode, the LIST returns the expected file list. Update: the line 2128 is not frequently reached. In most tests, the read timeout is in line 2134:

            if LReadList.ContainsSocket(LDataSocket) then
            begin
              LPortSv.Listen(0);
              Self.GetResponse([125, 150, 154]); <--------------

[rlebeau]

I don't know what to tell you. The code checked in for #301 was thoroughly tested as working against multiple servers, and I can't revert that code at this point.

The default TIdFTP.ListenTimeout is 10secs, and a timeout on LSocketList.SelectReadList() means that there is no response arriving on the command channel, and no incoming connection arriving on the data channel, during that 10sec period. And a timeout on Self.GetResponse() means the data channel received an inbound connection, which LPortSv.Listen() accepted successfully, but then no response arrived on the command channel to acknowledge that connection.

So, this HAS to be a networking issue, not an Indy issue. But why it is affecting all of you similarly, I have no idea.

I have NO WORKING DEBUGGER installed right now, so I have NO WAY to test this myself, so I NEED logs and sniffer captures to see what is going on with the traffic on the wire.

I'm not joined to the German side of Delphi-PRAXIS site, only the English side. Can someone post a reply to https://www.delphipraxis.net/1497486-post7.html pointing them to this ticket?

[DavidIzadaR]

I used the project IndyActiveFtpBug to capture the network traffic between my PC and test server test.rebex.net. Instead of using Wireshark, I used SoftPerfectPacketAnalyzer. To facilitate the analysis, I'm attaching the capture, capture report (including packet data), and the log created by the program. I'm using RAD Studio 11 on Windows 11. LOG-Error.txt LOG-Error.zip CaptureReport-Error.txt

[DavidIzadaR]

To make the analysis even easier, I'm attaching the same information using the previous code (reverting #301) and returning the correct directory from the FTP site. AllLogs.zip

[DavidIzadaR]

I was comparing the logs for the two scenarios: FTP LIST request working with the old code vs the same request failing with the new code. I confirmed the correct transfer to the "active" port the client sends to the server (using PORT IP address, port). I also added a few comments to the logs to highlight that the differences started after sending the LIST command in a slightly different way. The working scenario is sending LIST while the failing scenario is sending just LIST. After this request, the behavior of the server changes (sending information to different ports). CaptureComparison.zip

[rlebeau]

I have merged the fix, thanks.

[nickgabb]

IDFTP1.List timeout in Delphi11 I have tried changing Write to WriteLn in IDFTP.Pas. Didn't work If I change to Passive it is refused Even when trying with CoreFTP This is causing me a major headache. Can the code be extracted from the working 10.2 IDFTP.Pas or is it deep[er than that? Any other solutions? Code snippet Not using FTPS

     IdFTP1.Host:=FTPAddress;
     IdFTP1.Port:=FTPport;  //Hard code to standard ftp port   WSH needed 22
     //Set Passive to True to avoid sending problems- No Idea Why!!
     IdFTP1.Passive:=False;

     IdFTP1.Username:=FTPUsername;
     IdFTP1.Password:=FTPPassword;
     //Note: May have to also set IdFTP1.Passive:=true/false;
     Try
        If NOT IdFTP1.Connected then
           Begin
              if UseFTPS then
                 Begin
                    IdSSLIOHandlerSocketOpenSSL1.Host:=FTPaddress;
                    IdSSLIOHandlerSocketOpenSSL1.Port:=FTPport;
                    IdFTP1.IOHandler:=IdSSLIOHandlerSocketOpenSSL1;
                    IdFTP1.UseTLS:=utUseExplicitTLS;
                    IdFTP1.DataPortProtection:=ftpdpsPrivate;
                 End;
              IdFTP1.Connect;
              if Trim(FTPSubFolder) <> '' then
                 IdFTP1.ChangeDir(Trim(FTPSubFolder));

           End;

        IdFTP1.List(DirectoryList);   //Timeout here

[DavidIzadaR]

The latest code in Delphi 11.1 comes without the fix - IOHandler.WriteLn(ACommand) - and when I ran my previous project IndyActiveFtpBug I got a different error than before:

STAT> Connected.
RECV> 220 Microsoft FTP Service<EOL>
SENT> HOST test.rebex.net<EOL>
RECV> 504 Server cannot accept argument.<EOL>
SENT> USER demo<EOL>
RECV> 331 Password required for demo.<EOL>
SENT> PASS password<EOL>
RECV> 230 User logged in.<EOL>
SENT> FEAT<EOL>
RECV> 211-Extended features supported:<EOL> LANG EN*<EOL> UTF8<EOL> AUTH TLS;TLS-C;SSL;TLS-P;<EOL> PBSZ<EOL> PROT C;P;<EOL> CCC<EOL> HOST<EOL> SIZE<EOL> MDTM<EOL> REST STREAM<EOL>211 END<EOL>
SENT> TYPE A<EOL>
RECV> 200 Type set to A.<EOL>
SENT> SYST<EOL>
RECV> 215 Windows_NT<EOL>
SENT> TYPE A<EOL>
RECV> 200 Type set to A.<EOL>
SENT> PORT 192,168,4,40,201,172<EOL>
RECV> 501 Server cannot accept argument.<EOL>

In the past, the missing <EOL> was causing the timeout.

The previous PORT command is correct. My PC uses local IP 192.168.4.40 and it is listening to port 51628 ready to receive server messages.

Fixing this issue will require sniffing the network and capturing the command causing the error.

[nickgabb]

Hi The WriteLn Fix Worked! My program was not finding the amended source version Fixed now. Thanks to All!

Regards

Nick

@.***

From: David Izada Rodriguez @.> Sent: 31 March 2022 13:58 To: IndySockets/Indy @.> Cc: Nick Gabb @.>; Comment @.> Subject: Re: [IndySockets/Indy] FTP LIST timeout with latest Delphi 11 (and idFTP) (#377)

The latest code in Delphi 11.1 comes without the fix - IOHandler.WriteLn(ACommand) - and when I ran my previous project IndyActiveFtpBug I got a different error than before: STAT> Connected. RECV> 220 Microsoft FTP Service SENT> HOST test.rebex.net RECV> 504 Server cannot accept argument. SENT> USER demo RECV> 331 Password required for demo. SENT> PASS password RECV> 230 User logged in. SENT> FEAT RECV> 211-Extended features supported: LANG EN* UTF8 AUTH TLS;TLS-C;SSL;TLS-P; PBSZ PROT C;P; CCC HOST SIZE MDTM REST STREAM211 END SENT> TYPE A RECV> 200 Type set to A. SENT> SYST RECV> 215 Windows_NT SENT> TYPE A RECV> 200 Type set to A. SENT> PORT 192,168,4,40,201,172 RECV> 501 Server cannot accept argument.

In the past, the missing was causing the timeout. The previous PORT command is correct. My PC uses local IP 192.168.4.40 and it is listening to port 51628 ready to receive server messages. Fixing this issue will require sniffing the network and capturing the command causing the error.

— Reply to this email directly, view it on GitHubhttps://github.com/IndySockets/Indy/issues/377#issuecomment-1084544785, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATHSUCEJ765HGMALJ2R744DVCWOOVANCNFSM5EEKH6ZA. You are receiving this because you commented.Message ID: @.***>

[rlebeau]

@DavidIzadaR,

SENT> PORT 192,168,4,40,201,172 RECV> 501 Server cannot accept argument.

The previous PORT command is correct. My PC uses local IP 192.168.4.40 and it is listening to port 51628 ready to receive server messages.

Assuming that the FTP server is running outside of your LAN, and your router does not natively support FTP transfers, then active mode FTP transfers (TIdFTP.Passive=False) passing through the router will fail to connect, unless you do the following:

1. set the TIdFTP.ExternalIP property to your router's external WAN IP.
2. open the same TIdFTP.DataPort port on the router's WAN side
3. configure the router to forward inbound connections on that WAN IP/port to TIdFTP's listening IP/port.

This is why active mode should be avoided when running TIdFTP behind a router. Use passive mode (TIdFTP.Passive=true`) instead.

[nickgabb]

The Write( to WriteLn( change worked perfectly it would appear It failed at first because My program was not reading the correct IDftp.Pas Added it into the program to test and it worked perfectly. Removed the module Added the paths to the search path in "library" so it found it. and again it worked. Or my code stepped past it and it filled the Tstrings. Not gone past that as I do not want to download my Customer data and forget to put it back.

[auchri]

Hi! Does anyone know if there is a RAD studio version with the fix included?

[rlebeau]

I don't believe so yet. Embarcadero usually updates their copy of Indy only during major releases, not minor releases or patches. When they updated Indy for the release of RAD Studio 11.0, that included the fix for #300, which also introduced the timeout bug described in this ticket. That bug was fixed a couple of months after the 11.0 release. RSP-35840 relates to this ticket, but it has been closed as "No Longer Applies" since the fix is merged into Indy's GitHub code. So, the fix likely won't get rolled into Embarcadero's copy of Indy until RAD Studio 12.0, whenever that eventually comes out. In the meantime, you can just replace Embarcadero's copy of Indy with your own copy from Indy's GitHub code.