Version 8.6.0 broke FTP with data connection over SSL

Vogtinator commented 8 months ago

I did this

curl -l --ssl-reqd ftp://10.168.4.192

On that IP I've got vsftpd 3.0.5 running with SSL and anonymous read access enabled.

I expected the following

Dirlist gets returned and curl exits successfully.

What happens instead is that after a slight but noticable delay, the dirlist is shown but curl doesn't return immediately but instead times out.

This worked in version 8.5.0 just fine. Comparison with -vvv:

8.5.0:

> PWD
< 257 "/" is the current directory
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Extended Passive Mode (|||50680|)
* Connecting to 10.168.4.192 (10.168.4.192) port 50680
*   Trying 10.168.4.192:50680...
* Connected to 10.168.4.192 (10.168.4.192) port 21
* SSL reusing session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
> TYPE A
< 200 Switching to ASCII mode.
> NLST
< 150 Here comes the directory listing.
* Maxdownload = -1
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / prime256v1 / UNDEF
* Server certificate:
*  subject: C=DE; ST=DE; L=Nuremberg; O=SUSE; OU=QAM; CN=SUSE; emailAddress=suse@suse.com
*  start date: Nov 14 14:50:45 2018 GMT
*  expire date: Nov 11 14:50:45 2028 GMT
*  issuer: C=DE; ST=DE; L=Nuremberg; O=SUSE; OU=QAM; CN=SUSE; emailAddress=suse@suse.com
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
asdf
* TLSv1.3 (IN), TLS alert, close notify (256):
* Remembering we are in dir ""
* TLSv1.3 (OUT), TLS alert, close notify (256):
< 226 Directory send OK.
* Connection #0 to host 10.168.4.192 left intact

8.6.0:

> PWD
< 257 "/" is the current directory
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Extended Passive Mode (|||58372|)
* Connecting to 10.168.4.192 (10.168.4.192) port 58372
*   Trying 10.168.4.192:58372...
* Connected to 10.168.4.192 (10.168.4.192) port 21
* SSL reusing session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
> TYPE A
< 200 Switching to ASCII mode.
> NLST
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
< 150 Here comes the directory listing.
* Maxdownload = -1
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / prime256v1 / UNDEF
* Server certificate:
*  subject: C=DE; ST=DE; L=Nuremberg; O=SUSE; OU=QAM; CN=SUSE; emailAddress=suse@suse.com
*  start date: Nov 14 14:50:45 2018 GMT
*  expire date: Nov 11 14:50:45 2028 GMT
*  issuer: C=DE; ST=DE; L=Nuremberg; O=SUSE; OU=QAM; CN=SUSE; emailAddress=suse@suse.com
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS alert, close notify (256):
asdf

It is visible that 8.6.0 does not detect the end of the data transfer properly.

I've started a bisect to narrow down which change triggered it.

curl/libcurl version

8.6.0 and git master (curl-8_6_0-46-gcf5f60422)

operating system

openSUSE Tumbleweed

bagder commented 8 months ago

It is visible that 8.6.0 does not detect the end of the data transfer properly.

If you can, please try the current git master as well as I believe it might be fixed already.

Vogtinator commented 8 months ago

Already did, git master is also affected.

My bisect finished, the cause is ... *drumroll*... 9a90c9dd64d2f03601833a70786d485851bd1b53 (vtls: receive max buffer)

Vogtinator commented 8 months ago

Hm, fun. That commit got reverted and after the revert it does work, but apparently not always? My git bisect log shows current master (cf5f604229bb9fab4421ec0e498aaf2767feb335) as bad. Maybe that was just the "slight but noticable delay" and not a full failure.

icing commented 8 months ago

The "not always" would be the interesting part.

Vogtinator commented 8 months ago

Maybe that was just the "slight but noticable delay" and not a full failure.

I guess so, after ed09a99af57200643d5ae001e815eeab9ffe3f84 it does work reliably, tried a few 100 times.. Not sure why it didn't work for me initially.

Sorry for the inconvenience and thanks for the fast reply!

Vogtinator commented 8 months ago

The "not always" would be the interesting part.

Yes. It did definitely fail for me at least once, but does work reliably now. As long as it doesn't fail again for me I'm happy to ignore that. I'll reopen if it occurs again.

The "slight but noticable delay" appears to be caused by the SSL handshake, I'll continue investigating that on my side but it's unrelated to this issue.

curl / curl