"Network Error: unexpected end of file" error on some websites

[Shnatsel]

On some websites, e.g. http://correios.com.br, ureq fails with the following error:

Network Error: unexpected end of file

However, curl and Firefox work fine.

There's 162 such websites in the top million (I'm using Tranco list generated on the 3rd of February).

Archive with all occurrences: ureq-unexpected-eof.tar.gz

Code used for testing: https://github.com/Shnatsel/rust-http-clients-smoke-test/blob/f206362f2e81521bbefb84007cdd25242f6db590/ureq-smoke-test/src/main.rs

[algesten]

Hey @Shnatsel, thank you for running through these test cases!

This one I think is due to a late surfacing TLS error. http://correios.com.br is actually working for me now, but if I pick other ones in the tgz, I get the error.

The remote side hangs up the socket when we are sending the request header. The only explanation I have would be TLS. I haven't got wireshark running on my m1 mac, so I haven't looked deeper yet. That would be my next thing to do on this issue.

[jsha]

I think this is a variant of #318. Presumably we get one or the other error depending on whether the server sends a close_notify vs closing the connection outright.

Here's a command that fetches each of these URLs using curl with similar TLS and cipher suite settings to what we use:

tar tzf ~/Downloads/ureq-unexpected-eof.tar.gz | while read host ; do curl --max-time 60 --no-progress-meter --location --tlsv1.2 --ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-SHA,ECDHE-ECDSA-AES256-SHA,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-RSA-CHACHA20-POLY1305,RENEGOTIATION-INFO http://${host} >/dev/null ; done

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.cleanlink.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to rbc.org:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to sundarammutual.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.noip.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to tilde.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to dsbmobile.de:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to writing-online.net:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to opolskie.pl:443 
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to norcalpacing.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.suffolknews.co.uk:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.inverness-courier.co.uk:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.cambridgeindependent.co.uk:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to onestop.global:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to manpowergroup.us:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to qkb.gov.al:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to crossroadsbellevue.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.epunjabschool.gov.in:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.noip.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.kabelshop.nl:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to baladia.gov.kw:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.lakecountyclerk.org:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.uj.ac.za:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.bandmine.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.pabar.org:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to smeco.coop:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to web.gasline.com.tr:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to generationlighting.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to doodleordie.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to ataonline.com.tr:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to steelguru.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.ktb.co.kr:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.ncm.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to etitan.hu:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to kumakumakumabear.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to aavsb.org:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.tsbohemia.cz:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to dvat.gov.in:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to animetourism88.com:443 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.mingtian6.com:443 
...

[jsha]

Here's the result of letting the above command run through the full list. Of 510 input hostnames, we get 479 output lines with SSL_ERROR_SYSCALL. I think that's sufficient to conclude this is a symptom of servers not supporting the ciphersuites / TLS versions we need.

log.txt

[jsha]

I think this may also be related: https://github.com/ctz/rustls/issues/160

[algesten]

Some of these examples are now working (like http://correios.com.br). Unclear whether that's been fixed on the server end or by changes in rustls. However some still show the problem:

http://5thavenue.org/: Connection Failed: unexpected end of file

I unpacked all of @Shnatsel files in a subdir to my repo checkout and ran:

for f in *; do (cd ../ && cargo run --example cureq --all-features -- "http://$f" >/dev/null); done

I estimate roughly 50% of the original list still show the problem.

[mq1]

https://user.auth.xboxlive.com/user/authenticate throws

Error: Transport(Transport { kind: Io, message: None, url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("user.auth.xboxlive.com")), port: None, path: "/user/authenticate", query: None, fragment: None }), source: Some(Custom { kind: ConnectionAborted, error: "Unexpected EOF" }) })

Even when using native-tls

[algesten]

@mq1 I assume you're using a POST or some such? Do you have some test code you can share with us?

[mq1]

@mq1 I assume you're using a POST or some such? Do you have some test code you can share with us?

I think this sums up my issue https://github.com/mq1/ureq-issue

isahc seems to work fine, I included its output.

My code can also be useful https://github.com/mq1/libmc/blob/4169838dfee9daf21aced2c2701a676748a0c24b/src/accounts.rs#L135

[algesten]

Closing since we're moving to ureq 3.x.

I've been running the 1_000_000 top websites myself with the 3.x branch. I don't see currently, but let's reopen if we encounter it again.