TIdHTTP cannot connect to https://www.implisit.com using TLS

[rlebeau]

Using OpenSSL 1.0.2d, TIdHttp is not able to retrieve any page from https://www.implisit.com, the following exception is raised:

EIdOSSLConnectError Error connecting with SSL.
EOF was observed that violates the protocol

The server is closing the connection as soon as TIdHTTP sends its SSL/TLS "Client Hello" packet, without sending anything back in reply. A web browser connects OK, though. So I Wiresharked the connections and see my browser sending a TLS 1.2 hello, so I configure TIdHTTP to enable only TLS 1.2 but it ends up sending a TLS 1.0 hello instead, even though it actually does initialize TLS 1.2. I am guessing that there is a missing flag/feature that Indy does not support/enable yet that is causing OpenSSL to downgrade to TLS 1.0. However, when I configure my browser to disable TLS 1.1 and 1.2, it sends a TLS 1.0 hello and still connects OK. But the server is apparently rejecting TIdHTTP's TLS 1.0 hello. I have no clue what OpenSSL is failing on at this point, there is nothing coming from the server to explain it.

[misterborden]

I got the same exception trying to get any page from https://alphavantage.co with TIdHTTP. I tried each of SSL2, SSL3, TLS1, TLS1.1, TLS1.2 to no avail, but if I changed the URL to https://www.slashdot.org or https://www.embarcadero.com it works fine. And of course, a browser connects with no problem. It seems like some sites are lax enough to ignore whatever is wrong but others aren't. My next step was going to be digging into it with Wireshark, but if the problem is inside Indy that seems like a dead end. I don't suppose there's been any progress on this?

[rlebeau]

Some. It seems to be (at least partially) related to the call to SSL_CTX_set_cipher_list(fContext, SSL_DEFAULT_CIPHER_LIST) in TIdSSLContext.InitContext(). Indy defaults to TLS 1.0, which means it also uses TLS 1.0 ciphers by default. If you want to connect using TLS 1.2 instead, update the IOHandler's CipherList to enable TLS 1.2 ciphers. Otherwise, the server disconnects as soon as it gets the "Client Hello".

On one test, I enabled TLS 1.2 ciphers (or just skipped the call to SSL_CTX_set_cipher_list(SSL_DEFAULT_CIPHER_LIST) altogether), and the handshake was able to get much further along, but it still failed at a later stage. But I can't seem to reproduce that result again, the server is back to disconnecting right away again.

For what's its worth, I can't connect to https://alphavantage.co using FireFox or Chrome, either.

[misterborden]

Thanks, you solved one problem for me and raised another ;) It seems alphavantage.co requires the subdomain www to be prepended instead of defaulting to it like a self-respecting website should. (Didn't realize that until I tried to follow your link.) When I changed the URL to https://www.alphavantage.co it no longer fails with the EOF exception. Now it fails with:

exception class EIdOSSLUnderlyingCryptoError with message 'Error connecting with SSL. error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error'.

And I tried to grab https://www.implisit.com but it times out. Apparently Salesforce.com bought them in 2016 so I doubt the server is still there.

Thanks again, back to the bug hunt...

[arvanus]

Just to inform that the same for https://websocket.org, I simply used a idHTTP and a TIdSSLIOHandlerSocketOpenSSL.

IdHTTP1.Get('https://websocket.org'); and I got:

First chance exception at $73E508C2. Exception class EIdOSSLConnectError with message
'Error connecting with SSL.
EOF was observed that violates the protocol'.
Process aaaaa.exe (10804)

Any news on this error? I`m using Delphi 10.2.2, with default indy. Thanks!

[misterborden]

Non that I'm aware of. I don't think Indy implements the SSL spec 100% correctly and some websites are strict about it and some aren't. I was starting to dig into the problem with wireshark when I realized I could get the data I needed from another website, I just had to deal with some cookie issues. I solved that in an hour so I never went back to find the problem with Indy.

[rlebeau]

The EOF error means the server disconnected the TCP connection mid-handshake, it didn't even bother to send a TLS alert first. In order to access 'https://websocket.org' without error, you need to enable TLS v1.1 in the TIdSSLIOHandlerSocketOpenSSL.SSLOptions.SSLVersions property (only TLS 1.0 is enabled by default), then it works fine for me, even with the default TLS v1.0 ciphers. I didn't try with TLS v1.2.

[arvanus]

Great! It worked for me, I don't know how I didn't tested changing it before, I was so focused in the error that forgot to test, thanks! In advance, I`m creating an simple (and open-sourced) client protocol implementation for Websockets, using TIdTCPClient, because I couldn't find none free. (I'll publish it here in github) Thanks!

[rlebeau]

It seems to be (at least partially) related to the call to SSL_CTX_set_cipher_list(fContext, SSL_DEFAULT_CIPHER_LIST) in TIdSSLContext.InitContext().

FYI, Indy no longer uses SSL_DEFAULT_CIPHER_LIST, since it doesn't keep up with OpenSSL updates. If the CipherList property is empty, Indy lets OpenSSL use whatever default list it wants. That change was made awhile ago.

Not sure if this ticket still needs to be open, due to other factors that may or may not still be active. Need to retest.

[mezen]

As I am writing a new OpenSSL IO Handler, I used these domains for testing: https://www.implisit.com/ does not work (after some time it produces a timeout) https://alphavantage.co/ does not work https://websocket.org/ works very well

Both not working website also do not work in my browser (same behaviour as in my test program). For alphavantage.co my FF says PR_END_OF_FILE_ERROR.

[Tommo195]

I have the same problem with an old legacy product (D2007 indy 10) but it works in 10.3. I really would like to fix the issue in the old version if possible. Wireshark shows TLSv1 for D2007 and TLS1.2 for 10.3 (same open ssl dll versions) and both set to sslvTLSv1_2 (same test project). I tried skipping the set cipher list with no luck, anything else I could try.

(10.3) (2007)

[rlebeau]

Both of those screenshots are showing TLS v1.0 being requested. So the problem is not with different Delphi/Indy versions using different TLS versions. It is that you are setting TLS v1.2 in your code but getting TLS v1.0 on the wire in both cases. You did not say which version of the OpenSSL DLLs you are actually using, but Indy will silently downgrade to TLS v1.0 if it can't access OpenSSL's TLS v1.2 APIs. When the failure occurs, does Indy's IsOpenSSL_TLSv1_2_Available() function in the IdSSLOpenSSLHeaders unit return True or False?

[Tommo195]

But notice at the at the very top it shows TLSv1.2 (10.3) that might be the key difference, but yes below that they both say 1.0. Originally had ssl 1.0.1j now using 1.0.2u same result (SSL3_GET_RECORD:wrong version number) for 2007 and working correctly for 10.3. Yes IsOpenSSL_TLSv1_2_Available() does return true.

[Tommo195]

After doing some reading, looks like the Record Layer version may get updated (in wireshark) based on the server hello if it succeeds, so that probably not relevant in this case. So both are saying max supported client version is 1.0 (I would expect the API server to reject both here, but this web API seems to accept 1.0 and return 1.2 to be use from there on). The handshake for both is 1.2. so the only obvious difference between the working and non working is the list of ciphers, both are using the SSL_DEFAULT_CIPHER_LIST ('AES:ALL:!aNULL:!eNULL:+RC4:@STRENGTH'). The working one has 74 suites and the non working one has 86 suites. So I will look at the differences, not sure why they are different as the SSL dll's are the same? Where do these actually come from?

(10.3 Working) (2007 not working)

[Tommo195]

Just noticed also the servername missing in the extensions SNI, could that be the problem?

[Tommo195]

Yes that was the problem (SNI), have added the code to do that in the old D2007. Fixed Thanks

[rlebeau]

As I said earlier, use of SSL_DEFAULT_CIPHER_LIST was removed from Indy 2 years ago. And client-side SNI was added 3 years ago (see https://github.com/IndySockets/Indy/issues/160). So make sure your D2007 is using an up-to-date version of Indy 10, don't try to hack an old version to make it work.

[Tommo195]

As I said earlier, use of SSL_DEFAULT_CIPHER_LIST was removed from Indy 2 years ago. And client-side SNI was added 3 years ago (see #160). So make sure your D2007 is using an up-to-date version of Indy 10, don't try to hack an old version to make it work.

I have added lots of small changes over many years, so for now have made changes for it to work. If I have time to revisit I will try an upgrade. Will the newer Indy version work with 2007 or does it need to be unicode 2009-> ?

[rlebeau]

Will the newer Indy version work with 2007

For now, yes. Indy 10 still supports all the way back to Delphi 5.

does it need to be unicode 2009-> ?

Indy 11, which is work-in-progress but has not been finalized yet, will drop support for Delphi 2007 and earlier (and also FreePascal 2.x and earlier), yes.

[Tommo195]

Thanks appreciate your help