search

eduardsui / tlse

Single C file TLS 1.2/1.3 implementation, using tomcrypt as crypto library
Other
535 stars 87 forks source link

TLSE fails to contact Cloudflare server, where curl works fine, I eliminated every cause I could think of #81

Closed vpltd-kgalaj closed 1 year ago

vpltd-kgalaj commented 1 year ago

Hi,

I am sorry to bother you again; last time you wrote that the TLS layer seemed ok, and to look at my HTTP headers, and I made sure to use ALPN 'h2', and for a time I thought it worked; but it turned out I had another bug in code, and it didn't.

So to make sure my bugs don't influence the result, and that it's not a problem with HTTP headers, I downloaded fresh TLSe and compiled tlsclienthello example: 'gcc tlsclienthello.c -o tlsclienthello -DTLS_AMALGAMATION'

Then I ran 'curl -v -H "Accept-Encoding: gzip, deflate" -H "Accept: application/json" --no-alpn https://emm-api.com:443/region/ --output out.txt' to ensure that: 1) it caused a correct response from the server, and 2) I was able to see what exact HTTP headers it provided in the call.

It provided:

GET /region/ HTTP/1.1 Host: emm-api.com User-Agent: curl/7.86.0 Accept-Encoding: gzip, deflate Accept: application/json

and that worked - the server replied correctly with HTTP/1.1 200 OK .

So I modified the tllsclienthello code:

commented: char ref_argv[] = {"", "google.com", "443"}; added: char ref_argv[] = {"", "emm-api.com", "443"};

commented: const char request = "GET / HTTP/1.1\r\nConnection: close\r\n\r\n"; added: const char request = "GET /region/ HTTP/1.1\r\nHost: emm-api.com\r\nuser-agent: curl/7.86.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\n\r\n";

Compiled it, ran it (even made sure the GET mentions curl, even though it's not curl, just to have the headers exactly the same), and the server replied:

HTTP/1.1 403 Forbidden Server: cloudflare Date: Fri, 10 Mar 2023 16:09:54 GMT Content-Type: text/html Content-Length: 151 Connection: keep-alive CF-RAY: 7a5cd324deecbfa6-WAW

So... curl is doing something right, TLSe is doing something wrong. Both are using HTTP/1.1, both have ALPN disabled, both are using the exact same HTTP request and headers. With ALPN enabled and set to 'h2', the result is the same, but the headers are different; but it works in HTTP/1.1, so I try to avoid complicating the picture.

TLSe is using the tomcrypt that came with it; I am trying to use it because we are already using a tomcrypt library in the project; I was wondering if the cause of the problem wasn't that it was not in any way modified for TLSe, but the version that comes with TLSe, if modified, undoubtedly is modified as it should be.

What else could be causing such a reply from the server? I don't see curl doing anything special to get the server to answer correctly. I am at loss.

eduardsui commented 1 year ago

Try this:

    struct TLSContext *context = tls_create_context(0, TLS_V13);
    tls_sni_set(context, "emm-api.com");

I get:

HTTP/1.1 200 OK
Date: Fri, 10 Mar 2023 16:47:03 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
cache-control: private
allow: GET, HEAD, OPTIONS
x-frame-options: SAMEORIGIN
vary: Cookie
x-cache-status: MISS
content-encoding: gzip
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=LQukE2gyOIcmyXXeqEv3jLm6LDMY60dj%2Bkw025bMBsmb8FGxi%2F%2FmBU%2Fn58S9OY8zBEwZkzq158QGMeLtFsiv1e2d7McwNOMAzkEAiLTHMDxBWiudASCQ59O8rLJxtg%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7a5d09909ee43826-FRA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

Also user-agent - I think it should be User-agent (I'm not sure if it's case sensitive or not).

E.

vpltd-kgalaj commented 1 year ago

It does work in this precise configuration, but I am worried.

Because it doesn't work when using TLS_V12, and it doesn't work when enabling ALPN with: tls_add_alpn(context, "http/1.1"); tls_add_alpn(context, "h2");

depending on what is enabled, it fails in different ways. For example, when I do both, it replies:

Initializing dependencies Message type: 15, length: 2 ALERT MESSAGE 02 28 Consumed -12 bytes ERROR IN CONSUME: -12 CANNOT SWITCH TO kTLS

which means that the handshake failed (so it doesn't matter what request I would send later).

And when I run:

curl -v -H "Accept-Encoding: gzip, deflate" -H "Accept: application/json" https://emm-api.com:443/region/ --output out.txt

I can see that it's using TLS 1.2, ALPN, ending up negotiating HTTP/2, and the server replies just fine:

Connected to emm-api.com (188.114.97.13) port 443 (#0)

So it's as if it worked by lucky accident. If I let it rest there, and I move it to production code, I am worried that it might start mysteriously failing at a random time.

vpltd-kgalaj commented 1 year ago

My last message came out... definitely inconsiderate, I think.

I am very grateful that there is such a thing as TLSE, it's working very well, supporting a huge number of options, and I wish I could someday do something like this of my own, and put it out for public to use.

I don't expect anything; I mean only to let you know that there are connection setups for which TLSE somehow can't communicate with that server, and curl, doing seemingly the exact same thing, has no problem (when I put in --no-alps, it also asks in HTTP/1.1 and the server replies as well). Only in case you were interested in plugging these holes.

Thank you very much :)

eduardsui commented 1 year ago

Don't worry, I'm not a princess :).

You've found a bug. I think I've run some tests and I forgot to re-enable ECDSA for TLS 1.2 client side.

Can you check now? (this works for me now), #38011de

    struct TLSContext *context = tls_create_context(0, TLS_V12);
    tls_sni_set(context, "emm-api.com");
    tls_add_alpn(context, "http/1.1");
    // tls_add_alpn(context, "h2");

Thank you

vpltd-kgalaj commented 1 year ago

I confirm that the setup with TLS_V12 and http/1.1 now also works .

There might still be something wrong somewhere, as the version with h2 negotiated through ALPN should also be working, using a request with HTTP/2 inside, like this:

const char* request = "GET /region/ HTTP/2\r\nHost: emm-api.com\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\n\r\n";

and it doesn't, and with curl it does.

eduardsui commented 1 year ago

From my knowledge, HTTP/2 can be negotiated only via ALPN. Once negotiated, you need to swtich to HTTP2 binary protocol. I think that is possible that cURL is doing this for you. I don't think that text requests (GET / HTTP/2) are allowed over TLS.

I've implemented HTTP/2 a few years ago and I remembed that only "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" is sent as text (my implementation is here: https://github.com/Devronium/ConceptApplicationServer/blob/master/core/server/Include/HTTP2.con).

vpltd-kgalaj commented 1 year ago

Ah, you're right. I forgot that the way curl shows the text content of those messages doesn't mean they're sent like that. I'll test further in the app itself, where those are being correctly produced by the rest of the system I am working with.

Thank you :)

eduardsui commented 1 year ago

You're welcome! Thank you for using TLSe.