search

net4people / bbs

Forum for discussing Internet censorship circumvention
3.45k stars 82 forks source link

Cloudflare has enabled ECH (staged) #393

Open ValdikSS opened 2 months ago

ValdikSS commented 2 months ago

Seems like Cloudflare has gradually started to enable Encrypted ClientHello support. You can see it on rutracker.org and bo0om.ru for example.

ECH was instroduced on Cloudflare several years ago but has been disabled after very short test period. I could not find any announcement except for in CF documentation:

Caution

ECH is disabled globally, and cannot currently be enabled in the Cloudflare Dashboard.

Starting in August, 2024, ECH will be gradually released on free zones. It will not be possible to disable it. A toggle will be added to the Cloudflare Dashboard at a later point before ECH is made available for other zone plans.

Rutracker now opens in Firefox 130 in Russia without any circumvention methods. Apparently Firefox 130 does not require DNS-over-HTTPS for ECH to work.

Cloudflare uses a single ECH key & configuration for all the domains

$ dig +short rutracker.org HTTPS
1 . alpn="h3,h2" ipv4hint=104.21.32.39,172.67.182.196 ech=AEX+DQBBYQAgACB/ZNpruUIOMT7U9iv5DLgTo+oHQ7RI7GeHwd0tbccrCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3031::6815:2027,2606:4700:3034::a
c43:b6c4

$ dig +short crypto.cloudflare.com HTTPS
1 . alpn="http/1.1,h2" ipv4hint=162.159.137.85,162.159.138.85 ech=AEX+DQBBYQAgACB/ZNpruUIOMT7U9iv5DLgTo+oHQ7RI7GeHwd0tbccrCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:7::a29f:8955,2606:4700:7:
:a29f:8a55
SvcParam: ech
    SvcParamKey: ech (5)
    SvcParamValue length: 71
    ECHConfigList length: 69
    ECHConfig: id=97 cloudflare-ech.com
        Version: 0xfe0d
        Length: 65
        HKPE Key Config
            Config Id: 97
            KEM Id: DHKEM(X25519, HKDF-SHA256) (32)
            Public Key length: 32
            Public Key: 7f64da6bb9420e313ed4f62bf90cb813a3ea0743b448ec6787c1dd2d6dc72b08
            Cipher Suites length: 4
            Cipher Suites (1 suite)
        Maximum Name Length: 0
        Public Name length: 18
        Public Name: cloudflare-ech.com
        Extensions length: 0

cloudflare-ech.com is used as a canary domain (SNI) for TLS requests

Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 587
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 583
            Version: TLS 1.2 (0x0303)
            Random: cb6244609a92947f8ccf9c7e731dc013cf00f739468bad266be3e9d4b40dfea4
            Session ID Length: 32
            Session ID: 2587cb2bc557eb6af6c561df526f2f42111108418712bfe550331cf0a4b1af53
            Cipher Suites Length: 34
            Cipher Suites (17 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 476
            Extension: server_name (len=23) name=cloudflare-ech.com
            Extension: extended_master_secret (len=0)
            Extension: renegotiation_info (len=1)
            Extension: supported_groups (len=14)
            Extension: ec_point_formats (len=2)
            Extension: application_layer_protocol_negotiation (len=14)
            Extension: status_request (len=5)
            Extension: delegated_credentials (len=10)
            Extension: key_share (len=107) x25519, secp256r1
            Extension: supported_versions (len=5) TLS 1.3, TLS 1.2
            Extension: signature_algorithms (len=24)
            Extension: record_size_limit (len=2)
            Extension: encrypted_client_hello (len=217)
                Type: encrypted_client_hello (65037)
                Length: 217
                Client Hello type: Outer Client Hello (0)
                Cipher Suite: HKDF-SHA256/AES-128-GCM
                    KDF Id: HKDF-SHA256 (1)
                    AEAD Id: AES-128-GCM (1)
                Config Id: 97
                Enc length: 32
                Enc: 08181d7e3ce624682fe03c8c31698f5a6c198aff850bccfedf6780e8dea6267e
                Payload length: 175
                Payload [truncated]: 20d54aebde0806f5ea62f287b713d9dba7e93636885160b2588633befe1e986046991c997bf9bd3e96927d99a3a3c0075870644ed6b9d0cd25e8da2ca197ee00907eef3955a5507b83bddecff3a720ad62ff577ac2b685ede87ae196c75e5f4c5ed02566350834bbc945b5f380
            [JA4: t13d1713h2_5b57614c22b0_748f4c70de1c]
            [JA4_r: t13d1713h2_002f,0035,009c,009d,1301,1302,1303,c009,c00a,c013,c014,c02b,c02c,c02f,c030,cca8,cca9_0005,000a,000b,000d,0017,001c,0022,002b,0033,fe0d,ff01_0403,0503,0603,0804,0805,0806,0401,0501,0601,0203,0201]
            [JA3 Fullstring: 771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-156-157-47-53,0-23-65281-10-11-16-5-34-51-43-13-28-65037,29-23-24-25-256-257,0]
            [JA3: ed3d2cb3d86125377f5a4d48e431af48]
boredcircuit commented 1 month ago

image

https://support.mozilla.org/en-US/kb/faq-encrypted-client-hello#w_can-enterprises-disable-ech

wkrp commented 1 month ago

Linking the past thread #292. Cloudflare had first deployed ECH a year ago, on 2023-09-29, but then disabled it again on 2023-10-11.

2023-09-29 https://blog.cloudflare.com/announcing-encrypted-client-hello/ (archive)

Today we are excited to announce a contribution to improving privacy for everyone on the Internet. Encrypted Client Hello, a new proposed standard that prevents networks from snooping on which websites a user is visiting, is now available on all Cloudflare plans.

2023-10-11 https://community.cloudflare.com/t/early-hints-and-encrypted-client-hello-ech-are-currently-disabled-globally/567730 (archive)

This note is to inform you of the status of Early Hints and Encrypted Client Hello.

We have sadly had to disable both of these features globally whilst we address a number of issues with them. These issues are unrelated. We are in the process of adding a label to each of the toggles in dashboard to alert that they are disabled.

We expect to re-enable Early Hints in the coming weeks, with ECH re-enablement for Free coming later in the year with roll-out to those in the beta in early 2024.

maoist2009 commented 1 month ago

Seems like Cloudflare has gradually started to enable Encrypted ClientHello support. You can see it on rutracker.org and bo0om.ru for example.

ECH was instroduced on Cloudflare several years ago but has been disabled after very short test period. I could not find any announcement except for in CF documentation:

Caution ECH is disabled globally, and cannot currently be enabled in the Cloudflare Dashboard. Starting in August, 2024, ECH will be gradually released on free zones. It will not be possible to disable it. A toggle will be added to the Cloudflare Dashboard at a later point before ECH is made available for other zone plans.

Rutracker now opens in Firefox 130 in Russia without any circumvention methods. Apparently Firefox 130 does not require DNS-over-HTTPS for ECH to work.

Cloudflare uses a single ECH key & configuration for all the domains

$ dig +short rutracker.org HTTPS
1 . alpn="h3,h2" ipv4hint=104.21.32.39,172.67.182.196 ech=AEX+DQBBYQAgACB/ZNpruUIOMT7U9iv5DLgTo+oHQ7RI7GeHwd0tbccrCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3031::6815:2027,2606:4700:3034::a
c43:b6c4

$ dig +short crypto.cloudflare.com HTTPS
1 . alpn="http/1.1,h2" ipv4hint=162.159.137.85,162.159.138.85 ech=AEX+DQBBYQAgACB/ZNpruUIOMT7U9iv5DLgTo+oHQ7RI7GeHwd0tbccrCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:7::a29f:8955,2606:4700:7:
:a29f:8a55
SvcParam: ech
    SvcParamKey: ech (5)
    SvcParamValue length: 71
    ECHConfigList length: 69
    ECHConfig: id=97 cloudflare-ech.com
        Version: 0xfe0d
        Length: 65
        HKPE Key Config
            Config Id: 97
            KEM Id: DHKEM(X25519, HKDF-SHA256) (32)
            Public Key length: 32
            Public Key: 7f64da6bb9420e313ed4f62bf90cb813a3ea0743b448ec6787c1dd2d6dc72b08
            Cipher Suites length: 4
            Cipher Suites (1 suite)
        Maximum Name Length: 0
        Public Name length: 18
        Public Name: cloudflare-ech.com
        Extensions length: 0

cloudflare-ech.com is used as a canary domain (SNI) for TLS requests

Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 587
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 583
            Version: TLS 1.2 (0x0303)
            Random: cb6244609a92947f8ccf9c7e731dc013cf00f739468bad266be3e9d4b40dfea4
            Session ID Length: 32
            Session ID: 2587cb2bc557eb6af6c561df526f2f42111108418712bfe550331cf0a4b1af53
            Cipher Suites Length: 34
            Cipher Suites (17 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 476
            Extension: server_name (len=23) name=cloudflare-ech.com
            Extension: extended_master_secret (len=0)
            Extension: renegotiation_info (len=1)
            Extension: supported_groups (len=14)
            Extension: ec_point_formats (len=2)
            Extension: application_layer_protocol_negotiation (len=14)
            Extension: status_request (len=5)
            Extension: delegated_credentials (len=10)
            Extension: key_share (len=107) x25519, secp256r1
            Extension: supported_versions (len=5) TLS 1.3, TLS 1.2
            Extension: signature_algorithms (len=24)
            Extension: record_size_limit (len=2)
            Extension: encrypted_client_hello (len=217)
                Type: encrypted_client_hello (65037)
                Length: 217
                Client Hello type: Outer Client Hello (0)
                Cipher Suite: HKDF-SHA256/AES-128-GCM
                    KDF Id: HKDF-SHA256 (1)
                    AEAD Id: AES-128-GCM (1)
                Config Id: 97
                Enc length: 32
                Enc: 08181d7e3ce624682fe03c8c31698f5a6c198aff850bccfedf6780e8dea6267e
                Payload length: 175
                Payload [truncated]: 20d54aebde0806f5ea62f287b713d9dba7e93636885160b2588633befe1e986046991c997bf9bd3e96927d99a3a3c0075870644ed6b9d0cd25e8da2ca197ee00907eef3955a5507b83bddecff3a720ad62ff577ac2b685ede87ae196c75e5f4c5ed02566350834bbc945b5f380
            [JA4: t13d1713h2_5b57614c22b0_748f4c70de1c]
            [JA4_r: t13d1713h2_002f,0035,009c,009d,1301,1302,1303,c009,c00a,c013,c014,c02b,c02c,c02f,c030,cca8,cca9_0005,000a,000b,000d,0017,001c,0022,002b,0033,fe0d,ff01_0403,0503,0603,0804,0805,0806,0401,0501,0601,0203,0201]
            [JA3 Fullstring: 771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-156-157-47-53,0-23-65281-10-11-16-5-34-51-43-13-28-65037,29-23-24-25-256-257,0]
            [JA3: ed3d2cb3d86125377f5a4d48e431af48]

You can't prove that ECH works. In China, HTTP/3 (UDP) won't be banned by the GFW. However, chromium will try HTTP/2 first and use HTTP/3 if the website tells that the quic is available. But http/2 will be blocked.

But,
If you enable use aplans of the result of dns(this seems to be automatically enabled on the latest chromium), chrome may try quic first when the doh returns

alpn="h3,h2"

and then, you can visit the website.

So you can't prove that ECH works unless you banned UDP or you use wireshark.

0x391F commented 1 month ago

So you can't prove that ECH works unless you banned UDP or you use wireshark.

No, you can! Attach /cdn-cgi/trace to domain, for example, https://www.cloudflare.com/cdn-cgi/trace, then if sni=encrypted, it indicate that ECH work.

ValdikSS commented 1 month ago

You can't prove that ECH works.

That's a real world observation on Firefox 130. The blocked website unexpectedly started to be reachable on its own, without any circumvention tools, and I debugged why does it work. AFAIR it also works on Edge, haven't tried Chrome.

ECH should work on TCP and QUIC, regardless of HTTP version. That's "just" a TLS extension.

ValdikSS commented 1 month ago

Also thing to note is that in about:support on Firefox 130 right now I see the following study ("Remote Features", a dynamic code update pushed by Mozilla remotely): Encrypted Client Hello - Fallback Mechanism. Maybe it also changes the behavior.

RPRX commented 1 month ago

cloudflare-ech.com 是必要的吗

Is cloudflare-ech.com necessary?

0x391F commented 1 month ago

Some known hosts which have been supports ECH. You can also use about:networking#dnslookuptool to test on Firefox. If it shows echConfig=, indicate that the website supports ECH. Note: you need to enable DoH (DNS over HTTPS) first: https://codecguide.com https://dvddecrypter.org.uk https://qbittorrent.org https://rutracker.org https://sandboxie-plus.com https://servo.org https://www.softether.org https://transmissionbt.com https://whynotwin11.org

0x391F commented 1 month ago

cloudflare-ech.com 是必要的吗

不是啊，这个是外层的明文 SNI，似乎全部启用了 ECH 的、使用 Cloudflare 网站的外层 SNI 都是一样的。 No, this is outer plaintext SNI, it seems that all website which is using Cloudflare and enabled ECH, their outer SNI are all the same.

RPRX commented 1 month ago

cloudflare-ech.com 是必要的吗

不是啊，这个是外层的明文 SNI，似乎全部启用了 ECH 的、使用 Cloudflare 网站的外层 SNI 都是一样的。 No, this is outer plaintext SNI, it seems that all website which is using Cloudflare and enabled ECH, their outer SNI are all the same.

我是说 Cloudflare 是否强制使用这个 SNI，因为 GFW 可以封掉境外 DoH 并封掉这个 SNI

I mean, does Cloudflare force the use of this SNI, because the GFW can block overseas DoH and block this SNI?

0x391F commented 1 month ago

cloudflare-ech.com 是必要的吗

不是啊，这个是外层的明文 SNI，似乎全部启用了 ECH 的、使用 Cloudflare 网站的外层 SNI 都是一样的。 No, this is outer plaintext SNI, it seems that all website which is using Cloudflare and enabled ECH, their outer SNI are all the same.

我是说 Cloudflare 是否强制使用这个 SNI，因为 GFW 可以封掉境外 DoH 并封掉这个 SNI

很遗憾，是的。 Unfortunately, yes.

wkrp commented 1 month ago

The "public_name" cloudflare-ech.com comes from the ECHConfigList structure that is stored in DNS.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#section-4

public_name

The DNS name of the client-facing server, i.e., the entity trusted to update the ECH configuration. This is used to correct misconfigured clients, as described in Section 6.1.6.

See Section 6.1.7 for how the client interprets and validates the public_name.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#section-6.1-7.5.1

  1. [The client] SHOULD place the value of ECHConfig.contents.public_name in the "server_name" extension. Clients that do not follow this step, or place a different value in the "server_name" extension, risk breaking the retry mechanism described in Section 6.1.6 or failing to interoperate with servers that require this step to be done; see Section 7.1.
$ dig -t https +short rutracker.org
1 . alpn="h3,h2" ipv4hint=104.21.32.39,172.67.182.196 ech=AEX+DQBBNgAgACAEpOgP1JoIvzIHu8T1iZtXOUvD0aaMV8ur6jhOUOrtCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3031::6815:2027,2606:4700:3034::ac43:b6c4

$ echo 'AEX+DQBBNgAgACAEpOgP1JoIvzIHu8T1iZtXOUvD0aaMV8ur6jhOUOrtCAAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=' | base64 -d | xxd
00000000: 0045 fe0d 0041 3600 2000 2004 a4e8 0fd4  .E...A6. . .....
00000010: 9a08 bf32 07bb c4f5 899b 5739 4bc3 d1a6  ...2......W9K...
00000020: 8c57 cbab ea38 4e50 eaed 0800 0400 0100  .W...8NP........
00000030: 0100 1263 6c6f 7564 666c 6172 652d 6563  ...cloudflare-ec
00000040: 682e 636f 6d00 00                        h.com..

The ECH specification also says that clients must not retry without ECH, if the ECH connection is blocked.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#section-8.1.1

Unless ECH is disabled as a result of successfully establishing a connection to the public name, the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI. It MAY attempt to use another server from the DNS results, if one is provided.

wkrp commented 1 month ago

You can also use about:networking#dnslookuptool to test on Firefox. If it shows echConfig=, indicate that the website supports ECH.

For Cloudflare sites, another way to check is to request the URL path /cdn-cgi/trace. It will say either sni=plaintext or sni=encrypted. E.g. https://rutracker.org/cdn-cgi/trace.

EDIT: This trick was already posted above, my mistake.

JonSnowWhite commented 1 month ago

cloudflare-ech.com 是必要的吗

不是啊，这个是外层的明文 SNI，似乎全部启用了 ECH 的、使用 Cloudflare 网站的外层 SNI 都是一样的。 No, this is outer plaintext SNI, it seems that all website which is using Cloudflare and enabled ECH, their outer SNI are all the same.

我是说 Cloudflare 是否强制使用这个 SNI，因为 GFW 可以封掉境外 DoH 并封掉这个 SNI

很遗憾，是的。 Unfortunately, yes.

I just checked, Cloudflare enforces the outer SNI cloudflare-ech.com and fails with an encrypted HANDSHAKE_FAILURE fatal alert when any other hostname is provided. Edit: removing the outer SNI is also rejected

There were discussions about the necessity of a specific outer SNI during ECH development. It was ultimately decided to keep it to allow the server to handshake the outer SNI/ClientHello if it rejects the inner ClientHello.

Sadly, this necessity makes honest ECH distinguishable from GREASE ECH by crawling for publicly available echConfigs and their outer SNI hostnames.

RPRX commented 1 month ago

或许 IETF 的本意是以 SNI 作为不同 ECH 密钥对的区分标识，但 Cloudflare 整个 IP 都是自己的，强制使用某个 SNI 没有必要，有人去建议一下取消这种强制吗，还是说这是 Cloudflare 为了防止被完全封锁而做出的妥协之举？（它甚至与中国企业有合作）

不过 Cloudflare 可能不会改，因为在 ECH 之前它已停止支持域前置

Perhaps the IETF's original intention was to use SNI as a distinguishing identifier for different ECH key pairs, but Cloudflare owns the entire IP, and there is no need to force the use of a specific SNI. Can someone suggest that this restriction be lifted? Or is this a compromise made by Cloudflare to prevent being completely blocked? (It even has partnerships with Chinese companies.)

However, Cloudflare is unlikely to change, as it stopped supporting domain fronting before ECH.

JonSnowWhite commented 1 month ago

The ECH draft allows servers to reject different SNI hostnames than the one specified in the ECH Config:

Once the server has chosen the correct ECHConfig, it MAY verify that
   the value in the ClientHelloOuter "server_name" extension matches the
   value of ECHConfig.contents.public_name, and abort with an
   "illegal_parameter" alert if these do not match.  This optional check
   allows the server to limit ECH connections to only use the public SNI
   values advertised in its ECHConfigs.  The server MUST be careful not
   to unnecessarily reject connections if the same ECHConfig id or
   keypair is used in multiple ECHConfigs with distinct public names.

Other providers than Cloudflare might then be more lenient. I also suspect Cloudflare won't change their behavior as they are otherwise very strict about SNI usage.

kovalensky commented 1 month ago

There's a convenient addon for firefox which adds ECH support indicator to your address bar, no need for manually checking it.

Also it tells if the following requests from the site used ECH.

2024-09-25_102237

maoist2009 commented 1 month ago

或许 IETF 的本意是以 SNI 作为不同 ECH 密钥对的区分标识，但 Cloudflare 整个 IP 都是自己的，强制使用某个 SNI 没有必要，有人去建议一下取消这种强制吗，还是说这是 Cloudflare 为了防止被完全封锁而做出的妥协之举？（它甚至与中国企业有合作）

不过 Cloudflare 可能不会改，因为在 ECH 之前它已停止支持域前置

Perhaps the IETF's original intention was to use SNI as a distinguishing identifier for different ECH key pairs, but Cloudflare owns the entire IP, and there is no need to force the use of a specific SNI. Can someone suggest that this restriction be lifted? Or is this a compromise made by Cloudflare to prevent being completely blocked? (It even has partnerships with Chinese companies.)

However, Cloudflare is unlikely to change, as it stopped supporting domain fronting before ECH.

不是停止支持。是停止支持Free plan.

It's not that they discontinued support. They discontinued support for the free plan.

UjuiUjuMandan commented 1 month ago

They discontinued support for the free plan.

What you are talk about is not domain fronting, but the legacy compatibility of no SNI. You can only omit the SNI but the HTTP Host header must be the site owner.

They prohibited you from pretending to be connecting to another site.

ValdikSS commented 1 month ago

found out that on Windows fox uses the DNS API to understand whether DoH is registered in the system And it only works out of the box on Win11 On Win10 - no

if fox does not detect or does not want to detect DoH on the system, it requires local DoH from itself. if the DoH is nowhere to be found, it refuses the ECH

They write that on Win10 the detection is disabled due to some bug in Windows

on linux there is no way to detect the DoH layer in a universal way, so ECHs are always enabled

https://ntc.party/t/encrypted-clienthello-ech-is-now-enabled-on-cloudflare/10075/39

ValdikSS commented 1 month ago

The flag in the control panel just controls the DNS record. Since the ECH key is the same for all the domains, you can still connect to any (?) domain even if it has ECH disabled.

$ dig +short HTTPS gdeposylka.ru
1 . alpn="h3,h2" ipv4hint=104.25.106.6,104.25.107.6,172.67.80.113 ipv6hint=2606:4700:20::6819:6a06,2606:4700:20::6819:6b06,2606:4700:20::ac43:5071

$ (echo -e "GET / HTTP/1.0\r\nHost: gdeposylka.ru\r\n\r\n"; sleep 10) | ./openssl s_client -CApath /etc/ssl/certs/ -no_ssl3 -no_tls1 -no_tls1_1 -no_tls1_2 -connect gdeposylka.ru:443 -servername gdeposylka.ru -ech_config_list 'AEX+DQBBVgAgACC91NEyBX4eLKQ/XSZk9DabQ/MbtpGNoDC3hOhS7QPtNQAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=' -ech_alpn_outer outer,public,h2 -alpn inner,secret,http/1.1

Setting new_session_cb
Connecting to 2606:4700:20::6819:6a06
CONNECTED(00000003)
ECH client callback printing: 
SSL_ech_print
s=0x560765ac9410
ech_attempted=1
ech_attempted_type=0xfe0d
ech_atttempted_cid=0x56
ech_done=1
ech_grease=0
HRR=0
ech_returned=0x0
ech_returned_len=0
ech_backend=0
ech_success=1
1 ECHConfig value loaded
cfg(0): [fe0d,56,cloudflare-ech.com,0020,[0001,0001],bdd4d132057e1e2ca43f5d2664f4369b43f31bb6918da030b784e852ed03ed35,00,00]

depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R4
verify return:1
depth=1 C=US, O=Google Trust Services, CN=WE1
verify return:1
depth=0 CN=gdeposylka.ru
verify return:1
…
kovalensky commented 1 week ago

All Cloudflare ECH-enabled services are blocked in Russia. It may be identified by DPI using the cloudflare-ech.com value in the handshake.

Adding this domain to the blocklist in evading tools seems to fix the problem.

The solution for desperate administrators could be to disable TLS 1.3 on the dashboard. Although this is not recommended in the long run for performance, security reasons.

hellais commented 1 week ago

Thanks for sharing this information.

All Cloudflare ECH-enabled services are blocked in Russia. It may be identified by DPI using the cloudflare-ech.com value in the handshake.

Can you share some more details and possibly data supporting the claim that "All Cloudflare ECH-enabled services are blocked"? The cloudflare-ech.com domain has been added to OONI testing, however we do not see it blocked in the 23 networks it's been tested in so far: https://explorer.ooni.org/chart/mat?probe_cc=RU&since=2024-10-08&until=2024-11-08&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&domain=cloudflare-ech.com.

Do you know for example if this is only happening for users that have enabled Encrypted DNS in their browser? Firefox, for example, will use ECH if you have DNS over HTTPS enabled, but otherwise not.

If you could share some of these sites that have started to be blocked in Russia recently that it's speculated is a result of ECH rollout, we can try to take a look at them.

The blocking of only the cloudflare-ech.com SNI would be the happy case. What would be much more concerning is if the blocking were to be affecting ECH as a whole.

We have an upcoming test in OONI Probe that should be able to test this, so it would be useful to collect some information on that once we have it out: https://github.com/ooni/probe/issues/1453.

ValdikSS commented 1 week ago

They specifically block Cloudflare's ECH, it is detected by the presence of SNI = cloudflare-ech.com and ECH extension in ClientHello. Just SNI = cloudflare-ech.com without ECH extension is not affected, as well as ECH grease without SNI = cloudflare-ech.com.

Both HTTP2 (TCP) and HTTP3 (QUIC) are getting filtered.

ECH specification requires the user-agent to retry the connection without ECH if it fails with ECH for some reason, but the particular method Russia uses to filter prohibited requests on TSPU government boxes leads to very long timeout (about a minute) before the connection is marked as failed.

TSPU "freezes" the connection, all subsequent packets are dropped after prohibited connection is detected, instead of tearing it down or sending TLS alert, or anything more appropriate. This is generally not the case of commercial DPI systems, which also are still used in Russia.

ValdikSS commented 1 week ago

"Centre of Monitor and Control of the Internet" has issued the following statement:

We recommend you to opt out of the CloudFlare CDN services

CloudFlare, the American CDN service provider company, has enabled TLS ECH (Encrypted Client Hello) extension to be used by default on its servers in October. This technology is meant to bypass restrictions on access to information prohibited in Russia. Its use violates Russian legislation and is being limited using technical means of countering threats (TSPU).

We recommend that owners of informational resources disable the TLS ECH extension or, even better, use domestic CDN services that ensure reliable and secure operation of resources and protection from computer attacks.

In particular, protection against DDoS attacks can be provided by the National System for Combating DDoS Attacks (NSPA). During its operation (since March 2024), more than 10.5 thousand DDoS attacks on various organizations in the country were repelled.

Please note that CloudFlare was one of the BigTech companies that the US State Department brought together in September to discuss comprehensive and organized counteraction to countries that actively defend their information sovereignty (source).

https://cmu.gov.ru/ru/news/2024/11/07/рекомендуем-отказаться-от-cdn-сервиса-cloudflare/

It's rare to see blocking confirmation at all, let alone written in a direct non-bureaucratic language.

hellais commented 1 week ago

SNI = cloudflare-ech.com and ECH extension in ClientHello

Does this mean that having the cloudflare-ech.com SNI is not enough to trigger the blocking behavior and that the ECH extension needs to be also advertised in the ClientHello in addition to that SNI?

user-agent to retry the connection without ECH if it fails with ECH for some reason

According to the spec (https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-handshaking-with-clienthello) clients MUST not retry without ECH if it fails, which I believe is to mitigate against middleboxes trying to downgrade clients to non-ECH connections.

Probably the reason why people are experiencing difficulty accessing ECH enabled services is that Client do not failover to trying vanilla TLS 1.3 (I have yet to actually test if this is the behaviour on FF and Chrome).

ValdikSS commented 1 week ago

Do you know for example if this is only happening for users that have enabled Encrypted DNS in their browser? Firefox, for example, will use ECH if you have DNS over HTTPS enabled, but otherwise not.

I use Firefox 131 on Linux, and I have ECH enabled even without encrypted ECH. This is the case on my Windows VM as well.

ValdikSS commented 1 week ago

Does this mean that having the cloudflare-ech.com SNI is not enough to trigger the blocking behavior and that the ECH extension needs to be also advertised in the ClientHello in addition to that SNI?

That is correct (just edited the message with more info).

ValdikSS commented 1 week ago

clients MUST not retry without ECH if it fails

Hrm, indeed. 8.1.1:

Unless ECH is disabled as a result of successfully establishing a connection to the public name, the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI. It MAY attempt to use another server from the DNS results, if one is provided.

It seems I'm wrong. Maybe the previous drafts had some sort of fallback, as I remember reading about it.
I don't have ECH enabled explicitly, I don't have DoH enabled in the browser, yet Firefox retries without ECH after about a minute.

wkrp commented 1 week ago

Let's move the discussion of Cloudflare ECH blocking in Russia to a dedicated thread:

I summarized the information from this thread there.

wkrp commented 1 week ago

Apparently Firefox 130 does not require DNS-over-HTTPS for ECH to work.

According to the Mozilla wiki, the change to not require DoH happened in Firefox 129.

https://wiki.mozilla.org/index.php?title=Security/Encrypted_Client_Hello&oldid=1251602 https://wiki.mozilla.org/index.php?title=Security/Encrypted_Client_Hello&diff=1251602&oldid=1248531

Dependency on DoH

Originally, Firefox required DoH to be enabled in order for ECH to function. Since Firefox 129, Firefox can fetch the necessary information via the OS DNS Resolver to enable ECH, allowing ECH to be used in more circumstances. Due a blocking bug with the MacOS DNS integration, MacOS still requires DoH to be enabled for ECH to be used, the work to fix this is tracked in 1882856.

For the vast majority of users, their native OS resolver will use unencrypted DNS to contact their local router which in turn passes their queries unencrypted on to their network provider. Fetching ECHConfigs via unencrypted DNS means that the sites the user visits are still leaked in plaintext to the network and so ECH delivers less value in this scenario. For this reason, we recommend the use of DoH (whether with a self-hosted or external DoH service) in order to benefit fully from the privacy protections of ECH.

I found this from a post on NTC.