Blocking of Cloudflare ECH in Russia, 2024-11-05

[wkrp]

[Discussion moved from https://github.com/net4people/bbs/issues/393#issuecomment-2461603654. NTC threads are https://ntc.party/t/12837 (technical information) and https://ntc.party/t/12732 (discussion).]

Cloudflare's deployment of Encrypted Client Hello (ECH) is blocked in multiple networks in Russia since 2024-11-05. The blocking trigger is the presence of both of the following two elements in the Client Hello:

1. An SNI extension with the value cloudflare-ech.com.
2. An ECH extension.

Neither of these elements on its own is sufficient. That is, an SNI of cloudflare-ech.com without an ECH extension is not blocked, and ECH extensions that use an SNI other than cloudflare-ech.com are not blocked. In particular, you can still make ECH connections to servers that use a different public_name, such as defo.ie and tls-ech.dev; and GREASE ECH with SNI different from cloudflare-ech.com is not blocked.

Both TCP-based HTTP/2 and UDP-based HTTP/3 (QUIC) are affected. The blocking mechanism is packet dropping on the connection after the signature is detected (i.e., not a TCP RST or other overt teardown).

It seems that Firefox, at least, will retry the connection without ECH after a long delay (about a minute). Such a fallback to plaintext SNI apparently violates the ECH specification: "the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI."

The blocking of ECH was officially acknowledged in a notice from the Public Communications Network Monitoring and Control Center (ЦМУ ССОП):

https://cmu.gov.ru/ru/news/2024/11/07/рекомендуем-отказаться-от-cdn-сервиса-cloudflare/ (archive)

Рекомендуем отказаться от CDN-сервиса CloudFlare

Американская компания CloudFlare, поставщик услуг CDN, включила в октябре применение по умолчанию на своих серверах расширение TLS ECH (Encrypted Client Hello). Эта технология – средство обхода ограничений доступа к запрещенной в России информации. Его использование нарушает российское законодательство и ограничивается техническими средствами противодействия угрозам (ТСПУ).

Рекомендуем владельцам информационных ресурсов отключить расширение TLS ECH или, что правильнее, использовать отечественные CDN-сервисы, которые обеспечивают надежное и безопасное функционирование ресурсов и защиту от компьютерных атак.

В частности, защиту от DDoS-атак может обеспечить Национальная система противодействия DDoS-атакам (НСПА). За время ее работы (с марта 2024 года) отражено более 10,5 тыс. DDoS-атак на различные организации страны.

Обращаем внимание, что CloudFlare была одной из компаний BigTech, которые собирал Госдеп США в сентябре для обсуждения комплексного и организованного противодействия странам, активно защищающим свой информационный суверенитет (источник).

It is recommended to opt out of CloudFlare's CDN service

The American company CloudFlare, a provider of CDN services, in October enabled the default use of the TLS ECH (Encrypted Client Hello) extension on its servers. This technology is a means of circumventing restrictions on access to information banned in Russia. Its use violates Russian law and is restricted by the Technical Measure to Combat Threats (TSPU).

We recommend that owners of information resources disable the TLS ECH extension or, more correctly, use domestic CDN services that ensure reliable and secure functioning of resources and protection from computer attacks.

In particular, protection from DDoS attacks can be provided by the National System for Countering DDoS Attacks (NSPA). During its operation (since March 2024), more than 10.5 thousand DDoS-attacks on various organizations of the country have been reflected.

Note that CloudFlare was one of the BigTech companies that the U.S. State Department gathered in September to discuss a comprehensive and organized response to countries actively defending their information sovereignty (source).

OONI tests web connectivity to the SNI cloudflare-ech.com, and similarly GlobalCheck, but the measurements do not show blocking. That is because these tests do not have the other necessary part of the signature, the ECH extension. OONI is working on a dedicated ECH test.

https://github.com/net4people/bbs/issues/393#issuecomment-2462121200

Can you share some more details and possibly data supporting the claim that "All Cloudflare ECH-enabled services are blocked"? The cloudflare-ech.com domain has been added to OONI testing, however we do not see it blocked in the 23 networks it's been tested in so far: https://explorer.ooni.org/chart/mat?probe_cc=RU&since=2024-10-08&until=2024-11-08&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&domain=cloudflare-ech.com.

…

We have an upcoming test in OONI Probe that should be able to test this, so it would be useful to collect some information on that once we have it out: ooni/probe#1453.

[wkrp]

Other reporting and discussion.

• Habr: Роскомнадзор, вероятно, начал блокировать подключение к CloudFlare с использованием TLS ECH (Encrypted Client Hello) Roskomnadzor has probably started blocking connections to CloudFlare using TLS ECH (Encrypted Client Hello) (archive)

  Show text

  Роскомнадзор, вероятно, начал блокировать подключение к CloudFlare с использованием TLS ECH (Encrypted Client Hello)

  С полуночи сразу в нескольких российских локациях наблюдается следующая ситуация: Chrome отваливается по таймауту при попытке зайти на любой из множетсва сайтов, проксируемых через CloudFlare с включенным TLS 1.3 (ECH это расширение TLS 1.3). При этом те же самые сайты с той же самой машины можно скачать с использованием wget, который не поддерживает ECH. После выключения TLS 1.3 на стороне CloudFlare сайты становятся доступны через несколько минут (отключение TLS 1.3 отключает и ECH тоже). Сайты с TLS 1.2 и ниже работают как обычно. Воспроизводится из нескольких локаций у разных операторов на примерно сотне сайтов. Через VPN те же сайты всё это время работают нормально. Среди попавших под раздачу оказались: https://openstreetmap.org https://diary.ru https://kopilkaurokov.ru https://uchitelya.com https://opensubtitles.org Из 10000 самых трафиковых сайтов по версии рейтинга alexa.com на CloudFlare заведены примерно 2500. Из них примерно 700 имеют в DNS запись, информирующую о поддержке ECH.

  Roskomnadzor has probably started blocking connections to CloudFlare using TLS ECH (Encrypted Client Hello)

  Since midnight the following situation is observed in several Russian locations at once: Chrome crashes on timeout when trying to access any of the many sites proxied through CloudFlare with TLS 1.3 enabled (ECH is an extension of TLS 1.3). At the same time, the same sites from the same machine can be downloaded using wget, which does not support ECH. After disabling TLS 1.3 on the CloudFlare side, the sites become available in a few minutes (disabling TLS 1.3 disables ECH too). Sites with TLS 1.2 and below work as normal. Reproduced from several locations at different operators on about a hundred sites. The same sites have been working fine via VPN all this time. Among the sites that got hit were: https://openstreetmap.org https://diary.ru https://kopilkaurokov.ru https://uchitelya.com https://opensubtitles.org Of the 10,000 highest traffic sites according to alexa.com rankings, about 2,500 are hosted on CloudFlare. Of these, about 700 have a DNS record informing about ECH support.

• Cloudflare Community: Cannot access sites from Russia

  Hello. In Russia, websites through cloudflare have not been working, for several hours now

  In Russia, because of the activities of the RKN, sites are inaccessible due to the ECH, but the fact is that on free tariffs it cannot be turned off, which means that a huge number of sites are inaccessible.

[jesowozapoc]

Since they are "suggesting" people to move to local alternatives to cloudflare. How likely it is that they gonna block all of the ip addresses of cloudflare?

[hellais]

Thanks for summarizing this information.

The upcoming ECH test in OONI, although it only supports the GREASEd ECH extension, based on what has been reported so far should be enough to test for this behaviour.

Based on the input in this thread we are adding support for testing a different server_name values in the OuterClientHello that are not cloudflare-ech.com to check if it would be enough for cloudflare to deploy a different echconfig in order to avoid the block.

as per: https://datatracker.ietf.org/doc/html/draft-ietf-tls-esni-22#section-6.1-6

It SHOULD place the value of ECHConfig.contents.public_name in the "server_name" extension. Clients that do not follow this step, or place a different value in the "server_name" extension, risk breaking the retry mechanism described in Section 6.1.6 or failing to interoperate with servers that require this step to be done; see Section 7.1.

the value inside of the OuterClientHello server_name should be determined by the public_name value of the echconfig and so it can be changed just at the DNS level.

Here is the diff of the changes to the test: https://github.com/ooni/probe-cli/pull/1658 and specification: https://github.com/ooni/spec/pull/297

If there are any thoughts on what we might want to do additionally to this, let us know. We are planning to ship this release of the measurement engine next week.

[wkrp]

It seems that Firefox, at least, will retry the connection without ECH after a long delay (about a minute). Such a fallback to plaintext SNI apparently violates the ECH specification: "the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI."

From what I've been able to gather from asking on NTC, it seems that both Firefox and Chrome have a fallback to retry an ECH connection without ECH. Firefox falls back automatically after a certain amount of time. Chrome falls back after clicking the Reload button multiple times, but the fallback is disabled if DoH is configured. Treat these observations as preliminary.

Is anyone an expert at searching Bugzilla or the Chromium Issue Tracker, or exploring the source code of Firefox or Chromium, who can find where these apparent non-ECH fallback policies might be enacted?

• In Firefox, the browser internally retries the connection (without the user clicking the Reload button), and eventually reconnects without ECH, after about 40 to 60 seconds. This happens whether or not the browser is configured to use DoH.

  https://ntc.party/t/12837/1

  В Firefox 131 сайты с ECH открываются без ECH через минуту «загрузки» — открывается новое соединение с plain SNI, а старое закрывается. Причину не знаю, возможно, это из-за того, что у меня отключён DNS-over-HTTPS, но ECH всё равно используется.

  In Firefox 131 sites with ECH open without ECH after a minute of "loading" — a new connection with plain SNI is opened, and the old one is closed. I don't know the reason, maybe it's because I have DNS-over-HTTPS disabled, but ECH is still used.

  https://ntc.party/t/12732/310

  for example : https://www.hwinfo.com/download

  windows 10 ltsc / firefox / cf dns on router / cf doh in browser first attempt browser tries to open with ech:

  Client Hello (SNI=cloudflare-ech.com)

  then after few retransmissions it gets RST

  [RST, ACK]

  then second attempt with same actions

  Client Hello (SNI=cloudflare-ech.com)

  and then it opens with hwinfo client hello after 40 seconds

  Client Hello (SNI=www.hwinfo.com)

  Attempts mean attempts by browser , I didnt click refresh button

  https://ntc.party/t/12732/312

  DoH в Firefox принудительно или дефолт?

  у меня trr mode = 3 использовать только TRR (если ты про это спрашиваешь)

  DoH in Firefox forced or default?

  I have TRR mode = 3 use only TRR (if you ask about it)

• In Chrome, if you click the Reload button enough times (around 6 times), eventually the browser will retry without ECH. However, if the browser is configured to use DoH, the fallback does not occur.

  https://ntc.party/t/12732/65

  Проверил сайты (в последнем ungoogled chromum 130 на линуксе):

  С DoH и ECH под Йотой все не открываются. Без DoH и с провайдеровским DNS тоже почти все не открываются, но может иногда один начать открываться (sakhtv.ru) после чистки профиля или наоборот перестать открываться.

  Checked the sites (in the latest ungoogled chromum 130 on linux):

  With DoH and ECH under Yota, everyone does not open. Without DoH and with ISP DNS, almost all of them do not open either, but sometimes one of them starts to open (sakhtv.ru) after clearing the profile or vice versa stops opening.

  https://ntc.party/t/12732/311

  Chrome 130.0.6723.116 (официальная сборка) без DoH в браузере на Linux после нескольких нажатий “Обновить” отключает ECH для домена.

  Chrome 130.0.6723.116 (official build) without DoH in the browser on Linux after a few clicks of "Reload" disables ECH for the domain.

  https://ntc.party/t/12732/309

  Chrome 130.0.6723.116 (официальная сборка от Google) с DoH в браузере (1.1.1.1) на Linux не пытается установить соединение без ECH даже после нескольких нажатий "Обновить" (кроме того, и без кнопки "Обновить" он сам безуспешно пытается восстановить коннект). Таймаут после 1 минуты, а потом ошибка.

  Chrome 130.0.6723.116 (official build from Google) with DoH in the browser (1.1.1.1) on Linux does not try to establish a connection without ECH even after a few clicks of "Reload" (in addition, even without the "Reload" button, it unsuccessfully tries to restore the connection itself). Timeout after 1 minute, and then an error.

[FazziCLAY]

disable-ech.sh https://gist.github.com/FazziCLAY/75f72acc8b728530a637121fdee4dfb5

disable-ech.bat https://gist.github.com/FazziCLAY/38f56ab423a0e0a2f864985cf3ce21be