Throttling of Twitter domains in Russia

Starting at about 2021-03-10 07:00, access to Twitter-related domains, like twimg.com and t.co, was throttled in Russia. As a side effect, not only those domains, but also all domains containing "t.co" as a substring were throttled, including, for example, rt.com, reddit.com, and microsoft.com.

NTC has a thread all about it, in Russian. I will try to summarize the main points.

If you make an account at NTC and log in, there will be an automatic translation button under each post. That is the source of the translations into English here.

Classification is by TLS SNI only

The decision of what gets throttled depends only on the TLS SNI, not destination IP address or anything else. SNI-less TLS sessions to Twitter servers are not throttled. TLS sessions with a spoofed Twitter SNI, to non-Twitter servers, are throttled.

https://ntc.party/t/twitter/907/11

При обращении к IP-адресу напрямую, без SNI, скорость высокая.

When you access the IP address directly, without SNI, the speed is high.

https://ntc.party/t/twitter/907/24

Ещё потенциально может быть интересно проверить, что будет если послать тот же SNI на свой сервер

Валдик совершенно прав. Блокировка идет по SNI Прописал в hosts github-releases.githubusercontent.com 4 на IP своего VPS Создал там файл на 100 mb. Дернул по https через curl -k (игнорировать сертификат) Медленно. С разбиением запроса через nfqws - быстро

It may also be interesting to check what would happen if you sent the same SNI to your server

Valdik is absolutely right. Lock goes on SNI Prescribed in hosts github-releases.githubusercontent.com on the IP of his VPS Created a 100 mb file there. Pulled on https through curl-k (ignore certificate) Slowly. Breaking the request through nfqws - quickly

Throttled speed is reported to be 128 kbps

https://ntc.party/t/twitter/907/5

Подтверждаю: скорость замедлена примерно до 128 кбит/с. Провайдер OBIT, Санкт-Петербург. Трафик идёт транзитом через retn.

Видео с video.twimg.com 37 грузятся примерно с burst 512 kbps, speed 64 kbps.

I confirm that the speed is slowed to about 128 kbps. Traffic is transiting through retn.

The video from video.twimg.com is loaded with about burst 512 kbps, speed 64 kbps.

https://ntc.party/t/twitter/907/6

Game Service, Красноярск, магистральный - TTK, так же около 128 кбит/с. Надо бы посмотреть, не порезали ли еще чего акамаевского между делом.

Game Service, Krasnoyarsk, mainline - TTK, as well as about 128 kbps.

Domains with "t.co" anywhere are affected

The evident intent was to throttle traffic to t.co, which is Twitter's link-shortener domain. But throttling affects any domain with "t.co" anywhere as a substring. This includes, for example:

rt.com
reddit.com
microsoft.com
githubusercontent.com
appspot.com
ocsp.digicert.com

https://ntc.party/t/twitter/907/27

Я провёл больше тестов. Сообщаю вам первым. Сравнение доменов происходит по подстроке, и блокируется строка “t.co” (короткий домен твиттера)

Любой домен, содержащий в каком-либо месте в названии домена строку t.co, замедляется.

У Github домен, на котором видно замедление — http://githubusercontent.com

githubusercontent.com

I've done more tests. I'll let you know first. Domain comparisons take place by substline, and the line "t.co" is blocked (short Twitter domain)

Any domain containing a line t.co in any place in the domain name slows down.

Github has a domain that shows a slowdown - http://githubusercontent.com

githubusercontent.com

The unanchored substring matching is very obvious with "t.co", but it also appears to happen with "twimg.com":

https://ntc.party/t/twitter/907/11

Итак, ограничение скорости осуществляется по именам доменов, через DPI. Причем не только доменов twitter непосредственно, но и CNAME-имен Akamai. В частности, замедление видеоконтента наблюдается на video.twimg.com, eip-ntt.video.twimg.com.akahost.net, video.twimg.com.eip.akadns.net, но не на более технических доменах, таких как cs531.wpc.edgecastcdn.net.

So, the speed limit is by domain names, through DPI. And not only twitter domains directly, but also CNAME-names Akamai. In particular, video content is slowing down on video.twimg.com, eip-ntt.video.twimg.com.akahost.net, video.twimg.com.eip.akadns.net,but not on more technical domains such as cs531.wpc.edgecastcdn.net.

https://ntc.party/t/twitter/907/73

Немного личных наблюдений:

~~http с t.co в хосте не шейпится, только SSL~~ шейпится!

tco (без точки) не шейпится

Судя по всему, ВСЕ урлы выбираются регуляркой, просто на остальных это не так заметно (вряд ли у кого-то в домене случайно затесается pbs.twimg.com). Проверял на поддомене вида pbs.twimg.com.my.own.domain. Надо посмотреть, к каким еще достаточно коротким доменам у РКН есть претензии

A few personal observations:

~~http with t.co the host is not a skewer, only SSL~~ is shaping up!

tco (without a point) not to shake

Apparently, ALL urls are chosen by a regular, just on the rest it is not so noticeable (hardly someone in the domain accidentally squeezed pbs.twimg.com). Checked on the sub-domain of the pbs.twimg.com.my.own.domain kind.

The throttling devices are separate from ISPs' normal filtering devices

By using different circumvention techniques, it is possible to construct traffic that is detected and blocked by the usual censorship of Russian ISPs, but which bypasses the throttler. This indicates that the throttling is being done by a separate device, perhaps upstream from ISPs.

https://ntc.party/t/twitter/907/22

Подтверждаю: на моём провайдере zapret и GoodbyeDPI не могут обойти блокировки сайтов, но обходят ограничение скорости. С определенной долей уверенности это говоорит о наличии разного оборудования DPI на пути пакетов.

I confirm that on my provider zapret and GoodbyeDPI can not get around the blocking of sites, but bypass the speed limit. With a certain amount of certainty, this will say that there are different DPI equipment in the way of packets.

https://ntc.party/t/twitter/907/35

То, что goodbyedpi/zapret/добавление точки обходит ограничение, но не блокировки, с определенной долей уверенности говорит о наличии разного оборудования DPI на пути пакетов. Т.е. один может быть провайдерский (блокирующий), а другой — Роскомнадзоровский («суверенный», rdp ru), где-то дальше на магистрали. Такое встречается, это не нетипичная ситуация, но всё равно.

The fact that goodbyedpi/zapret/adding a point bypasses the restriction, but not the lock, with a certain degree of certainty suggests that there are different DPI equipment in the way of packages. That is, one can be a provider (blocking), and the other - Roskomnadzor ("sovereign," rdp ru), somewhere further on the highway. It's not an atypical situation, it's not an unusual situation, but it's still.

The throttling was temporarily disabled and then reenabled

2021-03-10 09:33 https://ntc.party/t/twitter/907/14

Похоже, отключили шейпер вообще. И гитхаб быстро грузит, и видео с твиттера на полной скорости грузится.

Looks like they've turned off the shaker at all. And githab quickly loads, and the video from Twitter at full speed loads.

2021-03-10 10:03 https://ntc.party/t/twitter/907/18

Ага, включили заново. Github опять замедляется.

Yes, they're on again. Github is slowing down again.

Doug Madory posted a graph that shows fluctuations in traffic to Rostelecom, with timing that matches up with these reports.

https://twitter.com/DougMadory/status/1369648537634545673 (archive) media_EwH26cKXMAchgyt

Roskomnadzor made a press release (archive) with the ostensible reason for the throttling.

Роскомнадзор принял меры по защите российских граждан от влияния противоправного контента

10 марта 2021 года

В связи с тем, что интернет-сервисом Twitter в период с 2017 года по настоящее время не удаляется контент, склоняющий несовершеннолетних к совершению самоубийств, содержащий детскую порнографию, а также информацию об использовании наркотических средств, Роскомнадзором было направлено свыше 28 тысяч первоначальных и повторных требований об удалении противоправных ссылок и публикаций.

По состоянию на 10 марта 2021 года не удаленными остаются 3168 материалов с запрещенной информацией (в том числе 2569 с призывами к совершению суицида несовершеннолетними, 450 с детской порнографией, 149 с информацией об использовании наркотиков). Последним ярким примером стало демонстративное игнорирование требований регулятора (в отличие от других социальных сетей Twitter не удалил материалы) об удалении призывов к несовершеннолетним о совершении массового суицида 3 марта 2021 года (напомним, что в этот день, по сообщениям правоохранительных органов, было предотвращено несколько попыток совершения суицида несовершеннолетними).

В соответствии с законом «Об информации, информационных технологиях и о защите информации» (149-ФЗ), распространение информации интернет-сервисом Twitter внесено в перечень угроз.

С целью защиты российских граждан и принуждения интернет-сервиса к исполнению законодательства на территории Российской Федерации в отношении Twitter с 10 марта 2021 года приняты меры централизованного реагирования, а именно первичное замедление скорости работы сервиса (согласно регламенту). Замедление будет реализовано на 100% мобильных устройств и на 50% стационарных устройств.

В случае продолжения игнорирования интернет-сервисом Twitter требования Закона, меры воздействия будут продолжены в соответствии с регламентом реагирования (вплоть до блокировки) до тех пор, пока призывы к совершению самоубийств несовершеннолетними, детская порнография, а также информация об использовании наркотических средств не будут удалены.

Roskomnadzor has taken measures to protect Russian citizens from the influence of illegal content

March 10, 2021.

Due to the fact that from 2017 to the present time the Internet service Twitter has not removed content inducing minors to commit suicide, containing child pornography, as well as information on the use of drugs, Roskomnadzor has sent over 28 thousand initial and repeated requests to remove illegal links and publications.

As of March 10, 2021, 3,168 materials with prohibited information remained to be removed (including 2,569 with calls to suicide by minors, 450 with child pornography, 149 with information on the use of drugs). The latest striking example was the demonstrative disregard for the regulator's demands (unlike other social networks, Twitter did not remove materials) to remove calls to minors to commit mass suicide on March 3, 2021 (recall that on that day, according to law enforcement reports, several attempts to commit suicide by minors were prevented).

In accordance with the law "On Information, Information Technology and Information Protection" (149-FZ), the dissemination of information by Twitter Internet service is included in the list of threats.

In order to protect Russian citizens and to force the Internet service to comply with the law on the territory of the Russian Federation, since March 10, 2021 measures of centralized response have been taken against Twitter, namely an initial slowdown of the service (according to the regulations). The slowdown will be implemented on 100% of mobile devices and 50% of stationary devices.

If Twitter continues to ignore the requirements of the Law, the measures will continue in accordance with the response regulations (up to and including blocking) until the calls to commit suicide by minors, child pornography, as well as information about the use of drugs will not be removed.

The Associated Press noted (archive) that at the onset of the Twitter block, some Russian government web sites also experienced slowdowns and outages.

As the Russian authorities slowed down Twitter, some government websites suffered outages and access problems. It’s not clear if the events were connected, and some experts suggested they could have been the result of unrelated cyberattacks. The Ministry of Digital Development acknowledged outages on some government websites but said they were linked to equipment problems at communications provider Rostelecom.

More discussion on Habr: РКН замедляет Twitter (archive).

Throttling of unrelated domains ends

It appears that the unintentional throttling of non-Twitter domains ended at about 2021-03-11 08:00 UTC. Twitter-related domains are still throttled as before.

https://ntc.party/t/twitter/907/84 2021-03-11 09:46

У меня перестало шейпить адреса, содержащие заблокированные домены.

В тесте от @darkk всё ок, на моих собственных поддоменах - аналогично
$ curl -Lo /dev/null -v https://pbs.twimg.com.kanamori.loweffort.media/100MB.bin -k
$ curl -Lo /dev/null -v https://t.co.kanamori.loweffort.media/100MB.bin -k
Оба выдают полную скорость

I've stopped making addresses containing blocked domains.

In the test from @darkk all over, on my own subdomes - similarly
$ curl -Lo /dev/null -v https://pbs.twimg.com.kanamori.loweffort.media/100MB.bin -k
$ curl -Lo /dev/null -v https://t.co.kanamori.loweffort.media/100MB.bin -k
Both give full speed

https://ntc.party/t/twitter/907/85 2021-03-11 09:58

Да. В околопровайдерском чате пишут, что около 11 утра по Москве перестала тормозилка тормозить ковровыми бомбардировками по *t.co* и осталась только на твиттере. Можно по логам ndt7 попробовать понять, во сколько именно.

Yes. In the near-state chat they write that about 11 a.m. in Moscow stopped braking carpet bombing on *t.co* and remained only on Twitter. You can use the ndt7 logs to try to figure out what time it is.

Test pages

Researchers have set up test pages for people to test their own connections.

Test for non-Twitter throttling, by @darkk with help from @bassosimone and @m-lab:

Test for Twitter throttling, by @4ndv:

https://ntc.party/t/twitter/907/68

Вдохновился идеей Здольникова и сделал чуть более методологически корректный тест в браузере: https://speed.gulag.link/

Исходники и собранный с помощью дорогого коллеги бинарь живут на GitHub - darkk/ru-twi-ndt7: Rapid test for *t.co* throttling Используются домены speed.gulag.link и t.co.speed.gulag.link. Сервер крутится на минимальном Startdust-инстансе в Scaleway, т.е. больше 100 мегабит он не выдаст никогда, но для детекта разницы между одним мегабитом и десятью – сойдёт.

He was inspired by the idea of zdolnikov and made a slightly more methodologically correct test in the browser: https://speed.gulag.link/

Sources and assembled with the help of a dear colleague binar live on GitHub - darkk/en-twi-ndt7: Rapid test for *t.co* throttling Domains are used speed.gulag.link and t.co.speed.gulag.link. Server spins on the minimum Startdust-instance in Scaleway, i.e. more than 100 megabits it will never give, but for the decant of the difference between one megabit and ten - the difference between one megabit and ten.

https://ntc.party/t/twitter/907/94

Собрал на коленке страничку, которая проверяет замедлен ли непосредственно твиттер: Is my Twitter slow or what?

Ссылка на гитхаб там же

Gathered on his knee a page that checks whether Twitter is slowed down directly: Is my Twitter slow or what?

Reference to github there

There is something that I have been seeing in Iran for a long time. Because of the bandwidth-throttling, the upload speed was much faster than the download speed. Or that UDP speeds were sometimes 30 times faster than TCP. https://twitter.com/MrOplus/status/1292222652355051521

I use a method to test this type of internet disruption that I don't know if there is a better way or how it can be improved. Is this method correct or will it cause abuse?

Probably this method can be tested in Russia, in a situation where only the download from Twitter has slowed down.

Create a header file with the following content:

'UploadSpeedTest: SOMERANDOMDATA'

That SOMERANDOMDATA must be a string in the same size of the download file.

Test upload:

# curl -X POST -d @fakeheader.txt  -Lo /dev/null -skw "\ntime_connect: %{time_connect}s\ntime_namelookup: %{time_namelookup}s\ntime_pretransfer: %{time_pretransfer}\ntime_starttransfer: %{time_starttransfer}s\ntime_redirect: %{time_redirect}s\ntime_total: %{time_total}s\n\n" https://video.twimg.com/robots.txt
time_connect: 0.063130s
time_namelookup: 0.048475s
time_pretransfer: 0.124583
time_starttransfer: 0.124592s
time_redirect: 0.000000s
time_total: 5.449413s

Test download (as they did in ntc.party)

# curl  -Lo /dev/null -skw "\ntime_connect: %{time_connect}s\ntime_namelookup: %{time_namelookup}s\ntime_pretransfer: %{time_pretransfer}\ntime_starttransfer: %{time_starttransfer}s\ntime_redirect: %{time_redirect}s\ntime_total: %{time_total}s\n\n" https://video.twimg.com/ext_tw_video/1201955484183531521/pu/vid/720x1280/6VWb4aD7I6HrBQqD.mp4?tag=10

time_connect: 0.049789s
time_namelookup: 0.035611s
time_pretransfer: 0.124363
time_starttransfer: 1.924300s
time_redirect: 0.000000s
time_total: 2.804413s

(This sample test was performed in Netherlands.)

The point is that the server first receives the HTTP data completely and then processes it. (And we download almost nothing.)

Some users on the NTC thread reported an occasional lack of throttling.

@darkk 2021-03-17 16:08 https://ntc.party/t/twitter/907/125

Сегодня в 17:40 наблюдал на МТС в Питере отсутствие замедления по abs.twimg.com. В 19:00 всё ещё не было. Переподключил устройство – замедление вернулось. Видимо, на мобильных “брасах” тоже не на 100% замедлялка замедляет %)

Today at 17:40 I saw no slowdown on abs.twimg.com on MTS in St. Petersburg. At 19:00 there was still no slowdown. Reconnected my device - the slowdown is back. Apparently, on mobile "bras" too, the slowdown is not 100% slow %)

@ValdikSS 2021-03-19 20:20 https://ntc.party/t/twitter/907/128

У меня сегодня весь день твиттер работает без замедлений на проводном канале.

I've got Twitter all day today without slowing down on the wired channel.

Censored Planet published a report on the ongoing Twitter throttling, with detailed technical measurements.

Throttling of Twitter in Russia 2021-04-06

One of the main observations is that throttling is being managed in a different way from how site blocking in Russia is done. In site blocking, though the list of sites to block is managed and dictated centrally by Roskomnadzor, it is individual ISPs that are responsible for enforcing the blocks, a situation that Censored Planet has previously called "decentralized control". In contrast, this research shows that not only are the devices that perform throttling separate from ISPs' usual blocking filters, they are highly uniform across ISPs and probably centrally operated by Roskomnadzor. This centralized mode of operation more resembles the way censorship is done in other countries, like China and Iran. The current belief is that the throttling devices are TSPU (ТСПУ, технические средства противодействия угрозам) DPI boxes, and that this is the first major application of them.

The research team conducted measurements from 7 landline and mobile vantages in Russia. After verifying that all 7 vantages experienced throttling, they began experiments using a "record and replay" technique. They recorded a traffic capture of an unthrottled host in Russia requesting an image from a Twitter server, then replayed that traffic towards a controlled server in the US from the throttled vantages, testing the effects of various modifications to the traffic.

The essential feature that triggers throttling is the TLS ClientHello record—specifically the critical data fields are ContentType=handshake and HandshakeType=client_hello. The detection devices actually parse ClientHello records: it is not enough for the Twitter domain simply to be present as a substring, it must actually be contained in the server_name extension (SNI). The ClientHello does not have to be in the first TCP segment past the handshake, but it does have to appear at the beginning of a segment, and cannot span multiple segments. The detection boxes can detect a ClientHello with an actionable SNI even if it is preceded by up to about 20 other TLS records, or up to 101 non-TLS bytes (which means that throttling even affects Twitter connections through plaintext proxies, like SOCKS or HTTP proxies). Detection is not symmetric, in the sense that only connections that originate in Russia are throttled. But once the connection is established, directionality does not matter: a matching ClientHello results in throttling even if sent by the TCP server.

Throttling is implemented by packet dropping, which is observable as gaps in received IP ID sequences. (One mobile vantage showed evidence of delay-based, rather than drop-based, throttling on uploads, but that may have been caused by other traffic controls implemented by the ISP, unrelated to Twitter throttling.) Both upstream and downstream are throttled independently, to about 128 kbps. If you leave a throttled connection idle for about 10 minutes, throttling will expire and the connection will be usable at full speed again—but sending packets before those 10 minutes expire will renew the timeout. Sending a FIN or RST packet, which has proven useful in confusing some middlebox state machines, does not work in this case to expire the throttling.

Using limited-TTL experiments, the authors found that the throttling devices are located closer to end users (≤5 hops) than are the usual network filter devices (5–8 hops). This is consistent with TSPU installation guidelines sent to ISPs by Roskomnadzor. The high degree of uniformity in detection and throttling behavior across ISPs points to such a unified implementation.

The report observes ways to circumvent throttling:

Use any encrypted proxy or VPN.
Split the ClientHello record across TCP segments.
- This feature is implemented in GoodbyeDPI (Windows) and Zapret (Linux).
- You can also induce TLS record splitting by padding the ClientHello to a large size.
Prepend the ClientHello record with some other TLS record (such as ChangeCipherSpec) in the same TCP segment.
Leave the TCP connection idle for 10 minutes.
Append a trailing dot to the SNI.

You can also induce TLS record splitting by padding the ClientHello to a large size.

@ValdikSS reports that the strategy of padding the ClientHello is being blocked in some cases since 2024-01-14.

Примерно с 14 января ТСПУ начал правильно анализировать и блокировать пакеты с большим TLS Padding (https://ntc.party/t/http-headerstls-padding-as-a-censorship-circumvention-method/168/2), если SNI передаётся до padding’а (и в нём сайт, который должен блокироваться). Блокируются такие пакеты не на все направления, но на Cloudflare точно.

Around January 14, TSPU began to properly parse and block packets with a large TLS Padding (https://ntc.party/t/http-headerstls-padding-as-a-censorship-circumvention-method/168/2) if the SNI is transmitted before the padding (and the site that should be blocked). Such packages are not blocked in all directions, but on Cloudflare for sure.

net4people / bbs