Shutdown in Kazakhstan since 2022-01-05, with brief periods of connectivity

[wkrp]

The government is Kazakhstan has imposed an Internet shutdown since 2022-01-05 10:30 (16:30 Almaty time). Since then, it looks like access has been occasionally restored, for about 3 hours at a time, at irregular intervals. About 20 hours before the full shutdown, there was a partial shutdown of mobile ISPs.

• BBC article: "Kazakhstan unrest: Internet cut amid fuel protests" (archive)
• NTC thread that started before the full shutdown, reports mobile network blocking at 2022-01-04 19:06
• Cloudflare blog post (archive) using evidence from Cloudflare radar
• IODA dashboard 2022-01-02 to 2022-01-08 04:43

The Cloudflare blog post has some good details and a graph.

Cloudflare Radar shows that the full shutdown happened after 10:30 UTC (16:30 local time) [2022-01-05]. But it was preceded by restrictions to mobile Internet access yesterday [2022-01-04].

The first disruptions reported affected mobile services, and we can see that at around 14:30 UTC yesterday, January 4, 2022, there was significantly less mobile devices traffic than the day before around the same time.

When we focus on other ASNs besides Kaz Telecom such as the leading mobile Internet services Tele2 or Kcell we can see a big drop in traffic yesterday [2022-01-04] after 16:00 UTC, confirming local reports. Mobile traffic did not drop to zero which may indicate throttling rather than a full shutdown. Today [2022-01-05], however, the Internet, mobile or not, is shut down.

https://radar.cloudflare.com/kz (archived 2022-01-08 04:44)

You can see in the Cloudflare graph that traffic has risen above zero 3 times, for about 3 hours at a time, since the start of the shutdown. This looks kind of like an Internet curfew (as has happened, for example, in Myanmar), except that the intervals of access do not occur at the same time of day. The IODA dashboard graph shows that a fourth interval of access started about 2.5 hours ago, at 2022-01-08 02:30.

[wkrp]

I have not been able to find out yet whether even DNS is blocked during the shutdown. I am reminded of the 2019 shutdown in Iran, where DNS was not blocked and DNS tunnels might have worked. If it is not blocked, then we can perhaps help with setting up tunnels for people in Kazakhstan to use.

[oo0000oo]

There has been reported that telegram works using some SOCKS proxy during the shutdown. Not sure if it was shadowsocks or not.

Also, it seems that some resources hosted in Kazakhstan are available within the country. Online banking from kaspi.kz and media tengrinews.kz

[wkrp]

There has been reported that telegram works using some SOCKS proxy during the shutdown. Not sure if it was shadowsocks or not.

Thanks. Do you know what ISP or network it was on?

On the NTC thread, there is a report that dnstt (a DNS tunnel) is working for at least one user in Kazakhstan. I posted instructions for others to test whether it might work for them.

https://ntc.party/t/network-shutdown-all-around-kazakhstan/1601/8

one guy wrote me from Kazakhstan right now. He said that DNS works and its possible to use it. But need to setup proxy servers

https://ntc.party/t/network-shutdown-all-around-kazakhstan/1601/11

That guy already tunnel traffic via dnstt to his own server. It works. Speed isn’t high, but possible to write text massages.

Apparently ICMP does not work:

https://ntc.party/t/network-shutdown-all-around-kazakhstan/1601/13

ICMP не работает. Я пробовал снаружи пингануть пару адресов, но ни один не ответил. Изнутри ICMP идёт только к 8.8.8.8. TCP, UDP не работает, за исключением dns на 53 к операторскому и гугловскому резолверам. Вообще, у каждого провайдера свой тип блокировки. Сейчас пишу про Билайн, но говорят, что на Казахтелекоме намного проще, там можно просто https проксей. Но, ещё раз говорю, в каждом регионе и у каждого оператора свои заморочки.

ICMP does not work. I tried pinging a couple of addresses outside, but none responded. From the inside, ICMP only goes to 8.8.8.8. TCP, UDP does not work, except dns on 53 to operator and Google resolvers. In general, each provider has its own type of blocking. Now I write about Beeline, but they say that it is much easier on Kazakhtelecom, there you can just https proxy. But, I repeat, every region and every operator has its own problems.

[oo0000oo]

@wkrp SOCKS proxies over 3785 port work on many ISP in different regions now. EDIT: it works on KazakTelecom ISP which is the main ISP in the country.

[wkrp]

You can see in the Cloudflare graph that traffic has risen above zero 3 times, for about 3 hours at a time, since the start of the shutdown. This looks kind of like an Internet curfew (as has happened, for example, in Myanmar), except that the intervals of access do not occur at the same time of day.

In the past 3 days the pattern of accessibility has been more regular, beginning at 02:30 UTC (08:30 Almaty time) and ending at 06:00 or 10:00 UTC (12:00 or 16:00 Almaty time). Times are approximate, made by eyeballing the chart.

date	start UTC (Almaty)	end UTC (Almaty)	duration
2022-01-05	17:30 (23:30)	22:00 (28:00)	4.5 hours
2022-01-06	10:30 (16:30)	14:00 (20:00)	3.5 hours
2022-01-07	02:30 (08:30)	06:00 (12:00)	3.5 hours
2022-01-08	02:30 (08:30)	10:00 (16:00)	7.5 hours
2022-01-09	02:30 (08:30)	06:00 (12:00)	3.5 hours

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641317887&until=1641748027

The Cloudflare blog post has been updated with information about the first three temporary restorations of access. It says the timing of periods of restored access corresponds with government announcements.

The nationwide Internet shutdown in Kazakhstan is now in its third day. However, we have observed brief periods of Internet restoration over the last several days. Cloudflare Radar shows that the three times that connectivity was restored were around the same time the Kazakh President, Kassym-Jomart Tokayev, made public announcements.

The first restoration of some Internet services (mainly on the largest telecommunication company in the country, Kaz Telecom) occurred on January 5, around 18:00 UTC (midnight local time). It was at the same time Kazakh President Kassym-Jomart Tokayev announced in a televised speech that he had appealed to a Russia-led security bloc to assist and "protect the state". He Internet shutdown resumed at 21:30 UTC.

The second restoration was similar, though with less impact, and was also mainly seen on Kaz Telecom. It took place after 10:45 UTC on January 6, and it lasted again for about three hours (until 13:35 UTC). That was around the same time the results of the session of the Kazakh Security Council, under the chairmanship of President Tokayev, were announced in a statement with several “urgent instructions”.

The third restoration on Friday, January 7 was the most significant one thus far, as seen in the chart above. It started at around 02:50 UTC (08:50 local time) and it ended three hours later (05:50 UTC).

This time, in addition to Kaz Telecom, there were some mobile networks like Beeline (represented in the chart in light green with the name IPNET_KA) and Tele2 (in red) that showed some traffic.

[wkrp]

SOCKS proxies over 3785 port work on many ISP in different regions now.

It's reported that a Tor obfs4 bridge on port 3785 works as well.

Some other ports to try are 179, 646, 3784, 4784, 5060. I did an Nmap port scan of the /24 neighborhood of gov.kz, which the Cloudflare blog post reported as being inaccessible from outside, to see if any would respond on port 3785. There was one hit:

# nmap -PS3785 -sn -n gov.kz/24
Nmap scan report for 195.12.114.89
Host is up (0.21s latency).
Nmap done: 256 IP addresses (1 host up) scanned in 15.57 seconds

Then I scanned that host to see which other ports were responsive:

# nmap -n -PS3785 -p- --reason 195.12.114.89
Nmap scan report for 195.12.114.89
Host is up, received reset ttl 236 (0.21s latency).
Not shown: 65529 filtered ports
Reason: 65529 no-responses
PORT     STATE  SERVICE       REASON
179/tcp  closed bgp           reset ttl 233
646/tcp  closed ldp           reset ttl 236
3784/tcp closed bfd-control   reset ttl 234
3785/tcp closed bfd-echo      reset ttl 234
4784/tcp closed bfd-multi-ctl reset ttl 233
5060/tcp open   sip           syn-ack ttl 50

Nmap done: 1 IP address (1 host up) scanned in 344.21 seconds

This by itself is not conclusive evidence that these ports always get special treatment. For example, it could be that this IP address is special, or it could be that incoming packets are treated differently than outgoing packets. But these other ports are at least worth a try.

A port scan can also serve as a tool to see what ports might be reachable from inside Kazakhstan, if you can a host that is responsive (sends a SYN/ACK or a RST) on every port, like scanme.nmap.org. The ports that have a reason of no-response are the ones blocked by the shutdown; the ones with reset or syn-ack are making it through the shutdown.

# nmap -v -n -Pn -p- -T4 --reason scanme.nmap.org
# nmap -v -n -Pn -p- -T4 --reason -6 scanme.nmap.org

[oo0000oo]

It's night in Kazakhstan. No one to scan with nmap.

One person has tested all ports with SOCKS proxy except 3785 all others are not available.

My contact could scan with nmap a bit later.

[oo0000oo]

It has been reported that 179 port is open

[wkrp]

The Tor Project has posted a guide, in Russian and English, on how to get bridges that work in Kazakhstan. Because, at this point, access requires specific port numbers, you cannot simply use the normal BridgeDB or Moat interface. Instead, email frontdesk@torproject.org with the subject "bridge kz". I suppose you will have to send the email during the few hours each day when there is normal access.

https://forum.torproject.net/t/internet-shutdown-in-kazakhstan-how-to-circumvent-censorship-with-tor/1679

Получение моста

1. Отправьте письмо по адресу frontdesk@torproject.org, указав в теме письма “bridge kz”.
2. Из полученного письма скопируйте строку с адресом моста целиком. Ниже - инструкция, как вручную добавить мост в Tor Browser.

@adamfisk writes that Lantern servers are now listening on specific ports to try to work around the shutdown:

We’ve switched all Lantern (https://lantern.io) servers in the region to listen on 3785, 5060, as well as randomized high ports.

It has been reported that 179 port is open

I opened ports 179 and 3785 on an obfs4 bridge. But the address of this bridge is public and it may be easily blocked, so for regular use it's better to ask the Tor frontdesk for a private bridge.

Bridge obfs4 172.105.56.235:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

Bridge obfs4 172.105.56.235:3785 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:179 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

Bridge obfs4 [2400:8904::f03c:92ff:fe93:f42d]:3785 DD9769A0D6A9F18C24FCE731583597012E66273F cert=AEu2dF5cSjzQwA8kDx4R+38u10TReImk3ERjWFmzBGA0tPGyFxnsJRke5iSBef6+QDejew iat-mode=0

[wkrp]

Just noting for posterity something that I haven't seen reported on. The address of the president of Kazakhstan of 2022-01-01 (the one with the "open fire without warning" statement) declares the intention for a controlled and partial restoration of Internet access at specific times. The impression I get from the English version of the address is that the plan is more circumscribed than what the CNN article says: "internet is gradually being restored as the situation stabilizes."

I also want to record archive links for the page, because gov.kz is not accessible from outside while the shutdown is in effect. At the moment it's online.

https://www.gov.kz/news/details/309489?lang=kk (archive)

Жағдайдың тұрақталғанына қарай еліміздің кейбір өңірлерінде интернет-байланысты белгілі бір уақыттарда қосу туралы шешім қабылдадым.

https://www.gov.kz/news/details/309489?lang=ru (archive)

В связи со стабилизацией обстановки мною принято решение включить Интернет-связь в отдельных регионах страны на определенные временные интервалы.

https://www.gov.kz/news/details/309489?lang=en (archive)

Due to the stabilization of the situation, I have decided to enable Internet communication in certain regions of the country for specific time intervals.

It's also perhaps noteworthy that access had been restored for limited times even before the "I have decided" of the 2022-01-07 address.

[wkrp]

It appears that the shutdown has ended since 2022-01-11 00:00, about 67 hours ago.

https://ioda.caida.org/ioda/dashboard#lastView=overview&view=inspect&entity=country/KZ&from=1641168000&until=1642032000

https://twitter.com/OliverLinow/status/1481536681656426497

After the 5-day Internet Shutdown in #Kazakhstan, the Internet is now back for more than 50 hours. #KeepitOn

Thx @caida_ioda, everyone can check for shutdowns themselves. Just go to https://ioda.caida.org/ioda/dashboard, select a country and a time period. That's it!

Although there was only limited success in gaining access during this shutdown, what success there was gives me a hopeful feeling. Even in a reputed shutdown, all is not necessarily lost. Let's keep these lessons in mind for the future.

[wkrp]

@anadahz pointed me to a RIPE Labs blog post by @emileaben on the shutdown. It notes that despite being "shut down," networks in Kazakhstan were still present in the global BGP routing tables, which matches our experience with certain ports being unblocked. It also has some analysis of different levels of access in e.g. data centers versus residential connections.

https://labs.ripe.net/author/emileaben/the-kazakhstan-outage-as-seen-from-ripe-atlas/

It is difficult to pinpoint the cause of the outage. However, the affected networks have remained visible in the global routing system (BGP), which means they've remained "connected" to the Internet even though they have not been able to send or receive packets. The timing of the outage was synchronised, suggesting it was the result of some centralised action, although we do see small variations per region.

If we try to distinguish between RIPE Atlas vantage points in infrastructure - i.e. RIPE Atlas anchors and other probes with tags that suggest they are in data centres - we see differences in how connectivity developed over the last few days.

The figure below shows infrastructure vantage points in red. While connectivity for most of these vantage points went down in the last few days, it looks like most are able to send and receive packets to/from the Internet again since around midnight UTC on Friday 7 January. The other vantage points, which we think are mostly near end-users show that over the last few days there were periods of multiple hours where some of these vantage points had Internet connectivity.

After a few hours where almost all of our RIPE Atlas vantage points were online again, we see a drop again. If we look at infrastructure (data centres) versus other probes we do see that roughly half of the other probes (homes, offices, etc.) go down again, but many stay connected.

The comments on the post link to an interactive notebook for analyzing outages using RIPE Atlas.

https://observablehq.com/@aguformoso/internet-outages-as-seen-by-ripe-atlas

[wkrp]

An article by Katia Patin gives some details about how working ports were discovered, and efforts to establish proxies.

2022-01-27 Kazakhstan shut down its internet. These programmers opened a backdoor (archive) Обойти национальный шатдаун: как молодые IT-специалисты вернули интернет тысячам казахстанцев (archive)

A senior software engineer at LinkedIn in Toronto, Maksat Kadyrov jumped into action when he lost touch with his brother in Almaty. He went live on Instagram, looking to crowdsource a way to reach his family. ... He live streamed on Instagram for hours as they scanned some of the more than 65,000 existing ports. During the live stream, they found five open ports, tested them and were able to establish a connection. They later learned that it was a bug in outdated Cisco equipment, used widely by Kazakh telecom operators, which had accidentally kept these ports open. Kadyrov, Maksut and the others used these open ports to support their operation, crowdsourcing funds or footing the cloud computing bill themselves from service providers like Digital Ocean and Amazon.

Over the next few days, the loosely organized group set up dozens of proxy servers — first for Telegram and later even for internet browsers like Firefox. Maksut admits their user estimates aren’t exact; not all of them had a chance to collect data. But more recently, on January 19, Zharaskhan Aman, a software engineer at Facebook in London, rounded up some of the numbers he had from Telegram analytics showing that the 9 servers he raised alone had 155,762 users from Kazakhstan between January 4 and 11. ... Based on user traffic provided by Telegram, Maksut estimates the group got between 300,000 to 500,000 people online on the message app during the five-day shutdown. ... Sharing connection instructions by Telegram, email and text, members of the group said they were overwhelmed with demand. Within 24 hours Kadyrov said he had more than 2,000 requests for access to his servers, which he had been sending out one-by-one.

Когда старший инженер-программист LinkedIn в Торонто Максат Кадыров потерял связь со своим братом в Алматы, он решил, что пора действовать. ... В течение нескольких часов он вел прямую трансляцию в Instagram, пока они сканировали несколько из более чем 65 тысяч существующих портов. Во время прямого эфира они обнаружили пять открытых портов, протестировали их и смогли установить соединение. Позже они узнали, что некоторые порты оказались случайно открыты из-за ошибки в устаревшем оборудовании Cisco, широко используемом казахстанскими операторами связи. Кадыров, Максут и другие использовали эти открытые порты для поддержки своей операции и покупали серверы у Digital Ocean, Amazon и других провайдеров на деньги, собранные краудсорсингом или свои собственные средства.

В течение следующих нескольких дней группа энтузиастов установила десятки прокси-серверов — сначала в Telegram, а затем даже в интернет-браузерах, таких как Firefox. Максут признает, что его оценка количества пользователей не точна — не у всех была возможность собрать данные. Но 19 января Жарасхан Аман, инженер-программист, работающий в Facebook в Лондоне, изучил аналитику Telegram и посчитал, что только 9 поднятыми им серверами с 4 по 11 января воспользовались 155 762 пользователя из Казахстана. ... По оценкам Максута, основанным на данных о посещаемости Telegram, за время пятидневного отключения группа дала доступ к приложению от 300 до 500 тысячам человек. ... Обмениваясь инструкциями по подключению через Telegram, электронную почту и СМС, члены группы говорили, что они не справлялись с потоком запросов. Кадыров говорит, что всего за 24 часа ему поступило более 2 тысяч запросов на доступ к его серверу, которые он рассылал по одному.

I found Katia's article as a reference in the paper Government Internet Shutdowns Are Changing. How Should Citizens and Democracies Respond? (2022-03-31), which was recommended by @fortuna.