Open bassosimone opened 1 year ago
To mitigate the immediate issue, we have rotated the TH on another data center. There are more activities that we could perform to improve the situation here. I'll leave this issue open for a while more until I've spelled them all out.
Today a user reported two cases of Web Connectivity v0.4 measurements where the resulting
x_status
was16
meaning that the test helper failed. Both measurements were using the same TH: 0.th.ooni.org.I repeated those measurements manually to confirm the initial diagnosis. Here is the output I have seen:
and:
It's a bummer that the TH has a limited set of errors. This fact makes debugging more complex.
Anyway, here's what I see if I use curl on the 1.th.ooni.org box:
I think the underlying issue is that the TH, which is deployed on Digital Ocean, is hosted in a country (apparently, India) where there is upstream censorship.
Now, let's see whether iterative network tracing can help us to understand where censorship happens. To this end, I am using
miniooni
and the newtlsmiddlebox
experiment. I am using as test helper the IP address ofwww.example.com
. I am editing the output of the experiment to make it easier to understand (removing all "in progress" messages).So, the experiment has started and now we see the control run. We're using the
93.184.216.34:443
with theexample.com
server name. We expect to see successes here:We have actually seen many timeouts before reaching out to a certain TTL.
After the control run, comes the experiment run, where we're using the possibly offending SNI:
So, you basically see that for TTL lower than 6 we see a timeout and for higher TTLs we have a connection reset by peer.
The experiment now continues and we see its report ID etc:
Let's now repeat the tlsmiddlebox experiment for the other possibly offending SNI (I will provide less commentary now):
We can now use
mtr
to try to guess which could be the seventh hop. Becauseminiooni
does not run with root privileges we cannot perform this analysis inside it. Also, keep in mind that routing may change after TCP connect for censorship purposes (as well as for other legitimate purposes), hence we cannot be super confident about identifying the ISP:So, with all the above-mentioned caveats, AS14061 is Digital Ocean LLC (where we host our 1.th.ooni.org). The seventh hop instead is AS9498 BHARTI Airtel Ltd.