Don't automatically follow-up to bogons

[bassosimone]

Discussed with @hellais yesterday. There are cases, like Iran, where this is not optimal. We want to enable this functionality only in specific cases. Should be easy to do that.

[hellais]

Here is an example of such a measurement in Iran: https://explorer.ooni.org/measurement/lx0Wn3cGU4j3ZSIlTEYEeZqqQZxNprIPAZ6cvSEuymqMTBbv9LacYGgv0LY5pMQE?input=ablogformen2.blogspot.com

{
  "resolver_hostname": "81.12.12.201",
  "query_type": "A",
  "hostname": "ablogformen2.blogspot.com",
  "answers": [
    {
      "ipv4": "10.10.34.36",
      "answer_type": "A",
      "ttl": "889"
    }
  ],
  "failure": null,
  "resolver_port": 53
}

[bassosimone]

@hellais however, blogspot.com is not ever going to serve anything from 10.10.34.36 so, once we see that address, we already know that there is DNS based censorship. The question seems to choose one of the following paths:

1. continue the measurement knowing that the address is a bogon, perhaps because we want to see the blockpage, thus following the experience of a user

2. use another resolver to discover other possible ways in which such domain could be blocked

3. do both

4. don't decide here in netx but rather decide in ooni/probe-engine (which is probably the right choice in terms of mechanism/policy separation, but still we have to choose eventually)

[bassosimone]

The solution indicated in the issue title has been implemented in #139. I have a feeling we should discuss this further in the future, because there is a fundamental tussle between following the bogon and getting the block page and enumerating all the ways in which a resource is blocked.

I am going to leave this issue open so we can chat about it during the next sprint review meeting.

[fortuna]

I think it makes sense to not follow bogons. You already determined the page is blocked. Ideally we would indicate in the measurement JSON that this is censorship, and why we determined that.

You could still collect the block page and record it, but it shouldn't be part of "determining the site is blocked". It's not clear to me the block page is that useful though.

[hellais]

I would argue that it's valuable to follow bogons so as to collect the actual content of the blockpage.

The reasons are the following:

1. It's useful to track the evolution of blockpages so that we can understand how and if the content of them changes
2. We have existing heuristics based on blockpage content which we can use for analyzing these sorts of measurements
3. The content itself of the blockpage can content useful information on the legal reason by which the blocking is implement

Especially in the case of bogon IPs I would say it's important to record the blockpage, beacause it is in fact not possible to retrieve the content of the blockpage from a location outside of the vantage point of the user, as is the case with publicly routed IPs.

[bassosimone]

The way in which this feature has been modified is fine, because it's consistent with this use case:

1. when we're talking with OONI servers we don't want to accept bogons

2. when we're measuring we may want to accept bogons

With respect to the second statement, I don't fully agree though. I am more leaning towards the opinion of @fortuna here. I especially think that not following bogons is leading to less interesting results. To better investigate this, I am going to work on ooni/jafar#7. I will update the title of this issue and to further clarify the real topic it is about. It is not clearly about stopping treating bogons as errors as this functionality has now been disabled by default in #39.

[xhdix]

If a gambling site address is on the test list, a lawsuit will be filed for that person in Iran. (Of course after trying 3 times the same site or different sites.)

https://www.shodan.io/search?query=pooyesh''pardaz.co They don't have permanent IP.

[bassosimone]

I have came to the conclusion that, as far as netx is concerned, it's correct not to automatically flag bogons as errors, but we should have a way to know we did get a bogon and we should configure netx to treat bogons as errors every time we know a website cannot be beside a bogon. This applies to our own infrastructure, as well as to any other global website. (I would argue that our test list only contains such sites, but anyway, for excess of precaution let's say I think we should enable this functionality when we're connecting to an HTTPS website.) For this reason, I am going to push an update to netx that basically christens what has been done in #139 and allows a user of netx to know whether any domain resolution included a bogon.

[bassosimone]

With the merge of #149, and having reasoned further on the matter, I think we can close this issue and there is nothing extra bogons related we need in this repo for now.