Add logic for detecting which circumvention strategy should be used when speaking to backends

[hellais]

Related to: https://github.com/ooni/probe/issues/886

[bassosimone]

Reduced effort because I'm merging https://github.com/ooni/probe-engine/issues/521, which I worked on as part of this greater issue.

[bassosimone]

I am going to swap out this issue from the Sprint. I am creating the following issues to track the time spent trying to complete this issue (in addition to https://github.com/ooni/probe-engine/issues/521, which has already been mentioned above):

• https://github.com/ooni/probe-engine/issues/610

• https://github.com/ooni/probe-engine/issues/608

[bassosimone]

Swapped in again, as there is some final work to create issues, etc.

[bassosimone]

Okay, now I can move to next sprint. Tracked final bits of work for Sprint 13 in https://github.com/ooni/probe-engine/issues/613.

[bassosimone]

Work done in Sprint 13

The gist of the working has been (1) adding DoH as the primary DNS for the session, with the system resolver as the fallback; (2) support fallback to cloudfronting; (3) support persistent proxy like tor or psiphon; (4) recognise that we cannot use psiphon to contact the probe services because it depends on the probe services being available, so we need to spend more time thinking about it (https://github.com/ooni/probe-engine/issues/586); (5) failover is difficult to implement where there are so many possibilities (i.e. use can configure bouncer but not collector or collector but not bouncer - see https://github.com/ooni/probe-engine/issues/407)

The following list summarises the main steps in which I organized the work:

• merge bouncer and collector into probeservices, because implementing circumvention is definitely going to be simpler if done in a single place

• netx/dialer/proxy.go: allow extemporary proxy set using context: this is a bit of a hack but also the simplest solution to force a specific proxy onto requests

• jsonapi: allow cloudfronting by setting the host header and tunnelling by taking advantage of the possibility of setting the proxy into the context of every request we generate

• jsonapi: refactoring to enable easier further changes

• introduce netx/selfcensor: simpler-than-jafar censorship mechanism that does not require Linux and still is accurate enough for the purpose of running a bunch of integration tests

• factor a DNSClient out of urlgetter, so we can reuse that for the session

• session: significantly simplify initialization such that we can be confident that the bouncer and collector configured by the user are not poised to change during the session lifecycle

• probeservices: allow fallbacking to cloudfronting (we honour the cloud front entries returned by the bouncer and configure a cloudfronted default for the bouncer)

• session: by default use Cloudflare DoH for the default HTTP client, and still fallback to the system resolver, thus reducing issues with DNS blocked services (e.g. github)

• probeservices: try http first, then cloudfronted, then onion (where possible, see below)

• session: make sure we use onion when we have a tor tunnel and all other options fail

[bassosimone]

In yesterday's sprint meeting we concluded no further actions are required. Therefore, I am going to strip this issue from its assigned milestone and effort estimate, and I'm going to close it.