[v3 onion migration] If v2 and v3 are both used, set Alt-Svc header in Apache configs

[redshiftzero]

Description

We can transparently move user traffic over to v3 using the HTTP Alternative Service (Alt-Svc) header. If v2 and v3 are both used, we should set the Alt-Svc header to point to the v3 onion.

References for Alt-Svc:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Alt-Svc https://tools.ietf.org/html/rfc7838

Subticket of #2951

[conorsch]

For now, we should limit use of the Alt-Svc header to the Source Interface. If we also apply it to the Journalist Interface, we run the risk of breaking access for Journalists during the v2->v3 migration window: the v3 auth tokens will not be present on the Journalist Workstation until an Admin provides them, due to the need to fetch the v3 info from the server, via shell access, which Journalist Workstations lack.

[conorsch]

Took a look at potential implementations for adding the Alt-Svc header. In short, we have full flexibility to use a conditional logic and edit the Apache configs directly, via Ansible. Since Admins must run the Ansible logic in order to add v3 URLs, we'll be able to use the v{2,3}_onion_services values to determine whether it's appropriate to add the Alt-Svc header—as well as the value, which must be the hostname for the new v3 Onion—directly to the Source Interface apache config:

https://github.com/freedomofpress/securedrop/blob/de8cbcf66e86164b7a386858d717ea3ac58d9c6d/install_files/ansible-base/roles/app/templates/sites-available/source.conf#L48-L56

We can look up the hostname for the v3 URL at that point in the Ansible run, because the Tor updates have already been applied. Then the template can insert the relevant Alt-Svc header if-and-only-if both v2 & v3 services are enabled.

[redshiftzero]

So the way that Alt-Srv works is that it'll still use v2 if the v3 onion isn't available (I tested the behavior in Tor Browser visiting the v2 onion with an Alt-Srv header set to a v3 onion that isn't working), so I think we can still add the journalist interface alt-srv to v3 if both v2 and v3 are enabled.

Now for the Alt-Srv header handling in Tor Browser right now, it's worth noting that it's actually not prioritizing onion URLs (and also not prioritizing v3 onion URLs) but this behavior may change: https://trac.torproject.org/projects/tor/ticket/27502 (added note on upstream ticket saying that it would be cool if v3 were prioritized over v2). You can test for yourself via https://perfectoid.space/test.php which for me only uses the Alt-Srv some of the time.

In RFC7838 there's also this Alt-Used header which would be nice to inspect to know that this is working:

When a client uses an alternative service for a request, it can indicate this to the server using the Alt-Used header field (Section 5).

From looking at the Firefox source, there is some logic about setting the Alt-Used header (see unit tests about this here: https://github.com/mozilla/gecko-dev/blob/b8157dfaafc42deb3b9824ab6aab1c3e11636d76/netwerk/test/unit/test_altsvc.js#L246), but in testing it looks like Tor Browser isn't setting this. There's also no indicator via the UI (ticket on Tor's side: https://trac.torproject.org/projects/tor/ticket/27590) about this, so it's pretty hard to tell if the Alt-Srv is in use (and it might not actually get used all the time until Tor Browser changes its behavior). Regardless I think we should add the headers while we have admins running the playbooks.

Otherwise my plan right now to try to test using tor debug level logging (on the client side), else I could do something like provide a patch for test that adds a separate vhost for the v3 onion and access logging so that we can use apache access logs to figure out whether v2 or v3 is being visited. If anyone has a smarter idea than this let me know.

[conorsch]

@redshiftzero Peeking at the WIP in https://github.com/freedomofpress/securedrop/compare/v3_alt_srv, I see that we're using cat /var/lib/tor/services/sourcev3/hostname again to retrieve the v3 URL. We do the same in restrict-direct-access already:

https://github.com/freedomofpress/securedrop/blob/1e0082281e6ce14ab96e181630b5109c94ae5950/install_files/ansible-base/roles/restrict-direct-access/tasks/fetch_tor_config.yml#L13

Maybe it's worthwhile to store this information server-side in a machine-readable format, e.g. via ansible local facts. We'd have a straightforward interface for accessing the Onion URL info, which will come in handy during #4631. Note that in #4631, we can inspect the headers such as Host, but cannot directly read the tor hostname info via the webapp user. Thoughts?

[redshiftzero]

hmm yeah using ansible facts seems like a good approach here - we'd need to make sure its readable by www-user for the purpose of #4631 - would that be the case?

also flagging for this ticket - without making much broader changes to the apache configuration this header will get served for both v2 and v3, though it won't do anything for v3. I obviously don't love this, but making much broader system changes (e.g. serving v3 source interface on port 81, v2 source interface on port 80) for a temporary feature for the v2/v3 transition I dislike even more.

[conorsch]

we'd need to make sure its readable by www-user for the purpose of #4631 - would that be the case?

That's right, 644 should do it. Additionally, we should write a static file (e.g. JSON), rather than a dynamic script that would read from the /var/lib/tor/ locations. That way, even www-data can access the information. Only hostnames and service names would be required, no client auth or privkey details.

[redshiftzero]

[removing from 1.0.0 milestone, at this point we likely won't do this at all, but it definitely won't happen for 1.0.0]

[eloquence]

Tentatively closing, we're unlikely to do this at this point (it would be a heavy lift for 1.1.0 and would be unlikely to matter soon after). @redshiftzero please reopen if you disagree.