Avoid SecureDrop server identification due to apt traffic analysis

[garrettr]

Both SecureDrop servers (the Application server and the Monitor server) update their package lists every 24h from the following servers (see install_files/ansible-base/roles/common/templates/security.list):

1. http://security.ubuntu.com
2. http://deb.torproject.org
3. https://apt.freedom.press

There are a number of potential problems with the way this is currently done:

1. The update process does not use Tor. As a result, the package servers (and network eavesdroppers who are appropriately positioned, e.g. either near the update servers or near the SecureDrop servers) learn the real IP address of the SecureDrop servers (technically, it's the real IP address of the SecureDrop network firewall).

   For many current SecureDrop installations, the location of their servers does not need to be a secret, and so this is not a serious concern for them. However, we would like to support the use case where the location of a SecureDrop installation should be kept secret as much as possible.

2. The update occurs every day at a random time between 4am and 5am (see install_files/ansible-base/roles/common/files/cron-apt). For an attacker observing the network near a SecureDrop instance, this is a consistent and predictable signal that could be used to identify the instance.
3. Some of the repos are updated over HTTP, which may make it easier for an attacker to MITM the connection and insert a malicious update to be installed by SecureDrop instance (although they would also need to compromise one of the apt repository signing keys first). This concern is discussed more in depth in #1169.
4. SecureDrop servers receive updates from 3rd party servers run by Ubuntu and the Tor Project. If these servers were compromised, it's possible they could be used to deliver malicious updates to SecureDrop instances. Furthermore, these updates could potentially be targeted specifically for SecureDrop (making their detection more difficult) if a SecureDrop server can be reliably identified when it connects to the update server (e.g. due to timing and/or fingerprinting based on comparing the set of packages installed by SecureDrop with the set of packages requested from the update server).

To resolve this issue, I think we should implement the following mitigations in order:

• [ ] Use apt-transport-tor or something similar to fetch updates over Tor.
• [ ] Randomize the time of automatic updates so it is no longer predictable within a narrow range, but still occurs reliably at least every 24 hours.
• [ ] Consider FPF operating a package mirror for SecureDrop as a Tor Hidden Service. See https://github.com/freedomofpress/securedrop/pull/1169#issuecomment-153867317 for a detailed description of this idea.

[psivesely]

Regarding 2, we can run apt-get update && apt-get -d upgrade at something like (a) a random interval from 0-24 hours (so on average, we download check for and download new updates every 12 hours) or (b) at a random time within an evening interval and with 1/2 probability also at a random time w/in a morning interval. (b) is supposed to be more human-like because (a)--observed over the long run--will give this uncharacteristic even distribution of update traffic over the 24 hour cycle--humans sleep. These were just ideas we can tweak more. Then we can run apt-get upgrade at a set time every night. This allows for predictable resets and package upgrades and unpredictable updates.

4 seems like a big deal. I would totally run an Ubuntu mirror if I wanted to identify or mess w/ SD instances. Very much so think we should do more than consider running a THS mirror for these packages.

Overall, these seem like good steps towards allowing for individuals to operate SD instances w/ strong anonymity. :+1:

[tildelowengrimm]

If SecureDrop servers need to contact an FPF apt server anyway, it seems like a really good idea to run a caching apt proxy over the tor+https transport. I also like the idea of keeping upgrade times consistent from an admin's perspective, but fetching updated packages unpredictably.

[garrettr]

The only reason why I said "consider" is because mirroring all APT for SecureDrop instances through a single FPF server creates a single point of failure (as well as a single point of compromise, which it already is, but at least the FPF package signing key is kept offline and only used as part of an airgapped signing process).

We need to step up our game in terms of infrastructure security, monitoring, and reliability (all of which are ongoing projects internally) before we make ourselves the single point of failure for all SecureDrop instances to receive security updates.

[psivesely]

Just wanted to drop a couple links here:

• Found a HTTPS Ubuntu mirror list
• The Update Framework (TUF) security model provides a good outline of update-process-specific attacks we should work to protect ourselves and SD users from. Regarding some of these, I am not sure if APT does anything itself to protect against them. When APT doesn't have a built-in protection, we should try to figure out how else we can defend against such attacks.

[psivesely]

Just found http://vwakviie2ienjx6t.onion/ is a debian mirror run by https://qa.debian.org/developer.php?login=weasel%40debian.org.

[tildelowengrimm]

I don't think it's possible to implement TUF on top of APT without doing a bunch of additional work. Weasel is a longtime Tor Project contributor. If you were going to reply on someone else's APT mirror, his would be as good as any. But is there a good reason not to run your own infrastructure instead?

[garrettr]

But is there a good reason not to run your own infrastructure instead?

No. We would definitely want to run our own infrastructure for this to ensure reliability.

[conorsch]

It's already documented in the OP, but routing the server traffic over Tor, without the use of a hidden service, would still be a big win, and is much lower hanging fruit configuration-wise.

[psivesely]

Although not the topic of the original post, since issues related to mirror authenticity are tightly coupled with how we do updates, I thought I'd bring that up here.

Authentication

• Protect against freeze/ rollback and slow retrieval/ endless data attacks (these seem like most realistic and practical attacks mentioned in TUF with regards to APT)
  • Obviously signatures protect against serving backdoored packages
  • If admins check OSSEC regularly maybe they'll notice software hasn't been updating

Onion Mirrors

• No CAs (to compromise)
• 1024-bit RSA keys are weak (a problem with SD instances themselves as well)
• Slowest connection (7 hops through the Tor network)
• Connections can be unreliable
• We don't know of any for Ubuntu--do we really want to host our own? (Some more reasons why we maybe don't in the next section)

HTTPS Mirrors

• Better crypto
• CAs (to compromise)
• Only slightly slower than HTTP
• Ostensibly use safer crypto
• None for Ubuntu https://launchpad.net/ubuntu/+archivemirrors

Tor + regular mirrors

• While these do not use TLS between the client and mirror, they do make it practically impossible to do targeted attacks; a MitM attacker can't know if they are, say, performing a freeze attack against a SD or a Debian developer. Once the upgrade process starts they could identify a SD by the packages it downloads, but a priori there is no way of identifying a SD based on the generic package lists it requests. I think this arguably increases the chances the SD will receive authentic package data.

Onion Routing

• Hides the location of the SD
  • Probably not against an advanced attacker with enough time
  • Real IPs are probably already known for all the instances out there

Onion mirrors

• Client <-> rendezvous point circuit fingerprinting
• Quite slow
• Real IPs are probably known/ easily discoverable
• Gets much less traffic, making traffic correlation easier
• Rendezvous point circuits are not multiplexed in a TCP stream, making traffic analysis easier
• Again, none of them presently

Using Tor w/ regular mirrors

• Easiest to setup (just install apt-transport-tor and sed -i 's/http/tor+http/g' <sources files>)
• Not as slow as onion mirrors, but slower than HTTPS
• Gets tons of traffic, making correlation harder
• General circuits are multiplexed in a TCP stream, making traffic analysis harder

[ageis]

Debian official apt archives are now available as onion services:

• https://bits.debian.org/2016/08/debian-and-tor-services-available-as-onion-services.html
• https://blog.torproject.org/blog/debian-and-tor-services-available-onion-services

See also #306.

[conorsch]

Consider #1364 a reminder that forcing apt traffic over tor can break things in a big way.

[conorsch]

967 explicitly mentioned routing DNS requests over tor, which should be considered in this context as well.

[msheiny]

This issue is coming back up due to internalization efforts and different legal threat models abroad. A few things I will add to the debate:

• We need to run our own caching proxy apt servers - relying on upstream ubuntu servers is problematic for the QA process as we do not have time to vet how upstream upgrades affect the system.
• As we move to proxy tor packages in #2113 - there is less of a concern of tor breaking catastrophically for instances. So I think moving to apt over tor is less of a concern for SD going down (since we control any tor updates).
• The routing of all outbound internet traffic over tor probably fits in this discussion (to jump onto the awesome #967 that @conorsch brought up).
• We can prevent freeze/ rollback attacks by using Valid-Until in our Release files. This would involve us pulling down and re-signing the ubuntu packages so that process obviously adds a lot of overhead. We already have to do this for tor mirroring to maintain our sanity and not having to fight to QA before upstream tor's valid-until date runs out.

[ageis]

Have a couple servers where I've successfully transparently proxied all TCP traffic in/out over Tor. Their own docs should be helpful to you on this front, so I'll put the link here demonstrating a couple different approaches: https://trac.torproject.org/projects/tor/wiki/doc/TransparentProxy

Only thing I'll add is that if you use the Tor DNS proxy + DNSPort, it will be advantageous for you to put a local caching DNS server on top of that, such as with dnsmasq. This tends to help with latency and responsiveness and smooths out the whole experience; additionally it will reduce the number of outbound DNS requests (which would be beneficial in itself even without Tor).

For example, to begin by only intercepting DNS requests and routing them over Tor...

/etc/tor/torrc:

VirtualAddrNetworkIPv4 10.192.0.0/10
AutomapHostsOnResolve 1
AutomapHostsSuffixes .onion
TransPort 9040
DNSPort 5353

Some iptables rules here, might be redundant (like grabbing TCP DNS requests in addition to UDP) but I put them together myself (in your use case you'll also want to include an explicit DROP):

 iptables -t nat -A OUTPUT -d 127.0.0.1/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 53
 iptables -t nat -A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 53

 iptables -t nat -A PREROUTING -i eth0 -p udp --dport 53 -j REDIRECT --to-ports 53
 iptables -t nat -A PREROUTING ! -i lo -p udp -m udp --dport 53 -j REDIRECT --to-ports 53

 iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 53 -j REDIRECT --to-ports 53
 iptables -t nat -A PREROUTING ! -i lo -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 53

 iptables -t nat -A OUTPUT -p udp --dport 53 -j REDIRECT --to-ports 53           
 iptables -t nat -A OUTPUT -p tcp --dport 53 -j REDIRECT --to-ports 53           

 iptables -t filter -A OUTPUT -p udp -m udp --dport 53 -j ACCEPT                 
 iptables -t filter -A OUTPUT -p tcp -m tcp --dport 53 -j ACCEPT                 

 iptables -t nat -A OUTPUT ! -o lo -p udp -m udp --dport 53 -j REDIRECT --to-ports 53
 iptables -t nat -A OUTPUT ! -o lo -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 53

/etc/dnsmasq.conf:

no-resolv
no-poll
port=53
server=127.0.0.1#5353
listen-address=127.0.0.1
cache-size=1024
local-ttl=3600
min-cache-ttl=3600
log-queries

/etc/resolv.conf:

nameserver 127.0.0.1

With the Tor TransPort (not covered above), there are several options you may want to be aware of. I'll list some: OnionTrafficOnly, NoDNSRequest, (C̶a̶c̶h̶e̶D̶N̶S̶,̶ ̶C̶a̶c̶h̶e̶I̶P̶v̶4̶D̶N̶S̶,̶ ̶U̶s̶e̶D̶N̶S̶C̶a̶c̶h̶e̶,̶ ̶U̶s̶e̶I̶P̶v̶4̶C̶a̶c̶h̶e̶—edit: most uses of the DNS cache are being deprecated and recommended against due to risks). And remember the TransPort and SOCKSPort are not an HTTP proxy, it's TCP and SOCKS, respectively, so if any of your applications require one you'll have to chain it with something like Privoxy.

[ageis]

Don't know why earlier commenters on this issue have said there are no Ubuntu HTTPS mirrors, there are plenty, just not from Canonical.

[msheiny]

Don't know why earlier commenters on this issue have said there are no Ubuntu HTTPS mirrors, there are plenty, just not from Canonical.

I don't think using a third-party repo is a good choice for stability. We should be running our own mirror to more tightly control what packages get distributed. See #3095

[zenmonkeykstop]

One other point in favour here (tho more a developer convenience one) - delivering at least the apt.freedom.press packages over Tor would make it easier for devs to stand up temporary apt repos and deploy to hardware for testing purposes.