search

freedomofpress / securedrop

GitHub repository for the SecureDrop whistleblower platform. Do not submit tips here!
https://securedrop.org/
Other
3.6k stars 685 forks source link

Upgrading to tor 0.2.8.6 breaks hidden services #1364

Closed conorsch closed 8 years ago

conorsch commented 8 years ago

The new version of tor, 0.2.8.6, causes hidden services to fail in SecureDrop. Release notes for 0.2.8.6 here.

This means that all SecureDrop instances are inaccessible. This includes Source Interface, Document Interface, and SSH access.

In a VM, the problem was simple to reproduce:

vagrant@app-staging:~$ sudo apt-get install --only-upgrade tor
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  mixmaster torbrowser-launcher socat tor-arm obfsproxy obfs4proxy
The following packages will be upgraded:
  tor
1 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.
Need to get 1,065 kB of archives.
After this operation, 217 kB of additional disk space will be used.
Get:1 http://deb.torproject.org/torproject.org/ trusty/main tor amd64 0.2.8.6-1~trusty+1 [1,065 kB]
Fetched 1,065 kB in 1s (693 kB/s)
(Reading database ... 44639 files and directories currently installed.)
Preparing to unpack .../tor_0.2.8.6-1~trusty+1_amd64.deb ...
 * Stopping tor daemon...
   ...done.
Unpacking tor (0.2.8.6-1~trusty+1) over (0.2.7.6-1~trusty+1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot
Setting up tor (0.2.8.6-1~trusty+1) ...
Installing new version of config file /etc/apparmor.d/system_tor ...
Warning from profile system_tor (/etc/apparmor.d/system_tor) ptrace rules not enforced
Warning from profile system_tor (/etc/apparmor.d/system_tor) signal rules not enforced
Warning from /etc/apparmor.d/system_tor (/etc/apparmor.d/system_tor line 22): profile system_tor network rules not enforced
 * Starting tor daemon...
Aug 04 17:17:43.779 [warn] Directory /var/lib/tor/services/ssh cannot be read: Permission denied
Aug 04 17:17:43.780 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
Aug 04 17:17:43.780 [err] Reading config failed--see warnings above.
   ...fail!
invoke-rc.d: initscript tor, action "start" failed.
Tor was unable to start due to configuration errors.
Please fix them and manually restart the tor daemon using
 ´service start tor´
vagrant@app-staging:~$

Investigating solutions now.

garrettr commented 8 years ago

It appears likely that Tor 0.2.8.6 changed the included AppArmor profile, which is causing this breakage. Others are reporting the same issue in the comments on the official release notes. Unfortunately there is no mention of changes to AppArmor in the release notes themselves, so I'm diffing the current and previous packages to try to find what changed.

garrettr commented 8 years ago

Relevant syslog from AppArmor:

Aug  4 17:25:08 mon-staging kernel: [  186.495333] audit: type=1400 audit(1470331508.136:26): apparmor="DENIED" operation="open" profile="system_tor" name="/var/lib/tor/services/ssh/" pid=643 comm="tor" requested_mask="r" denied_mask="r" fsuid=0 ouid=106
garrettr commented 8 years ago

For those following along at home:

garrettr commented 8 years ago

Ok, after discussing this issue with some Tor developers on IRC, here's what happened.

Explanation of the issue

  1. On Linux, when Tor is started as a service, it starts as root, does some stuff, then drops privileges to become the debian-tor user.
  2. Between Tor 0.2.7 and 0.2.8, there were various changes to the check_private_dir function, which is used to check various things about the state of the filesystem while Tor is in the initial phase of running as root. This includes checking for the existence any hidden service directories specified in torrc.
  3. Tor's debian package includes an AppArmor profile. In the initial version of this package for Ubuntu trusty (0.2.8.6-1~trusty+1), the profile allowed debian-tor to read any files under /var/lib/tor. See https://gitweb.torproject.org/debian/tor.git/tree/debian/tor.apparmor-profile?id=debian-tor-0.2.8.6-1#n7, specifically the line: owner /var/lib/tor/** rwk.
    • This should allow SecureDrop to work, because our torrc stores all hidden service configuration in a subdirectory of /var/lib/tor.
  4. Due to the changes in check_private_dir, Tor also needs to be allowed to read files in subdirectories of /var/lib/tor while it's running as root. However, the AppArmor rule in 0.2.8.6-1~trusty+1 only allowed this for owner, aka the debian-tor user.

The simplest fix is to change the AppArmor profile to allow Tor to read all files under /var/lib/tor, regardless of which user it's running as. This is implemented in 0.2.8.6-2 and documented in the changelog.

This fix has been tested in a SecureDrop staging environment and was confirmed to work there.

What should SecureDrop administrators do?

The updated packages for Ubuntu will become available on the Tor Project's Debian repository shortly (we'll post here when we've confirmed it's available). To fix this issue, you should install the updated Tor package, 0.2.8.6-2~trusty+1, which includes the fixes to the AppArmor profiles.

For most people, the best move is to sit tight and wait for SecureDrop to automatically upgrade, which will happen for everybody within 24 hours of the updated Tor package's release. This should automatically fix the issue, and within minutes your hidden services will begin working again and your SecureDrop will be available again.

If you want to get your SecureDrop online ASAP, you will still need to wait for Tor to release the updated packages. Once they do, you can log in to the servers and manually run cron-apt as root, which will trigger the same update as would have happened automatically, immediately. It is important to note that SecureDrop's SSH is only available as a Tor Hidden Service, so as a result of this issue with Tor, it is unavailable. Therefore, if you want to hasten the update by applying it manually, you will need console access to your SecureDrop servers.

conorsch commented 8 years ago

Great write-up @garrettr, thanks for your hard work on this! And kudos to the Tor Project for their fast response time.

conorsch commented 8 years ago

The updated package is live:

vagrant@app-staging:~$ apt-cache policy tor
tor:
  Installed: 0.2.8.6-2~trusty+1
  Candidate: 0.2.8.6-2~trusty+1
  Version table:
 *** 0.2.8.6-2~trusty+1 0
        500 http://deb.torproject.org/torproject.org/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.2.4.27-1build0.14.04.1 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
     0.2.4.20-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

Hidden services are working normally again.

conorsch commented 8 years ago

Instances are recovering well. If you run SecureDrop and are having trouble, contact us: http://securedrop-support.readthedocs.org/