Closed garrettr closed 7 years ago
For background: SecureDrop runs on amd64 only, using an Ubuntu 14.04.4 LTS userland with a grsecurity-kernel hardened maintained by Freedom of the Press Foundation. The current version of the grsecurity kernel provided by FPF is 3.14.79. SecureDrop automatically checks for and installs any available security updates from the Ubuntu, Tor, and Freedom of the Press Foundation APT repositories every 24 hours.
This is a fairly complex issue, and it's possible that this is only the beginning of a deluge of "stack clash"-type vulnerabilities. At the moment, the following vulnerabilities from the Qualys security advisory appear relevant to SecureDrop:
All of the other exploits/PoCs described in the Qualys security advisory do not appear to potentially impact SecureDrop because they are either specific to the i386 architecture, exploit vulnerabilities in software that is not included in SecureDrop (e.g. exim, SELinux), or exploit operating systems other than Linux.
The IV.1.6. Grsecurity/PaX
section of the Qualys security advisory is fairly reassuring when it comes to the prospects for the practical exploitation of "stack clash"-type vulnerabilities on grsecurity/PaX. While Qualys was able to demonstrate control of EIP on grsecurity, other grsecurity protections make a full exploit infeasible.
I have a couple of ideas for things we should consider doing to ensure SecureDrop is protected against this class of vulnerabilities. These recommendations are based on a careful reading of the Qualys security advisory and the grsecurity blog post on Stack Clash vulnerabilities.
sudo
).IV.1.6. Grsecurity/PaX
) are either configurable or depend on cooperation with userland. We should verify that all of these measures are enabled on SecureDrop systems so we can be confident that we are fully protected.
hardening-check
might be useful here.GRKERNSEC_PROC_MEMMAP
RLIMIT_STACK
of SUID binaries to 8MB, described as an obstacle in the Qualys security advisory.GRKERNSEC_BRUTE
PAX_MEMORY_STACKLEAK
GRKERNSEC_KSTACKOVERFLOW
PAX_MEMORY_STACKLEAK
and GRKERNSEC_KSTACKOVERFLOW
protect the kernel from "stack clash" attacks (and stack overflow attacks generally), according to the grsecurity blog post.We should investigate applying one or more of the solutions described in III. Solutions
from the Qualys security advisory:
/proc/sys/vm/heap_stack_gap
.-fstack-check
on Ubuntu's Security Features page. That may be about to change in the aftermath of this new report.Regarding PIE-enabled userland on Ubuntu:
The list of PIE-enabled packages linked to from the "Built as PIE" section of the Ubuntu security page does not include an explicit column for 14.04, so I am assuming the list of PIE-enabled packages for the most recent Ubuntu version listed (11.10) is accurate for 14.04.
On the bright side, this list includes some packages that I would consider "high risk" components of SecureDrop, including ssh
, apache2
, and ntp
. I confirmed this by provisioning a staging environment and running hardening-check
on these 3 applications.
Also good is that another key "high risk" component of SecureDrop, the Tor binary provided by Tor Project's package repository, is PIE-enabled:
root@app-staging:/home/vagrant# hardening-check $(which tor)
/usr/sbin/tor:
Position Independent Executable: yes
Stack protected: yes
Fortify Source functions: yes (some protected functions found)
Read-only relocations: yes
Immediate binding: yes
On the not-so-bright side, one key SecureDrop dependency that is not PIE-enabled on Ubuntu 14.04 is Python. This was reported as a bug, but closed Won't Fix
. The only mitigation appears to be to upgrade to a more recent version of Ubuntu that enables PIE on every package. I would consider this to be a point in favor of making #1530 a higher priority.
We should also consider updating the SecureDrop grsecurity kernel, because it's been a while since the last update. While I don't have any specific concerns about the current kernel, it's good security hygiene to update it periodically.
Probably a good time to point out that the grsec config used to build our kernels has not been pushed to any public repository, which is quite bad for transparency. All the grsec options you listed in your comment above @garrettr are enabled in https://github.com/freedomofpress/ansible-role-grsecurity/blob/0ad2bc7018bef479db8e2315ef8046aa36ac2f96/roles/build-grsec-kernel/files/config-digitalocean, which is a good indicator they are in our SD kernel as well, but we should be able to confirm this with certainty. Halp @conorsch!
Probably a good time to point out that the grsec config used to build our kernels has not been pushed to any public repository, which is quite bad for transparency
Manual build process is documented here, and states that we're using the Ubuntu Trusty config with the grsecurity high security protections, with no further customizations.You access the full config for inspection at /boot/config-$(uname -r)
e.g. in an app-staging VM.
Went through the list outlined above and checked off confirmation of the grsecurity kernel options currently live on production instances. Will continue with other aspects of confirmation on the short-term mitigations checklist.
/proc/sys/vm/heap_stack_gap
is set to the default 64KB. To permanently change this we want to add vm.heap_stack_gap=1048576
to /etc/sysctl.conf
. An additional testinfra test should be added to confirm this setting persists.
We discussed this in a meeting today and decided that our next steps will be:
heap_stack_gap
via both Ansible playbooks and a postinst
hook in the securedrop-grsec
metapackage. This will ensure the most feasible mitigation recommended by Qualys will be deployed ASAP.After we release 0.4, we decided we will prioritize #1530, so we can have PIE enabled for all of our userland and take full advantage of the the protections offered by grsecurity's ASLR.
"Wait and see" how community and upstream distributions react to this vulnerability class; hopefully there will be an effort to implement-fstack-check
or another long-term solution, which we can hopefully take advantage of along with the rest of the open source community.
Confirmed cron-apt
is working as expected on hardware, and the sudo
update to version 1.8.9p5-1ubuntu1.4
(from 1.8.9p5-1ubuntu1.3
, as described in USN-3304-1) is installed.
The lack of PIE on most Trusty binaries is disconcerting, but as @garrettr mentioned, most of our critical binaries are covered, and the long-term solution is to take advantage of a newer distro.
Built working rc deb packages and ran through QA on them. I'm satisfied, and we'll get started with the release process. Lost some time rebuilding due to a bad Version
in the linux-image package that wouldn't resolve as "greater than" in the context of apt upgrades, so had to tweak that value with the --revision
flag to make-kpkg
to force a resolution. Here's a high-level overview of confirming the results on a hardware test instance:
# Updated the apt sources list on `app` with testing repo.
# Overnight looks like the metapackage update worked:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576
mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 65536
# But the kernel-image update did NOT work, due to version parsing.
# Long story short, versioning in Debian packages is Hard.
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom
mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom
# Updated apt sources list on both machines and tried another upgrade.
# The command to trigger an upgrade is `sudo cron-apt -i -s`. Ran it on mon.
# It worked!
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom
mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom
# Ran `sudo cron-apt -i -s` on app to confirm the same. It worked!
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom
mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom
# Oh noes, mon still shows bad sysctl value:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576
mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 65536
# But of course! The mon server didn't get the metapackage overnight,
# which means it hasn't rebooted yet since the metapackage install:
$ ansible all -i inventory -u fpf -a 'uptime'
app | SUCCESS | rc=0 >>
14:29:03 up 9:28, 0 users, load average: 0.02, 0.07, 0.06
mon | SUCCESS | rc=0 >>
14:29:03 up 10:09, 0 users, load average: 0.00, 0.04, 0.05
# Rebooted mon:
$ ansible all -i inventory -u fpf -a 'uptime'
app | SUCCESS | rc=0 >>
14:30:36 up 9:30, 0 users, load average: 0.00, 0.05, 0.06
mon | SUCCESS | rc=0 >>
14:30:47 up 0 min, 0 users, load average: 0.00, 0.00, 0.00
# Confirmed good sysctl value was applied via metapackage:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576
mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576
# Just to be absolutely sure, let's make sure we have that latest kernel:
ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom
There will be PRs forthcoming on https://github.com/freedomofpress/ansible-role-grsecurity/ to document the new changes.
@redshiftzero @conorsch: It seems that currently ASLR is not effective on python3 as it is not compiled with -fpie
I just reported it to the Debian and Ubuntu security team trying to push for a quick fix.
https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1452115
evilaliv3@remotehost:~/Development/securedrop-app-code_1.0.0+xenial_amd64/data/opt/venvs/securedrop-app-code/bin$ hardening-check python3
python3:
Position Independent Executable: no, normal executable!
Stack protected: yes
Fortify Source functions: yes (some protected functions found)
Read-only relocations: yes
Immediate binding: no, not found!
Do you mind to check on this as well?
Update: finally this is now under revision by the Ubuntu security team https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1452115
Some good updates, thanks to @mmaker we have identified why python3 is not hardened.
We are now trying to understand if its due to an intended choice for some reasons or if its just a bug.
The -fpie hardening flag is explicitly removed with: https://sources.debian.org/src/python3.7/3.7.4-4/debian/rules/#L6
@mmaker has already proposed a new package to Debian that is waiting for evaluation: https://salsa.debian.org/maker-guest/python3
All is tracked by this Debian ticket: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919134
The maintsainers of the package for ubuntu and debian is the same so hopefully we will have this fixed soon.
@redshiftzero @conorsch : actually we confirm that Ubuntu and Debian are not enabling -fpie on python3 for a concern around performances.
We verified that other distributions are enabling it with no concern and this includes Centos and Fedora.
Actually with @mmaker we are trying to implement some scripts so that if you like you could run an we could prove all together that the performance issue is minor with respect to the improved security.
@meejah @reaperhulk @Alex I think this could be relevant to many of your projects too.
Is pyca/cryptography not compiled with -fpic
(it surely must be...) that shold be the only requirement for ASLR to remain effective. Unless I missed something and there's another concern here?
Hello there!
I have been trying to push the Debian for building python with PIE, but I didn't have the best of luck. I think at this time it's best if someone else continues to push forward the issue at my place.
This is what I think could be relevant for you guys:
You can build your own version of python with ASLR with the following commands on a debian sid:
apt install devscripts
apt source python3.8
cd python3.8-3.8.0/
sed -i '2,7d' debian/rules
mk-build-deps
apt install ./python3.8-build-deps_3.8.0-4_amd64.deb
rm ./python3.8-build-deps_3.8.0-4_amd64.deb
DEB_BUILD_OPTIONS="nocheck nobench " dpkg-buildpackage -us -uc
cd ..
I don't really buy the argument of performance loss in a language like python, BUT in the Debian issue I tried adding some benchmarks.
DISCLAIMER: I had a factor 2-5x loss. I didn't disable Intel Turbo, and I had a browser running in the background.
You can set up your own benchmark environment with:
modprobe msr
apt install python3.8 python3.8-dev python3-pip
pip3 install pyperformance
python3 -m pyperf system tune
Then run (on the python3.8 available on the official repositories):
pyperformance run --python=/usr/bin/python3.8 -o py3.8.json
and compare with the one you generated:
apt install ./*.deb
pyperformance run --python=/usr/bin/python3.8 -o py3.8-pie.json
pyperformance compare py3.8.json py3.8-pie.json
Since the build is very slow, I uploaded a compiled version on my academic website. Hope this will be useful to you!
wget -r -np -nd --cut-dirs=2 -A '*.deb' https://www.di.ens.fr/~orru/pie/
@alex I am not sure what @evilaliv3 wanted to say there, but I think it would be really really helpful if other people kindly advocate in the Debian/Ubuntu issue for adding -fPIE when compiling python. It's just a switch, but it drastically reduces the attack surface!
Cheers,
There you go @mmaker!
I tested the performance against python 3.7 on debian buster on a a VPS on Scaleway.
Actually @mmaker i do not see any big loss; actually sometime is a loss of 1x and sometimes is gain of 1x.
pyperformance compare py3.7.json py3.7-pie.json
py3.7.json
==========
Performance version: 0.9.1
Report on Linux-4.19.0-5-amd64-x86_64-with-debian-10.2
Number of logical CPUs: 2
Start date: 2019-11-20 12:57:54.490063
End date: 2019-11-20 13:47:14.201201
py3.7-pie.json
==============
Performance version: 0.9.1
Report on Linux-4.19.0-5-amd64-x86_64-with-debian-10.2
Number of logical CPUs: 2
Start date: 2019-11-20 14:42:16.831054
End date: 2019-11-20 15:32:38.021224
### 2to3 ###
Mean +- std dev: 1.48 sec +- 0.07 sec -> 1.49 sec +- 0.09 sec: 1.01x slower
Not significant
### chameleon ###
Mean +- std dev: 39.6 ms +- 2.8 ms -> 40.4 ms +- 3.8 ms: 1.02x slower
Not significant
### chaos ###
Mean +- std dev: 440 ms +- 27 ms -> 455 ms +- 31 ms: 1.03x slower
Significant (t=-2.88)
### crypto_pyaes ###
Mean +- std dev: 396 ms +- 27 ms -> 408 ms +- 31 ms: 1.03x slower
Significant (t=-2.24)
### deltablue ###
Mean +- std dev: 26.8 ms +- 2.9 ms -> 27.5 ms +- 2.6 ms: 1.03x slower
Not significant
### django_template ###
Mean +- std dev: 505 ms +- 37 ms -> 513 ms +- 46 ms: 1.02x slower
Not significant
### dulwich_log ###
Mean +- std dev: 261 ms +- 17 ms -> 263 ms +- 20 ms: 1.00x slower
Not significant
### fannkuch ###
Mean +- std dev: 1.52 sec +- 0.07 sec -> 1.51 sec +- 0.07 sec: 1.00x faster
Not significant
### float ###
Mean +- std dev: 391 ms +- 30 ms -> 409 ms +- 32 ms: 1.04x slower
Significant (t=-3.05)
### genshi_text ###
Mean +- std dev: 135 ms +- 14 ms -> 140 ms +- 14 ms: 1.04x slower
Not significant
### genshi_xml ###
Mean +- std dev: 286 ms +- 19 ms -> 289 ms +- 32 ms: 1.01x slower
Not significant
### go ###
Mean +- std dev: 925 ms +- 73 ms -> 970 ms +- 77 ms: 1.05x slower
Significant (t=-3.33)
### hexiom ###
Mean +- std dev: 33.3 ms +- 3.9 ms -> 36.8 ms +- 3.3 ms: 1.10x slower
Significant (t=-5.21)
### html5lib ###
Mean +- std dev: 378 ms +- 36 ms -> 381 ms +- 48 ms: 1.01x slower
Not significant
### json_dumps ###
Mean +- std dev: 47.5 ms +- 6.4 ms -> 47.7 ms +- 4.5 ms: 1.00x slower
Not significant
### json_loads ###
Mean +- std dev: 91.0 us +- 9.4 us -> 93.8 us +- 9.6 us: 1.03x slower
Not significant
### logging_format ###
Mean +- std dev: 40.4 us +- 3.1 us -> 42.9 us +- 3.9 us: 1.06x slower
Significant (t=-3.91)
### logging_silent ###
Mean +- std dev: 666 ns +- 69 ns -> 639 ns +- 86 ns: 1.04x faster
Not significant
### logging_simple ###
Mean +- std dev: 38.2 us +- 3.2 us -> 36.8 us +- 4.3 us: 1.04x faster
Significant (t=2.01)
### mako ###
Mean +- std dev: 65.6 ms +- 6.9 ms -> 66.9 ms +- 6.9 ms: 1.02x slower
Not significant
### meteor_contest ###
Mean +- std dev: 371 ms +- 32 ms -> 415 ms +- 29 ms: 1.12x slower
Significant (t=-7.77)
### nbody ###
Mean +- std dev: 439 ms +- 40 ms -> 470 ms +- 38 ms: 1.07x slower
Significant (t=-4.26)
### nqueens ###
Mean +- std dev: 332 ms +- 22 ms -> 370 ms +- 23 ms: 1.11x slower
Significant (t=-9.26)
### pathlib ###
Mean +- std dev: 78.3 ms +- 7.5 ms -> 80.1 ms +- 7.4 ms: 1.02x slower
Not significant
### pickle ###
Mean +- std dev: 35.2 us +- 4.7 us -> 36.4 us +- 4.0 us: 1.03x slower
Not significant
### pickle_dict ###
Mean +- std dev: 80.9 us +- 8.2 us -> 94.6 us +- 8.4 us: 1.17x slower
Significant (t=-9.05)
### pickle_list ###
Mean +- std dev: 11.9 us +- 1.1 us -> 12.4 us +- 1.4 us: 1.04x slower
Significant (t=-2.32)
### pickle_pure_python ###
Mean +- std dev: 1.83 ms +- 0.23 ms -> 2.00 ms +- 0.20 ms: 1.09x slower
Significant (t=-4.24)
### pidigits ###
Mean +- std dev: 489 ms +- 27 ms -> 511 ms +- 37 ms: 1.05x slower
Significant (t=-3.80)
### python_startup ###
Mean +- std dev: 39.6 ms +- 4.2 ms -> 40.5 ms +- 4.6 ms: 1.02x slower
Significant (t=-2.02)
### python_startup_no_site ###
Mean +- std dev: 27.7 ms +- 3.0 ms -> 28.1 ms +- 3.0 ms: 1.01x slower
Not significant
### raytrace ###
Mean +- std dev: 1.90 sec +- 0.09 sec -> 1.99 sec +- 0.09 sec: 1.04x slower
Significant (t=-5.18)
### regex_compile ###
Mean +- std dev: 695 ms +- 55 ms -> 740 ms +- 55 ms: 1.06x slower
Significant (t=-4.42)
### regex_dna ###
Mean +- std dev: 531 ms +- 40 ms -> 564 ms +- 38 ms: 1.06x slower
Significant (t=-4.57)
### regex_effbot ###
Mean +- std dev: 11.0 ms +- 0.9 ms -> 11.0 ms +- 1.4 ms: 1.00x slower
Not significant
### regex_v8 ###
Mean +- std dev: 74.1 ms +- 6.4 ms -> 79.9 ms +- 9.5 ms: 1.08x slower
Significant (t=-3.93)
### richards ###
Mean +- std dev: 253 ms +- 24 ms -> 288 ms +- 18 ms: 1.14x slower
Significant (t=-8.97)
### scimark_fft ###
Mean +- std dev: 1.22 sec +- 0.07 sec -> 1.27 sec +- 0.05 sec: 1.04x slower
Significant (t=-4.00)
### scimark_lu ###
Mean +- std dev: 573 ms +- 57 ms -> 597 ms +- 42 ms: 1.04x slower
Significant (t=-2.55)
### scimark_monte_carlo ###
Mean +- std dev: 347 ms +- 29 ms -> 378 ms +- 38 ms: 1.09x slower
Significant (t=-4.96)
### scimark_sor ###
Mean +- std dev: 626 ms +- 37 ms -> 651 ms +- 57 ms: 1.04x slower
Significant (t=-2.77)
### scimark_sparse_mat_mult ###
Mean +- std dev: 14.9 ms +- 1.5 ms -> 14.1 ms +- 1.8 ms: 1.06x faster
Significant (t=2.80)
### spectral_norm ###
Mean +- std dev: 434 ms +- 34 ms -> 476 ms +- 41 ms: 1.10x slower
Significant (t=-6.04)
### sqlalchemy_declarative ###
Mean +- std dev: 936 ms +- 72 ms -> 955 ms +- 52 ms: 1.02x slower
Not significant
### sqlalchemy_imperative ###
Mean +- std dev: 145 ms +- 27 ms -> 139 ms +- 23 ms: 1.04x faster
Not significant
### sqlite_synth ###
Mean +- std dev: 10.6 us +- 0.9 us -> 11.3 us +- 0.8 us: 1.07x slower
Significant (t=-4.77)
### sympy_expand ###
Mean +- std dev: 1.66 sec +- 0.09 sec -> 1.75 sec +- 0.10 sec: 1.05x slower
Significant (t=-4.89)
### sympy_integrate ###
Mean +- std dev: 105 ms +- 11 ms -> 115 ms +- 10 ms: 1.10x slower
Significant (t=-5.49)
### sympy_str ###
Mean +- std dev: 1.24 sec +- 0.08 sec -> 1.26 sec +- 0.08 sec: 1.02x slower
Not significant
### sympy_sum ###
Mean +- std dev: 883 ms +- 91 ms -> 928 ms +- 67 ms: 1.05x slower
Significant (t=-3.09)
### telco ###
Mean +- std dev: 24.0 ms +- 2.1 ms -> 23.7 ms +- 2.6 ms: 1.01x faster
Not significant
### tornado_http ###
Mean +- std dev: 1.54 sec +- 0.11 sec -> 1.55 sec +- 0.10 sec: 1.00x slower
Not significant
### unpack_sequence ###
Mean +- std dev: 158 ns +- 18 ns -> 152 ns +- 16 ns: 1.04x faster
Significant (t=2.10)
### unpickle ###
Mean +- std dev: 51.5 us +- 5.2 us -> 49.1 us +- 5.5 us: 1.05x faster
Significant (t=2.49)
### unpickle_list ###
Mean +- std dev: 12.4 us +- 1.1 us -> 12.4 us +- 1.1 us: 1.00x faster
Not significant
### unpickle_pure_python ###
Mean +- std dev: 1.37 ms +- 0.17 ms -> 1.44 ms +- 0.18 ms: 1.05x slower
Significant (t=-2.10)
### xml_etree_generate ###
Mean +- std dev: 408 ms +- 33 ms -> 407 ms +- 27 ms: 1.00x faster
Not significant
### xml_etree_iterparse ###
Mean +- std dev: 593 ms +- 49 ms -> 600 ms +- 47 ms: 1.01x slower
Not significant
### xml_etree_parse ###
Mean +- std dev: 687 ms +- 37 ms -> 685 ms +- 42 ms: 1.00x faster
Not significant
### xml_etree_process ###
Mean +- std dev: 332 ms +- 31 ms -> 352 ms +- 27 ms: 1.06x slower
Significant (t=-3.94)```
Hi @legoktm ! @zenmonkeykstop just told me that you may have good new on this. Please feel free to reach out in case i could help somehow.
Thank you for soliciting this to the Debian Sec Team!
I forgot to comment here when it happened, but https://tracker.debian.org/news/1316028/accepted-python310-3104-3-source-all-amd64-into-unstable-unstable/ - Python 3 now has PIE enabled and there's a separate -nopie package for people who want to disable it. That should be in bookworm and eventually make its way into Ubuntu (I assume).
Today Qulays published a writeup of a set of security vulnerabilities collectively called "The Stack Clash": their security advisory has the full technical details. Of special interest is the fact that Qualys was able to develop a PoC that works against grsecurity/PaX; see section
IV.1.6. Grsecurity/PaX
of the Qualys security advisory.We should research this issue and determine what, if any, steps should be taken to mitigate the potential threat to SecureDrop installations.