freedomofpress / securedrop

GitHub repository for the SecureDrop whistleblower platform. Do not submit tips here!
https://securedrop.org/
Other
3.62k stars 688 forks source link

Mitigate "The Stack Clash" #1861

Closed garrettr closed 7 years ago

garrettr commented 7 years ago

Today Qulays published a writeup of a set of security vulnerabilities collectively called "The Stack Clash": their security advisory has the full technical details. Of special interest is the fact that Qualys was able to develop a PoC that works against grsecurity/PaX; see section IV.1.6. Grsecurity/PaX of the Qualys security advisory.

We should research this issue and determine what, if any, steps should be taken to mitigate the potential threat to SecureDrop installations.

garrettr commented 7 years ago

For background: SecureDrop runs on amd64 only, using an Ubuntu 14.04.4 LTS userland with a grsecurity-kernel hardened maintained by Freedom of the Press Foundation. The current version of the grsecurity kernel provided by FPF is 3.14.79. SecureDrop automatically checks for and installs any available security updates from the Ubuntu, Tor, and Freedom of the Press Foundation APT repositories every 24 hours.

garrettr commented 7 years ago

This is a fairly complex issue, and it's possible that this is only the beginning of a deluge of "stack clash"-type vulnerabilities. At the moment, the following vulnerabilities from the Qualys security advisory appear relevant to SecureDrop:

All of the other exploits/PoCs described in the Qualys security advisory do not appear to potentially impact SecureDrop because they are either specific to the i386 architecture, exploit vulnerabilities in software that is not included in SecureDrop (e.g. exim, SELinux), or exploit operating systems other than Linux.

garrettr commented 7 years ago

The IV.1.6. Grsecurity/PaX section of the Qualys security advisory is fairly reassuring when it comes to the prospects for the practical exploitation of "stack clash"-type vulnerabilities on grsecurity/PaX. While Qualys was able to demonstrate control of EIP on grsecurity, other grsecurity protections make a full exploit infeasible.

I have a couple of ideas for things we should consider doing to ensure SecureDrop is protected against this class of vulnerabilities. These recommendations are based on a careful reading of the Qualys security advisory and the grsecurity blog post on Stack Clash vulnerabilities.

Short-term

Long-term

We should investigate applying one or more of the solutions described in III. Solutions from the Qualys security advisory:

garrettr commented 7 years ago

Regarding PIE-enabled userland on Ubuntu:

The list of PIE-enabled packages linked to from the "Built as PIE" section of the Ubuntu security page does not include an explicit column for 14.04, so I am assuming the list of PIE-enabled packages for the most recent Ubuntu version listed (11.10) is accurate for 14.04.

On the bright side, this list includes some packages that I would consider "high risk" components of SecureDrop, including ssh, apache2, and ntp. I confirmed this by provisioning a staging environment and running hardening-check on these 3 applications.

Also good is that another key "high risk" component of SecureDrop, the Tor binary provided by Tor Project's package repository, is PIE-enabled:

root@app-staging:/home/vagrant# hardening-check $(which tor)
/usr/sbin/tor:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: yes (some protected functions found)
 Read-only relocations: yes
 Immediate binding: yes

On the not-so-bright side, one key SecureDrop dependency that is not PIE-enabled on Ubuntu 14.04 is Python. This was reported as a bug, but closed Won't Fix. The only mitigation appears to be to upgrade to a more recent version of Ubuntu that enables PIE on every package. I would consider this to be a point in favor of making #1530 a higher priority.

garrettr commented 7 years ago

We should also consider updating the SecureDrop grsecurity kernel, because it's been a while since the last update. While I don't have any specific concerns about the current kernel, it's good security hygiene to update it periodically.

psivesely commented 7 years ago

Probably a good time to point out that the grsec config used to build our kernels has not been pushed to any public repository, which is quite bad for transparency. All the grsec options you listed in your comment above @garrettr are enabled in https://github.com/freedomofpress/ansible-role-grsecurity/blob/0ad2bc7018bef479db8e2315ef8046aa36ac2f96/roles/build-grsec-kernel/files/config-digitalocean, which is a good indicator they are in our SD kernel as well, but we should be able to confirm this with certainty. Halp @conorsch!

conorsch commented 7 years ago

Probably a good time to point out that the grsec config used to build our kernels has not been pushed to any public repository, which is quite bad for transparency

Manual build process is documented here, and states that we're using the Ubuntu Trusty config with the grsecurity high security protections, with no further customizations.You access the full config for inspection at /boot/config-$(uname -r) e.g. in an app-staging VM.

Went through the list outlined above and checked off confirmation of the grsecurity kernel options currently live on production instances. Will continue with other aspects of confirmation on the short-term mitigations checklist.

psivesely commented 7 years ago

/proc/sys/vm/heap_stack_gap is set to the default 64KB. To permanently change this we want to add vm.heap_stack_gap=1048576 to /etc/sysctl.conf. An additional testinfra test should be added to confirm this setting persists.

garrettr commented 7 years ago

We discussed this in a meeting today and decided that our next steps will be:

Short-term steps

Medium-term

After we release 0.4, we decided we will prioritize #1530, so we can have PIE enabled for all of our userland and take full advantage of the the protections offered by grsecurity's ASLR.

Long-term

"Wait and see" how community and upstream distributions react to this vulnerability class; hopefully there will be an effort to implement-fstack-check or another long-term solution, which we can hopefully take advantage of along with the rest of the open source community.

conorsch commented 7 years ago

Confirmed cron-apt is working as expected on hardware, and the sudo update to version 1.8.9p5-1ubuntu1.4 (from 1.8.9p5-1ubuntu1.3, as described in USN-3304-1) is installed.

The lack of PIE on most Trusty binaries is disconcerting, but as @garrettr mentioned, most of our critical binaries are covered, and the long-term solution is to take advantage of a newer distro.

conorsch commented 7 years ago

Built working rc deb packages and ran through QA on them. I'm satisfied, and we'll get started with the release process. Lost some time rebuilding due to a bad Version in the linux-image package that wouldn't resolve as "greater than" in the context of apt upgrades, so had to tweak that value with the --revision flag to make-kpkg to force a resolution. Here's a high-level overview of confirming the results on a hardware test instance:

# Updated the apt sources list on `app` with testing repo.
# Overnight looks like the metapackage update worked:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576

mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 65536

# But the kernel-image update did NOT work, due to version parsing.
# Long story short, versioning in Debian packages is Hard.
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom

mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom

# Updated apt sources list on both machines and tried another upgrade.
# The command to trigger an upgrade is `sudo cron-apt -i -s`. Ran it on mon.
# It worked!
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-10.00.Custom

mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom

# Ran `sudo cron-apt -i -s` on app to confirm the same. It worked!
$ ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom

mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom

# Oh noes, mon still shows bad sysctl value:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576

mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 65536

# But of course! The mon server didn't get the metapackage overnight,
# which means it hasn't rebooted yet since the metapackage install:
$ ansible all -i inventory -u fpf -a 'uptime'
app | SUCCESS | rc=0 >>
 14:29:03 up  9:28,  0 users,  load average: 0.02, 0.07, 0.06

mon | SUCCESS | rc=0 >>
 14:29:03 up 10:09,  0 users,  load average: 0.00, 0.04, 0.05

# Rebooted mon:
$ ansible all -i inventory -u fpf -a 'uptime'
app | SUCCESS | rc=0 >>
 14:30:36 up  9:30,  0 users,  load average: 0.00, 0.05, 0.06

mon | SUCCESS | rc=0 >>
 14:30:47 up 0 min,  0 users,  load average: 0.00, 0.00, 0.00

# Confirmed good sysctl value was applied via metapackage:
$ ansible all -i inventory -u fpf -m shell -a 'sysctl -a | grep heap' -s
app | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576

mon | SUCCESS | rc=0 >>
vm.heap_stack_gap = 1048576

# Just to be absolutely sure, let's make sure we have that latest kernel:
ansible all -i inventory -u fpf -m shell -a 'aptitude show linux-image-3.14.79-grsec | grep ^Version'
mon | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom

app | SUCCESS | rc=0 >>
Version: 3.14.79-grsec-11.00.Custom

There will be PRs forthcoming on https://github.com/freedomofpress/ansible-role-grsecurity/ to document the new changes.

evilaliv3 commented 5 years ago

@redshiftzero @conorsch: It seems that currently ASLR is not effective on python3 as it is not compiled with -fpie

I just reported it to the Debian and Ubuntu security team trying to push for a quick fix.

https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1452115

evilaliv3@remotehost:~/Development/securedrop-app-code_1.0.0+xenial_amd64/data/opt/venvs/securedrop-app-code/bin$ hardening-check python3
python3:
 Position Independent Executable: no, normal executable!
 Stack protected: yes
 Fortify Source functions: yes (some protected functions found)
 Read-only relocations: yes
 Immediate binding: no, not found!

Do you mind to check on this as well?

evilaliv3 commented 5 years ago

Update: finally this is now under revision by the Ubuntu security team https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1452115

evilaliv3 commented 5 years ago

Some good updates, thanks to @mmaker we have identified why python3 is not hardened.

We are now trying to understand if its due to an intended choice for some reasons or if its just a bug.

The -fpie hardening flag is explicitly removed with: https://sources.debian.org/src/python3.7/3.7.4-4/debian/rules/#L6

@mmaker has already proposed a new package to Debian that is waiting for evaluation: https://salsa.debian.org/maker-guest/python3

All is tracked by this Debian ticket: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919134

The maintsainers of the package for ubuntu and debian is the same so hopefully we will have this fixed soon.

evilaliv3 commented 4 years ago

@redshiftzero @conorsch : actually we confirm that Ubuntu and Debian are not enabling -fpie on python3 for a concern around performances.

We verified that other distributions are enabling it with no concern and this includes Centos and Fedora.

Actually with @mmaker we are trying to implement some scripts so that if you like you could run an we could prove all together that the performance issue is minor with respect to the improved security.

@meejah @reaperhulk @Alex I think this could be relevant to many of your projects too.

alex commented 4 years ago

Is pyca/cryptography not compiled with -fpic (it surely must be...) that shold be the only requirement for ASLR to remain effective. Unless I missed something and there's another concern here?

mmaker commented 4 years ago

Hello there!

I have been trying to push the Debian for building python with PIE, but I didn't have the best of luck. I think at this time it's best if someone else continues to push forward the issue at my place.

This is what I think could be relevant for you guys:

You can build your own version of python with ASLR with the following commands on a debian sid:

apt install devscripts
apt source python3.8
cd python3.8-3.8.0/
sed -i '2,7d' debian/rules 
mk-build-deps
apt install ./python3.8-build-deps_3.8.0-4_amd64.deb
rm ./python3.8-build-deps_3.8.0-4_amd64.deb
DEB_BUILD_OPTIONS="nocheck nobench " dpkg-buildpackage -us -uc
cd ..

I don't really buy the argument of performance loss in a language like python, BUT in the Debian issue I tried adding some benchmarks.
DISCLAIMER: I had a factor 2-5x loss. I didn't disable Intel Turbo, and I had a browser running in the background.

You can set up your own benchmark environment with:

modprobe msr
apt install python3.8 python3.8-dev python3-pip
pip3 install pyperformance
python3 -m pyperf system tune

Then run (on the python3.8 available on the official repositories):

pyperformance run --python=/usr/bin/python3.8 -o py3.8.json

and compare with the one you generated:

apt install ./*.deb
pyperformance run --python=/usr/bin/python3.8 -o py3.8-pie.json
pyperformance compare py3.8.json py3.8-pie.json

Since the build is very slow, I uploaded a compiled version on my academic website. Hope this will be useful to you!

wget -r -np -nd --cut-dirs=2 -A '*.deb' https://www.di.ens.fr/~orru/pie/

@alex I am not sure what @evilaliv3 wanted to say there, but I think it would be really really helpful if other people kindly advocate in the Debian/Ubuntu issue for adding -fPIE when compiling python. It's just a switch, but it drastically reduces the attack surface!

Cheers,

evilaliv3 commented 4 years ago

There you go @mmaker!

I tested the performance against python 3.7 on debian buster on a a VPS on Scaleway.

Actually @mmaker i do not see any big loss; actually sometime is a loss of 1x and sometimes is gain of 1x.


pyperformance compare py3.7.json py3.7-pie.json
py3.7.json
==========

Performance version: 0.9.1
Report on Linux-4.19.0-5-amd64-x86_64-with-debian-10.2
Number of logical CPUs: 2
Start date: 2019-11-20 12:57:54.490063
End date: 2019-11-20 13:47:14.201201

py3.7-pie.json
==============

Performance version: 0.9.1
Report on Linux-4.19.0-5-amd64-x86_64-with-debian-10.2
Number of logical CPUs: 2
Start date: 2019-11-20 14:42:16.831054
End date: 2019-11-20 15:32:38.021224

### 2to3 ###
Mean +- std dev: 1.48 sec +- 0.07 sec -> 1.49 sec +- 0.09 sec: 1.01x slower
Not significant

### chameleon ###
Mean +- std dev: 39.6 ms +- 2.8 ms -> 40.4 ms +- 3.8 ms: 1.02x slower
Not significant

### chaos ###
Mean +- std dev: 440 ms +- 27 ms -> 455 ms +- 31 ms: 1.03x slower
Significant (t=-2.88)

### crypto_pyaes ###
Mean +- std dev: 396 ms +- 27 ms -> 408 ms +- 31 ms: 1.03x slower
Significant (t=-2.24)

### deltablue ###
Mean +- std dev: 26.8 ms +- 2.9 ms -> 27.5 ms +- 2.6 ms: 1.03x slower
Not significant

### django_template ###
Mean +- std dev: 505 ms +- 37 ms -> 513 ms +- 46 ms: 1.02x slower
Not significant

### dulwich_log ###
Mean +- std dev: 261 ms +- 17 ms -> 263 ms +- 20 ms: 1.00x slower
Not significant

### fannkuch ###
Mean +- std dev: 1.52 sec +- 0.07 sec -> 1.51 sec +- 0.07 sec: 1.00x faster
Not significant

### float ###
Mean +- std dev: 391 ms +- 30 ms -> 409 ms +- 32 ms: 1.04x slower
Significant (t=-3.05)

### genshi_text ###
Mean +- std dev: 135 ms +- 14 ms -> 140 ms +- 14 ms: 1.04x slower
Not significant

### genshi_xml ###
Mean +- std dev: 286 ms +- 19 ms -> 289 ms +- 32 ms: 1.01x slower
Not significant

### go ###
Mean +- std dev: 925 ms +- 73 ms -> 970 ms +- 77 ms: 1.05x slower
Significant (t=-3.33)

### hexiom ###
Mean +- std dev: 33.3 ms +- 3.9 ms -> 36.8 ms +- 3.3 ms: 1.10x slower
Significant (t=-5.21)

### html5lib ###
Mean +- std dev: 378 ms +- 36 ms -> 381 ms +- 48 ms: 1.01x slower
Not significant

### json_dumps ###
Mean +- std dev: 47.5 ms +- 6.4 ms -> 47.7 ms +- 4.5 ms: 1.00x slower
Not significant

### json_loads ###
Mean +- std dev: 91.0 us +- 9.4 us -> 93.8 us +- 9.6 us: 1.03x slower
Not significant

### logging_format ###
Mean +- std dev: 40.4 us +- 3.1 us -> 42.9 us +- 3.9 us: 1.06x slower
Significant (t=-3.91)

### logging_silent ###
Mean +- std dev: 666 ns +- 69 ns -> 639 ns +- 86 ns: 1.04x faster
Not significant

### logging_simple ###
Mean +- std dev: 38.2 us +- 3.2 us -> 36.8 us +- 4.3 us: 1.04x faster
Significant (t=2.01)

### mako ###
Mean +- std dev: 65.6 ms +- 6.9 ms -> 66.9 ms +- 6.9 ms: 1.02x slower
Not significant

### meteor_contest ###
Mean +- std dev: 371 ms +- 32 ms -> 415 ms +- 29 ms: 1.12x slower
Significant (t=-7.77)

### nbody ###
Mean +- std dev: 439 ms +- 40 ms -> 470 ms +- 38 ms: 1.07x slower
Significant (t=-4.26)

### nqueens ###
Mean +- std dev: 332 ms +- 22 ms -> 370 ms +- 23 ms: 1.11x slower
Significant (t=-9.26)

### pathlib ###
Mean +- std dev: 78.3 ms +- 7.5 ms -> 80.1 ms +- 7.4 ms: 1.02x slower
Not significant

### pickle ###
Mean +- std dev: 35.2 us +- 4.7 us -> 36.4 us +- 4.0 us: 1.03x slower
Not significant

### pickle_dict ###
Mean +- std dev: 80.9 us +- 8.2 us -> 94.6 us +- 8.4 us: 1.17x slower
Significant (t=-9.05)

### pickle_list ###
Mean +- std dev: 11.9 us +- 1.1 us -> 12.4 us +- 1.4 us: 1.04x slower
Significant (t=-2.32)

### pickle_pure_python ###
Mean +- std dev: 1.83 ms +- 0.23 ms -> 2.00 ms +- 0.20 ms: 1.09x slower
Significant (t=-4.24)

### pidigits ###
Mean +- std dev: 489 ms +- 27 ms -> 511 ms +- 37 ms: 1.05x slower
Significant (t=-3.80)

### python_startup ###
Mean +- std dev: 39.6 ms +- 4.2 ms -> 40.5 ms +- 4.6 ms: 1.02x slower
Significant (t=-2.02)

### python_startup_no_site ###
Mean +- std dev: 27.7 ms +- 3.0 ms -> 28.1 ms +- 3.0 ms: 1.01x slower
Not significant

### raytrace ###
Mean +- std dev: 1.90 sec +- 0.09 sec -> 1.99 sec +- 0.09 sec: 1.04x slower
Significant (t=-5.18)

### regex_compile ###
Mean +- std dev: 695 ms +- 55 ms -> 740 ms +- 55 ms: 1.06x slower
Significant (t=-4.42)

### regex_dna ###
Mean +- std dev: 531 ms +- 40 ms -> 564 ms +- 38 ms: 1.06x slower
Significant (t=-4.57)

### regex_effbot ###
Mean +- std dev: 11.0 ms +- 0.9 ms -> 11.0 ms +- 1.4 ms: 1.00x slower
Not significant

### regex_v8 ###
Mean +- std dev: 74.1 ms +- 6.4 ms -> 79.9 ms +- 9.5 ms: 1.08x slower
Significant (t=-3.93)

### richards ###
Mean +- std dev: 253 ms +- 24 ms -> 288 ms +- 18 ms: 1.14x slower
Significant (t=-8.97)

### scimark_fft ###
Mean +- std dev: 1.22 sec +- 0.07 sec -> 1.27 sec +- 0.05 sec: 1.04x slower
Significant (t=-4.00)

### scimark_lu ###
Mean +- std dev: 573 ms +- 57 ms -> 597 ms +- 42 ms: 1.04x slower
Significant (t=-2.55)

### scimark_monte_carlo ###
Mean +- std dev: 347 ms +- 29 ms -> 378 ms +- 38 ms: 1.09x slower
Significant (t=-4.96)

### scimark_sor ###
Mean +- std dev: 626 ms +- 37 ms -> 651 ms +- 57 ms: 1.04x slower
Significant (t=-2.77)

### scimark_sparse_mat_mult ###
Mean +- std dev: 14.9 ms +- 1.5 ms -> 14.1 ms +- 1.8 ms: 1.06x faster
Significant (t=2.80)

### spectral_norm ###
Mean +- std dev: 434 ms +- 34 ms -> 476 ms +- 41 ms: 1.10x slower
Significant (t=-6.04)

### sqlalchemy_declarative ###
Mean +- std dev: 936 ms +- 72 ms -> 955 ms +- 52 ms: 1.02x slower
Not significant

### sqlalchemy_imperative ###
Mean +- std dev: 145 ms +- 27 ms -> 139 ms +- 23 ms: 1.04x faster
Not significant

### sqlite_synth ###
Mean +- std dev: 10.6 us +- 0.9 us -> 11.3 us +- 0.8 us: 1.07x slower
Significant (t=-4.77)

### sympy_expand ###
Mean +- std dev: 1.66 sec +- 0.09 sec -> 1.75 sec +- 0.10 sec: 1.05x slower
Significant (t=-4.89)

### sympy_integrate ###
Mean +- std dev: 105 ms +- 11 ms -> 115 ms +- 10 ms: 1.10x slower
Significant (t=-5.49)

### sympy_str ###
Mean +- std dev: 1.24 sec +- 0.08 sec -> 1.26 sec +- 0.08 sec: 1.02x slower
Not significant

### sympy_sum ###
Mean +- std dev: 883 ms +- 91 ms -> 928 ms +- 67 ms: 1.05x slower
Significant (t=-3.09)

### telco ###
Mean +- std dev: 24.0 ms +- 2.1 ms -> 23.7 ms +- 2.6 ms: 1.01x faster
Not significant

### tornado_http ###
Mean +- std dev: 1.54 sec +- 0.11 sec -> 1.55 sec +- 0.10 sec: 1.00x slower
Not significant

### unpack_sequence ###
Mean +- std dev: 158 ns +- 18 ns -> 152 ns +- 16 ns: 1.04x faster
Significant (t=2.10)

### unpickle ###
Mean +- std dev: 51.5 us +- 5.2 us -> 49.1 us +- 5.5 us: 1.05x faster
Significant (t=2.49)

### unpickle_list ###
Mean +- std dev: 12.4 us +- 1.1 us -> 12.4 us +- 1.1 us: 1.00x faster
Not significant

### unpickle_pure_python ###
Mean +- std dev: 1.37 ms +- 0.17 ms -> 1.44 ms +- 0.18 ms: 1.05x slower
Significant (t=-2.10)

### xml_etree_generate ###
Mean +- std dev: 408 ms +- 33 ms -> 407 ms +- 27 ms: 1.00x faster
Not significant

### xml_etree_iterparse ###
Mean +- std dev: 593 ms +- 49 ms -> 600 ms +- 47 ms: 1.01x slower
Not significant

### xml_etree_parse ###
Mean +- std dev: 687 ms +- 37 ms -> 685 ms +- 42 ms: 1.00x faster
Not significant

### xml_etree_process ###
Mean +- std dev: 332 ms +- 31 ms -> 352 ms +- 27 ms: 1.06x slower
Significant (t=-3.94)```
evilaliv3 commented 2 years ago

Hi @legoktm ! @zenmonkeykstop just told me that you may have good new on this. Please feel free to reach out in case i could help somehow.

Thank you for soliciting this to the Debian Sec Team!

legoktm commented 2 years ago

I forgot to comment here when it happened, but https://tracker.debian.org/news/1316028/accepted-python310-3104-3-source-all-amd64-into-unstable-unstable/ - Python 3 now has PIE enabled and there's a separate -nopie package for people who want to disable it. That should be in bookworm and eventually make its way into Ubuntu (I assume).