[Suggestion] Hardening binaries via compilation flags

[shibumi]

Description: What's your idea?

Quite a while ago, Arch Linux has turned on many binary security features via compilation flags (2016)¹ or turned off options that are known to help exploit software (debugging symbols, RPATH). Now we have 2020 and Arch Linux made good experience with the additional security options.

I would like to suggest, that the CNCF considers making certain security related compilation flags default for every project.

We made good experience on Arch Linux with the following flags so far:

• FULL RELRO (Full Relocation Read-Only)²
• STACK CANARY³
• NX-Bit⁴
• PIE (Position Independent Executable/Code)⁵
• Setting no RPATH ⁶
• Setting no Symbols
• FORTIFY ⁷

Some of these flags are known to have effects on performance. A 'call for assistance'⁸ in 2016 tried to measure these effects via running different operations with binaries that have above flags enabled. The project⁹ delivered good results¹⁰ and we decided to enable these flags.

For validating EFI binaries we use the tool checksec.sh. The follows snippet shows the example output of check-sec --file=/usr/bin/kubectl with kubectl version 1.19.2-1 on Arch Linux:

RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH  Symbols     FORTIFY Fortified   Fortifiable FILE
Full RELRO      Canary found      NX enabled    PIE enabled     No RPATH   No RUNPATH   No Symbols    Yes   2       2       /usr/bin/kubectl

We achieved a fully hardened kubectl binary via the following methods:

1. We set CGO flags for passing enhanced security flags to the compiler:

export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"

These flags are:

• CPPFLAGS="-D_FORTIFY_SOURCE=2"
• CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
• CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
• LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"

2. We compile our Go packages with the following GOFLAGS:

export CGO_ENABLED=1
export GOFLAGS="-buildmode=pie -trimpath -mod=readonly -modcacherw"
export GOLDFLAGS="-linkmode=external"

Explanation for each flag:

• CGO_ENABLED needs to be set to 1 for most of the features.. this can make binary distribution problematic.
• -buildmode=pie enables PIE compilation for binary harderning.
• -trimpath important for Reproducible Builds so full build paths and module paths are not embedded.
• -mod=readonly ensure the module files are not updated in any go actions.
• -modcacherw is not important, but it ensures that go modules creates a write-able path. Default is read-only.
• -linkmode=external ensure we use an external linker, because the Go linker is not capable of all operations that are necessary.

Our full Go package guidelines can be found in our Wiki¹².

Impact: Describe your hopes for how this would reduce risk for the cloud native ecosystem. Who will this help? How will it help them?

The impact would be enhanced security for all compiled binaries. This approach would make it remote and local attackers more difficult to exploit known security vulnerabilities.

Scope: How much effort will this take? ok to provide a range of options if or "not yet determined"

I see the following challenges:

1. We might cannot enable all flags for all projects, because in some projects good performance can be key.
2. I see no problem with enabling such options for -ctl binaries. These are very often just helper tools for controlling an API endpoint.
3. Enabling these flags and validating them can lead to additional work in Makefiles, build processes and testing.
4. Due to the use of CGO_ENABLED=1 we will destroy our current static binary distribution process. However, the glibc is ABI stable, thus it should work for most linux distributions (except the linux distribution makes use of dietlibc or other libc implementations).
5. Other operating systems such like Mac OS or Windows may need different security flags.

Additional info:

1 https://lists.archlinux.org/pipermail/arch-dev-public/2016-October/028405.html 2 https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro 3 https://en.wikipedia.org/wiki/Stack_buffer_overflow#Stack_canaries 4 https://en.wikipedia.org/wiki/NX_bit 5 https://en.wikipedia.org/wiki/Position-independent_code 6 https://en.wikipedia.org/wiki/Rpath 7 https://access.redhat.com/blogs/766093/posts/1976213 8 https://www.archlinux.org/news/test-sec-flags-call-for-assistance/ 9 https://github.com/pid1/test-sec-flags 10 https://github.com/pid1/test-sec-flags/wiki 11 https://github.com/slimm609/checksec.sh 12 https://wiki.archlinux.org/index.php/Go_package_guidelines#Flag_meaning

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[anthraxx]

Dear friendly bot, it may be that just nobody ever answered from cncf, which is sad :smirk_cat:

[TheFoxAtWork]

Deepest apologies on not getting to this sooner. I've got this scheduled for discussion January 27th. @shibumi would you be willing to discuss? This looks related to the #501 discussion as well as some framework automation in #496

[TheFoxAtWork]

Should be more language agnostic, recommend inclusion into our security assessments. @JustinCappos @lumjjb potential for inclusion in automated pipeline checks

[shibumi]

Hey @TheFoxAtWork sorry for the late reply. I am a very shy person, so take the issue more as an anonymous input/idea/suggestion for future discussions. It was meant as an idea for future binary distribution and as a possible way how to achieve this via having a look on how Arch Linux is doing this right now. Sorry :S

[jlk]

@shibumi - no reason to be sorry at all!! :) And thanks for the submission.

As this sits in my mind after the sig-security call, the idea's growing on me. If we could maybe do a guidelines doc for the top 3-4 CNCF languages (eg do these 10 things for golang), I bet that'd be super helpful to people...

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[anthraxx]

Annoyingly this bot forces us to bump threads yet again :cat:

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[shibumi]

bump :)

[lumjjb]

Looks like this issue has a fleeting interest, we will keep it open for now, but I think the next stale cycle, we would have to put it into the suggestion archive.

[shibumi]

@lumjjb Good decision. I guess, nobody likes a wall of bump messages :)

[lumjjb]

haha we also have too many issues and triaging becomes a bit of a pain so closing issues is becoming a bit more of a consideration now

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[anvega]

hey @shibumi and @anthraxx, I'd like to bring your suggestion back to life with a few modifications.

I understand the original goal sought to make compilation flags. That may not be achievable, given that every project makes its own technical choices. In that regard, the CNCF is, in a sense, a federation.

What is doable is we can have you formulate a recommendation targeted at all projects, which we can help publish and disseminate. In it, compelling arguments as to why applicable projects should follow it by presenting defensible arguments, including the tradeoffs. The first step would be to convert what you wrote in your original post into a document, and we can then merge that into this repo and then broadcast it in the CNCF mailer list and on cncf.io/blog.

If this is still something of interest to you, I'd be happy to see it through. Let me know what you think.

[anvega]

Closing this issue due to inactivity. Should you be interested in pursuing the idea down the line again with possibily broadening the audience to all foundations in The LF by aiming to publish the guidance LF-wide, feel free to comment/reopen the issue.