GCC has unwanted flags

[andrewrk]

andy@xps:~/tmp$ NIX_DEBUG=true gcc
(some good flags for dealing with /nix/store/* and then...)
  -O2
  -D_FORTIFY_SOURCE=2
  -fstack-protector-strong
  --param
  ssp-buffer-size=4
  -fPIC
  -fno-strict-overflow
  -Wformat
  -Wformat-security
  -Werror=format-security

These flags should be enabled when compiling nix packages, but they should not be always on for general compiler use.

This has caused a broken debugging experience for me on the project I was working on.

Most projects have a "debug build" that has optimizations off, and this prevents that from working correctly.

NixOS version 16.09pre90254.6b20d5b (Flounder)

[Ralith]

@vcunat I do not want something different. I want real nixpkgs builds to respect the flags I pass.

[ElvishJerricco]

Frankly, I think cc-wrapper needs to be as simple as possible. The more rules someone has to know in order to use $CC properly, whether in a nix-shell or whatever else, the more likely it is to generate problems like this issue. Rather than adding rules about gcc and stdenv.cc being different and keeping complex rules about which arguments are automatically added where, it makes more sense to me to remove rules and simply say “there may be some arguments prepended to your argument list.”

[cstrahan]

@edolstra et al:

Is my PR (https://github.com/NixOS/nixpkgs/pull/28029/files) not suitable? If not, why? I think it gets us most of the way there - I’m willing to make necessary changes to fit the precise semantics we want with respect to failing fast (e.g. in the presence of multiple -O). Just tell me how to improve it and I’ll do so.

I should note that, as it stands, it makes nix-build and nix-shell behave the same way, while making system-wide installed gcc behave like what most people would expect (unhardened). So that part should please @Ralith, I’d think.

[andrewrk]

I'm going to throw out a crazy idea here:

It's up to each individual application how they want their software compiled. We can have hardening by default, but if they override it to disable hardening, the application wanted it that way. And that decision should be respected, and if we want to counter that decision we should file an upstream bug.

[musteresel]

@Ralith Yes, there are packages which must disable optimizations for (parts of) their build. An example is (IIRC, it's been a while since I worked on its sources) CLISP, which does some non-C-standard stuff and thus relies on the optimizer not messing that up.

[cstrahan]

/ping

@edolstra I hate to bug you personally/directly, but would you be able to specify your desired approach to fixing this bug - or if you can't spare the cycles for that, could you name an individual or two that you'd feel comfortable deferring to?

There's enough subtlety to the problem that I doubt everyone here will come to anything approaching a consensus. Nonetheless, the problem needs fixing, and I'd like to do the work to fix it - if someone with authority can state what an acceptable solution is.

Just as a reminder, I have this PR open: #28029

There's some feedback regarding cross compilation from @Ericson2314 that I can address, but I'm first waiting to know if we're going to accept that general approach, a variant of it, or something else entirely.

[edolstra]

@cstrahan The PR looks good to me, thanks. It might be nice to issue an error/warning if there is a conflict between a hardening flag and a user-supplied -O flag, but that's not super important.

[cstrahan]

@edolstra Awesome - thanks! I'll address @Ericson2314's points and try to get it ready for merging soon as I can - probably before the weekend is up.

[andrewrk]

Even though I originally filed the bug, and it bit me again after that, I still forgot about it again and got bit by this issue again, wasting the musl-lib developers' time.

[Ericson2314]

Hopefully @cstrahan's PR will be done soon. My bad recently for reviewing it slowly when he took it up again.

[glaebhoerl]

I ran into this as well, even though I'm on 18.09. After writing out most of a comment here to ask why, I realized I needed to re-instantiate my default.nix in order to actually pull in the new stuff. After doing that, I promptly got:

/nix/store/akak0rxhbi4n87z3nx78ipv76frvj841-glibc-2.27-dev/include/features.h:381:4: error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Werror=cpp]
 #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)

whenever I tried to build my project (which uses -Werror, yes), with no indication of what to do about it. Is this the expected behavior?

Since I've been reading this thread, I knew to try hardeningDisable = [ "fortify" ];, which thankfully fixed it, but Google was of no help at all. Now that I've written this comment, hopefully that might at least change. :)

(Yes I could've disabled -Werror, and if I only have to do that when also using -O0 I guess it's not such a huge deal, but it's surprising and perplexing nonetheless.)

[nh2]

We continue to be plagued by this; I've opened a new issue on #60919.

Copying the descrpition here, so that people subscribed here can see what it's about and subscribe there if interested (I think the discussion should probably continue there):

---

This is in some way a continuation of #18995 GCC has unwanted flags.

Even though that one is marked as solved, I have over the last 2 years encountered many situations where silently -O2 -D_FORTIFY_SOURCE=2 is added to builds where I have explicitly requested that a debug build be done (which should result in -O0 / no -O flag being passed).

For example, right now on meson-based builds (on Darwin, but I believe on Linux too), if I set

mesonBuildType = "debug";

it does not have the desired effect unless I also add

hardeningDisable = [ "fortify" ];

because the fortify hardening adds -O2 -D_FORTIFY_SOURCE=2 no matter what.

The same happend to me in various C++ builds, and it's always a big drag in development time when you just can't get your -O0 debug builds working.

---

Reading the PR description of #28029, it says

user supplied -O flags should take precedence over the hardening flags

so I suspect that this isn't good enough, because most development builds don't add -O0 explicitly -- instead they add no -O flag at all, so even with the PR -O2 is still added.

I think we must do something about it because it's a total pain.

[burdiyan]

Got hit by that too in 2022. So apparently it's not fixed. Been using gcc in a Nix shell, and couldn't perform debug on tests in Go because of the fortify flag.

[zopsicle]

It's important to note that fortification isn't the only issue here. For instance, clang is also passed -target implicitly. Having any behavioral flags passed to the compiler implicitly is unexpected to people who use Nix to obtain a development environment for C and C++ (and even Rust, where bindgen also uses cc-wrapper).

There are two Nix use cases where cc-wrapper is relevant:

1. Package maintainers: Writing derivations for software packages, so that these packages can be offered to users through Nixpkgs, following the Nixpkgs packaging conventions.
2. Software developers: Obtaining development tools for software developers to use, as part of a development environment. These will be invoked outside a Nix derivation's builder!

The current cc-wrapper only caters to the first use case. For the second use case, the only thing one wants from cc-wrapper is to pass directories to the compiler, so that it can find crt1, libc, compiler-rt, etc. Developers expect their tools to be more or less stock, not having arbitrary modifications made to suit a particular Linux distro.

I think the cleanest and most flexible solution would be to have a Nix function that takes a toolchain configuration and returns a set of toolchain derivations that generate appropriate wrappers. For the first use case, this function would be applied to the list of flags that are currently supplied by cc-wrapper. For the second use case, one could easily configure it to just do nothing special.

[zot]

I was able to get go testing / delve working in VS Code by launching it as NIX_HARDENING_ENABLE="" code

[SohamG]

I do not understand why this issue is closed. NixOS binaries are almost always significantly different from Ubuntu/etc binaries. This is an issue for systems programmers (and for people in an Operating Systems class like me). At the very least we need an opt-out/escape hash especially in nix-shell.

EDIT: This is taking into consideration hardeningDisable and NIX_HARDENING_ENABLE

[YellowOnion]

Trying to code with -Werror is stupid and annoying, how the hell do you disable it?

[RaunakChhatwal]

Why is nobody mentioning the fact that it is completely and totally unacceptable that these flags are passed when the stated purpose of the wrapper is to pass include and link paths? I just lost a whole day to trying to figure out why clang is acting up! Why can't compilers work like in any other distro, with the wrapper only passing include and link nix store paths?

[vcunat]

And other flags would be added by yet another layer of wrapping?

[RaunakChhatwal]

How about create a package called llvmPackages_16.clang_hardened with the wrapper that is currently in clang?

[vcunat]

I don't know... manually adding yet another attribute for each compiler doesn't sound attractive to me, but something similar might perhaps be done.

[zopsicle]

There is already gcc-unwrapped and clang-unwrapped, which do not have a wrapper. They only suffer from not being able to find headers and libraries. If there were a derivation that outputs a sysroot that could be passed to the compiler with --sysroot, that includes crucial stuff like libc, crt0, etc., that would pretty much solve the problems I think. (Maybe this is an oversimplification. There's also -B, not really sure what that does.)

cc-wrapper also forwards $NIX_CFLAGS_COMPILE to the compiler. This variable is set by stdenv's builder and contains -I flags and the like. Depending on your use case, this may or may not be useful, but nonetheless it can easily be passed explicitly if desired. (I personally never want this because I use CMake or pkg-config to find libraries.)

[RaunakChhatwal]

I most lean towards a solution where compilers work in nixos out of the box like they would in any other distro, which probably will still require a wrapper passing header and link nix store paths since nix profiles don't have an include directory. Maybe just change hardeningDisable to be all by default?

[andrewrk]

why can't you just pass the flags when building nix packages but not pass them otherwise? seems pretty straightforward to me.

[RaunakChhatwal]

Exactly, and it seems to me that the least disruptive way to do this would be to set the default hardeningDisable to [ "all" ] and all nix packages explicitly set hardeningDisable = [ ];

[vcunat]

No, I don't see it as simply. It's mostly described above in years old posts, but let me reiterate shortly. When you build a package with nixpkgs stdenv (via nix-shell, nix-build, etc.), naturally everything is as close as possible to building an actual nixpkgs package. That includes the hardening options that are default in nixpkgs packages.

First we'd have to find a good way how to tell if the user's intention is to contribute to nixpkgs (i.e. use nixpkgs hardening) or something else. And even then I'd think hard whether it's worth creating another set of default behavior.

(I'm not really considering running a compiler outside of nix-shell or nix-build. It isn't well usable when you need libraries.)

[mboisson]

We tried for years to use Nix as a basis for something else on top, and because of issues like this, we just gave up. Nix is all beautifully coherent when you build within Nix, not when you want to build on top of Nix. My recommendation is if you want to use Nix, use it all the way, don't build things outside of it. If you don't want to do that, use something else than Nix (we went for Gentoo Prefix).

[SohamG]

Exactly, and it seems to me that the least disruptive way to do this would be to set the default hardeningDisable to [ "all" ] and all nix packages explicitly set hardeningDisable = [ ];

I would like to point out that this is simply not enough. I came across an issue when compiling the toy OS for my OS Class where the result of standalone compiled ELF binary was larger than the max file size supported in the OS. The instructor and other classmates (all on non-nix systems) did NOT have this issue. I can reproduce this with the nix-shell settings quoted.

[RaunakChhatwal]

No, I don't see it as simply. It's mostly described above in years old posts, but let me reiterate shortly. When you build a package with nixpkgs stdenv (via nix-shell, nix-build, etc.), naturally everything is as close as possible to building an actual nixpkgs package. That includes the hardening options that are default in nixpkgs packages.

First we'd have to find a good way how to tell if the user's intention is to contribute to nixpkgs (i.e. use nixpkgs hardening) or something else. And even then I'd think hard whether it's worth creating another set of default behavior.

(I'm not really considering running a compiler outside of nix-shell or nix-build. It isn't well usable when you need libraries.)

Given that nix-shell is touted as a development environment, it comes as a surprise to me that everything has to be as close as possible to production. As you mentioned, compilers outside of nix-shell are unusable due to the missing headers, so development needs to happen in nix-shell.

[andrewrk]

I think hardening should be deleted entirely. It's too aggressive. You shouldn't just apply random flags to applications that the upstream authors didn't ask for. If they wanted hardening flags they would have put them upstream, too.

It's also a half-measure; a false sense of security, just like ASLR. We should move on to better things.

[vcunat]

Given that nix-shell is touted as a development environment, ...

nix-shell was designed to provide as close environment to nix-build as practical for development.

[Artturin]

Vars for only nixpkgs packages (no nix-shell) can be set at genericBuild Currently, it's done for no gzip timestamps https://github.com/NixOS/nixpkgs/blob/680602e764930d0a0000f0959db395903c38214b/pkgs/stdenv/generic/setup.sh#L1543-L1545

[WesleyAC]

Just ran into this problem, and I agree that this hardening code should be deleted, or at the very least turned off be default. Overriding upstream compiler flags by default for everything is not a good idea.

[chrispickard]

just lost two days to this bug, I'm really not sure why we are configuring gcc differently than everyone else outside of just ensuring it doesn't load from "system" directories. the error messages I got weren't even close to the issue (which is that fortify was turned on) so it took a long time to search random phrases and then eventually stumble on this issue. we have to do something about this. I believe I have even run into this bug before I just didn't realize it at the time.

[SohamG]

Chris Pickard @.***> writes:

I got weren't even close to the issue (which is that fortify was turned on) so it took a long time to search random phrases and then eventually stumble on this issue. we have to do something about this.

Is there a problem of this issue not being on the radar of people "in charge" of gcc in nixpkgs? Can't imagine that to be the case, yet all I see is this thread getting longer with more frustration and no update from the toolchain derivation maintainers. To be fair this issue was incorrectly marked as resolved a while ago...

[bjornfor]

AFAIR, nobody had a good solution to this issue until october this year: https://github.com/NixOS/nixpkgs/issues/18995#issuecomment-1772209375. Someone "just" have to take that idea and apply it to the compiler wrapper?

[SohamG]

Bjørn Forsman @.***> writes:

AFAIR, nobody had a good solution to this issue until october this year: #18995 (comment). Someone "just" have to take that idea and apply it to the compiler wrapper?

I just took a look - I think this issue is to do with the flags "baked in" to GCC when its built and therefore cannot be overriden with environment variables, especially cause the program being built would set $CFLAGS etc for itself. I think the solution is a different GCC derivation for the builder environment, and a setting in nix shell to choose a "normal" GCC or the one with the extra flags.

[jmarmstrong1207]

Any updates @fpletz @cstrahan ?

[jmarmstrong1207]

I'm not sure what the issue is though. These flags can be disabled and they are enabled by default not just in NixOS, but also in other distros such as Fedora and Ubuntu to increase security. Is there something I'm missing?

[zopsicle]

Other distroes enable these flags when building packages, but not for general purpose compiler use (i.e. if the user is a programmer writing their own programs, not distro packages). Nixpkgs wraps the compilers to always set these flags, regardless of use case.

[vcunat]

Yes, but Nix generally do not split these use cases. The normal way to do development is that you get nix-shell with everything set up as if doing a nix package build.

[zopsicle]

I see. Then, is there a tool that can provide development environments with Nix packages, for when you’re not building a Nix package? I think "developing software that will exist outside the Nix ecosystem, using tools provided by Nix in an ephemeral environment" is a use case that should be officially supported, and is in fact something a lot of people currently abuse (apparently) nix-shell for.

[jmarmstrong1207]

I believe distrobox is a pretty good alternative and is what you're looking for