Virtualbox and Linux Kernel can't compile with ccache

[Ninlives]

Describe the bug Can't compile virtualbox and linux kernel with ccache enabled. Compiling linux failed with

Compiler lacks asm-goto support.

Compiling virtualbox failed with

** cannot execute ‘gcc -dumpversion’!

To Reproduce Steps to reproduce the behavior:

1. Add these configuration:

   programs.ccache.enable = true;
   programs.ccache.packageNames = [ "linux_5_3" "virtualbox" ];

2. run nixos-rebuild boot/build/switch

Metadata

• system: "x86_64-linux"
• host os: Linux 5.3.8, NixOS, 20.03pre200231.7827d3f4497 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.1
• channels(mlatus): "home-manager, nixpkgs-20.03pre200231.7827d3f4497, rolling-20.03pre200508.c4821a82f88"
• nixpkgs: /home/mlatus/.nix-defexpr/channels/nixpkgs

[Ninlives]

Add nix.sandboxPaths = [ config.programs.ccache.cacheDir ] and the compilation for linux kernel seems to be OK, but compiling virtualbox fails because configure could not find header files needed for 32bit support:

***** Checking 32-bit support *****                                                                                                                                       
compiling the following source file:                                                                                                                                      
#include <stdint.h>                                                                                                                                                       
int main(void)                                                                                                                                                            
{                                                                                                                                                                         
  return 0;                                                                                                                                                               
}                                                                                                                                                                         
using the following command line:                                                                                                                                         
gcc -m32 -O -Wall -o /build/VirtualBox-6.0.12/.tmp_out /build/VirtualBox-6.0.12/.tmp_src.c                                                                                
In file included from /nix/store/zg3y0jq36pd8xf8rd8rhj8bcpinyrs3s-glibc-2.27-dev/include/features.h:452,                                                                  
                 from /nix/store/zg3y0jq36pd8xf8rd8rhj8bcpinyrs3s-glibc-2.27-dev/include/bits/libc-header-start.h:33,                                                     
                 from /nix/store/zg3y0jq36pd8xf8rd8rhj8bcpinyrs3s-glibc-2.27-dev/include/stdint.h:26,                                                                     
                 from /nix/store/yvzq185jfr4vj74a9h4ap4dj4nnqbx7l-gcc-8.3.0/lib/gcc/x86_64-unknown-linux-gnu/8.3.0/include/stdint.h:9,                                    
                 from /build/VirtualBox-6.0.12/.tmp_src.c:1:                                                                                                              
/nix/store/zg3y0jq36pd8xf8rd8rhj8bcpinyrs3s-glibc-2.27-dev/include/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory                               
# include <gnu/stubs-32.h>                                                                                                                                               
           ^~~~~~~~~~~~~~~~                                                                                                                                               
compilation terminated.

Any clues? I'm digging it but I don't know where to check now.

[Ninlives]

The problem is that, virtualbox use stdenv_32bit to compile, but in the ccache module the package is overridden by the default ccacheStdenv. The issue can be fixed by manually change the stdenv:

  nixpkgs.overlays = [
    (self: super: {
      virtualbox = super.virtualbox.override {
        stdenv = with self; overrideCC stdenv_32bit (ccacheWrapper.override { cc = stdenv_32bit.cc; });
      };
    })
  ];

But I think this is too hacky because I need to manually inspect the actual stdenv used by each package. I tried to patch the programs.ccache module like this:

    # target configuration
    (mkIf (cfg.packageNames != []) {
      nixpkgs.overlays = [
        (self: super: genAttrs cfg.packageNames (pn: super.${pn}.override {
          stdenv = builtins.trace "with ccache: ${pn}" (self.overrideCC super.${pn}.stdenv self.ccacheWrapper.overrde { cc = super.${pn}.stdenv.cc; } );
        }))
      ...
      ];
    })

but this will not work because super.${pn}.stdenv is not overrideable. Do you have any suggestions?

[stale[bot]]

Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on irc.freenode.net.

[samuelgrf]

Still important to me.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[RossComputerGuy]

I've managed to run into this same issue with cross compiling for armv6l and armv7l but with nodejs. The missing header error is from v8 being used in nodejs.

[Bl4Cc4t]

It looks like #220204 is the PR that would (partially) fix this issue.

With the help of https://github.com/nodejs/node/issues/44357#issuecomment-1230010957, I was able to crosscompile node_18 for armv6l using the following workaround:

let
    inherit (pkgs) buildPackages;
in rec {
    nixpkgs.overlays = [
        (final: prev: {
            # https://discourse.nixos.org/t/nixos-on-raspberry-pi-zero-w/38018
            llvmPackages = prev.llvmPackages_14;
            cmake = prev.cmake.overrideAttrs (old: {
                env.NIX_CFLAGS_COMPILE = "-latomic";
            });
            # https://github.com/NixOS/nixpkgs/pull/220204
            nodejs_18 = buildPackages.pkgsi686Linux.callPackage "${toString pkgs.path}/pkgs/development/web/nodejs/v18.nix" {};
        })
        (final: prev: {
            # https://github.com/nodejs/node/issues/44357#issuecomment-1230010957
            nodejs_18 = prev.nodejs_18.overrideAttrs (old: {
                env.NIX_CFLAGS_COMPILE = "-march=armv6k -mfpu=vfpv2";
                env.NIX_CXXFLAGS_COMPILE = "-march=armv6k -mfpu=vfpv2";
            });
        })
    ];

This here might also work, although I haven't tested that: https://github.com/NixOS/nixpkgs/issues/163376#issuecomment-1063570579