ccache doesn't work for linux kernel

[WesleyAC]

Describe the bug

I am trying to use ccache to compile the linux kernel with the following config:

  programs.ccache.enable = true;
  programs.ccache.packageNames = [ "linuxPackages_latest" ];
  nix.sandboxPaths = [ config.programs.ccache.cacheDir ];
  boot.kernelPackages = pkgs.linuxPackages_latest;

But I get the following error:

error: attribute 'override' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/programs/ccache.nix:55:54

Steps To Reproduce

1. Use the four lines above in /etc/nixos/configuration.nix
2. Run nixos-rebuild switch

Expected behavior

Kernel is built using ccache.

Additional context

62121 may be related?

Notify maintainers

Not sure who this belongs to :(

Metadata

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.81, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11.334247.573095944e7"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[aviallon]

Easy workaround:

nix.sandboxPaths = [ (toString config.programs.ccache.cacheDir) ];

[WesleyAC]

@aviallon do you have more info on that? I still get the error error: attribute 'override' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/programs/ccache.nix:55:54 when I try it.

[milahu]

context of ccache.nix:55

    # target configuration
    (mkIf (cfg.packageNames != []) {
      nixpkgs.overlays = [
        (self: super: genAttrs cfg.packageNames (pn: super.${pn}.override { stdenv = builtins.trace "with ccache: ${pn}" self.ccacheStdenv; }))

so the nixpkgs.linuxPackages_latest.override attribute is missing

but nixpkgs.linuxPackages_latest.kernel.override exists:

$ nix repl '<nixpkgs>'
nix-repl> pkgs.linuxPackages_latest.kernel.override

probably this will not work, cos not implemented

programs.ccache.packageNames = [ "linuxPackages_latest.kernel" ];

[WesleyAC]

I tried setting programs.ccache.packageNames = [ "linuxPackages_latest.kernel" ]; — that compiles, but unfortunately does not seem to actually cache the build artifacts :(

[milahu]

{
      nixpkgs.overlays = [
        (self: super: {
          linuxPackages_latest.kernel = super.linuxPackages_latest.kernel.override {
            stdenv = self.ccacheStdenv;
          };
        })
      ];

?

you probably also need

nix.sandboxPaths = [ (toString config.programs.ccache.cacheDir) ];

then you can

sudo watch ccache --dir /nix/var/cache/ccache --show-stats

(not sure about the cache path)

[WesleyAC]

@milahu Thanks so much for the help! I just tried that, and unfortunately it doesn't seem to work — I get:

error: attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:31

--show-trace gives me:

error: while evaluating the attribute 'config.system.build.toplevel' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:302:5:
while evaluating 'foldr' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:52:20, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:133:12:
while evaluating 'fold'' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:55:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:59:8:
while evaluating the attribute 'assertions' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:171:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:304:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:585:9:
while evaluating the option `assertions':
while evaluating the attribute 'mergedValue' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:617:5:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:611:9:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:710:7:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:597:28, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:597:17:
while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix':
while evaluating 'dischargeProperties' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:669:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:598:137:
while evaluating 'dischargeProperties' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:669:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:675:11:
while evaluating the attribute 'content' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:753:14:
while evaluating the attribute 'boot.kernelPackages.kernel' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:171:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:304:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:585:9:
while evaluating the option `boot.kernelPackages':
while evaluating 'apply' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:578:35:
attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:31

Any idea what's up?

[patryk4815]

workaround:

  programs.ccache.enable = true;
  programs.ccache.cacheDir = "/opt/ccache";
  programs.ccache.packageNames = [ "linuxPackages_5_19.kernel" ];
  nix.settings.extra-sandbox-paths = [ "/opt/ccache" ];

  nixpkgs.overlays = [
    (self: prev: {
      kernel_cache = (prev.linuxPackages_5_19.kernel.override {
        stdenv = self.ccacheStdenv;
        buildPackages = prev.buildPackages // {
          stdenv = self.ccacheStdenv;
        };
      }).overrideDerivation (attrs: {
        preConfigure = ''
          export NIX_CFLAGS_COMPILE="$(echo "$NIX_CFLAGS_COMPILE" | sed -e "s/-frandom-seed=[^-]*//")"
        '';
      });
    })
  ];

  boot = {
    kernelPackages =  pkgs.linuxPackagesFor pkgs.kernel_cache;
    kernelPatches = [
      {
        name = "fix-something";
        patch = /opt/linux_5_19_9/linux.patch;
        extraConfig = ''
        '';
      }
    ];
  };

programs.ccache.packageNames don't override buildPackages.stdenv ;( And next problem is frandom-seed soo we need trim it - https://github.com/NixOS/nixpkgs/issues/109033 https://github.com/ccache/ccache/issues/1002

[aviallon]

@patryk4815 Fortunately, ccache's master has just been updated to work around the -frandom-seed problems. https://github.com/ccache/ccache/pull/1168

[jordanisaacs]

Curious if someone has an example of it working. It seems to be breaking because the ccache wrapper isn't outputting the right version information? Not entirely sure how the ccache wrapper works and do not want to go down the rabbit hole of figuring it out if it isn't necessary. The exact same build works with normal stdenv. This is the kernel build error:

make[1]: Entering directory '/build/linux-6.1/build'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
/nix/store/4g4vvpli8w1c7j4f6r6px1w9k1m46aj6-ccache-links-wrapper-/bin/cc: unknown assembler invoked
scripts/Kconfig.include:50: Sorry, this assembler is not supported.
make[2]: *** [../scripts/kconfig/Makefile:77: oldconfig] Error 1
make[1]: *** [/build/linux-6.1/Makefile:697: oldconfig] Error 2
make[1]: Leaving directory '/build/linux-6.1/build'
make: *** [Makefile:231: __sub-make] Error 2

This is the kernel script that causes the error and what is being invoked.

Interestingly calling the script manually with ./as-version.sh $(ccache_cc) outputs GNU 23900 with no errors. Seems to be something with the CLANG_FLAGS then?

This is an excerpt of my kernel derivation (full source is here):

    ((callPackage "${nixpkgs}/pkgs/os-specific/linux/kernel/manual-config.nix" {})
      {
        inherit src modDirVersion version kernelPatches configfile;
        inherit lib;

        stdenv = ccacheStdenv;
        ....
      })
    .overrideAttrs (old: {
      preConfigure = lib.optionalString ccache.enable ''
        export CCACHE_DIR=${ccache.dir}
        export CCACHE_UMASK=007
      '';
      ...
    })

[minhuw]

I encountered a similar issue as @jordanisaacs, and after some investigation, I realized that I had forgotten to update the extra-sandbox-paths setting in my nix.conf file. Once I added extra-sandbox-paths = /nix/var/cache/ccache to my nix.conf, the problem was resolved.

Initially, I wasn't receiving any error messages from ccache because its output was being redirected to /dev/null in the kernel compilation script. However, after removing these redirects with a simple patch, I was able to view the raw ccache output and identify the underlying problem.

I hope this information helps others who might encounter a similar issue.

[alberand]

I've had the same issues as @jordanisaacs, and with a set -ex in ./scripts/as-version.sh I found out the following error ccache: permission denied. For me it was wrong path in export CCACHE_DIR=....

Now kernel complies. Moreover, I see ccache stats changing with watch nix-ccache --show-stats. However, even simple commit to the kernel still causes a whole kernel recompile. This is also evident from ccache stats, as the only field growing is "Misses". Although, the "Hits" also seems to change but not a lot.

Rebuilding derivation which failed (kernel which doesn't compile) is working, though. I see a lot of Hits and not that many misses.

The setup is the same as describe by @jordanisaacs