An Emacs package cannot have more than 17 dependencies

mrkkrp commented 4 years ago

Describe the bug

It appears that when a Emacs package starts to have more than 17 dependent packages something related to implicit hooks goes wrong. Consider the following:

let
  pkgs = import <nixpkgs> {};
  mkConfig = epkgs: epkgs.trivialBuild {
    pname = "my-test";
    src = pkgs.lib.sourceByRegex ./. [
      "^.*\.el$"
    ];
    unpackCmd = ''
      echo "This is it: $curSrc"
      echo "Basename: $(basename -- "$curSrc")"
      echo "$(stripHash "$curSrc")"
      _defaultUnpack "$curSrc"
    '';
    packageRequires = with epkgs; [
      ace-link
      ace-popup-menu
      ace-window
      aggressive-indent
      auctex
      avy
      avy-menu
      char-menu
      counsel
      cyphejor
      direnv
      f
      yaml-mode
      zenburn-theme
      ztree
      zygospore
      zzz-to-char
    ];
  };
in mkConfig pkgs.emacs26Packages

In this form nix-build fails:

$ nix-build
these derivations will be built:
  /nix/store/4sviv2ilrkj4ww1c7qjdng2rk4xhirj2-emacs-my-test.drv
building '/nix/store/4sviv2ilrkj4ww1c7qjdng2rk4xhirj2-emacs-my-test.drv'...
unpacking sources
unpacking source archive /nix/store/zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
Current source: /nix/store/zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 82: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/basename: Argument list too long
Basename on curSrc:
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 805: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/basename: Argument list too long
Strip hash:
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 805: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/basename: Argument list too long
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 826: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/cp: Argument list too long
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 805: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/basename: Argument list too long
/nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup: line 826: /nix/store/3qgg8xzdyfd7nql0qlkb1xpnrfz4k1s2-coreutils-8.31/bin/cp: Argument list too long
do not know how to unpack source archive /nix/store/zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
builder for '/nix/store/4sviv2ilrkj4ww1c7qjdng2rk4xhirj2-emacs-my-test.drv' failed with exit code 1
error: build of '/nix/store/4sviv2ilrkj4ww1c7qjdng2rk4xhirj2-emacs-my-test.drv' failed

Looking at the /nix/store/ikywdrvfib46c2rwiphig16bppv8csv4-stdenv-linux/setup file, I found that it first sees that src points to a directory. So it tries to strip the hash from the directory name and copy it. basename invocation inside of stripHash fails. When I added an explicit invocation of basename, it reported line 82, this is inside of _callImplicitHook function definition. I have no idea what is going on here.

Interestingly, if I delete just one dependent package from the list, nix-build starts to pass:

$ nix-build
these derivations will be built:
  /nix/store/b3r3hbjrqczzmgrgdn5adfip5xqfzjp0-emacs-my-test.drv
building '/nix/store/b3r3hbjrqczzmgrgdn5adfip5xqfzjp0-emacs-my-test.drv'...
unpacking sources
unpacking source archive /nix/store/zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
Current source: /nix/store/zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
Basename on curSrc: zsz6k61jzdq8ykzsrknz04jnjm5wa3w5-emacs
Strip hash: emacs
source root is emacs
patching sources
configuring
no configure script, doing nothing
building
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/n2bl2cckismbwfjvgff8fbkws6xky0sv-emacs-my-test
strip is /nix/store/7wy2mifb0igrp9hzf9dzwddadbs7ai7d-binutils-2.31.1/bin/strip
patching script interpreter paths in /nix/store/n2bl2cckismbwfjvgff8fbkws6xky0sv-emacs-my-test
checking for references to /build/ in /nix/store/n2bl2cckismbwfjvgff8fbkws6xky0sv-emacs-my-test...
/nix/store/n2bl2cckismbwfjvgff8fbkws6xky0sv-emacs-my-test

It would be nice if we could fix this bug. I understand that this many dependencies is unusual for Emacs packages, but I'm experimenting with defining my configuration as a package which depends on packages that I use, so finally I can use emacsWithPackages and get Emacs with my configuration included and pre-built.

To Reproduce

You can use the snippet from above. I used b2935fbeceaea0b64df4401545d7c8ea29102120 of nixpkgs.

Metadata

system: "x86_64-linux"
host os: Linux 5.4.25, NixOS, 20.03.git.b2935fbecea (Markhor)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.4pre7250_94c93437
channels(mark): "nixos-18.09.2337.5d972dd2c94"
channels(root): "nixos-19.03.173651.ca6b6595392"
nixpkgs: /home/mark/nixpkgs

grahamc commented 4 years ago

I have more than 17 dependencies: https://github.com/grahamc/nixos-config/blob/master/packages/emacs/default.nix#L37

mrkkrp commented 4 years ago

This is different. You use emacsWithPackages, whereas I try to define a Emacs Lisp package itself with its own dependencies. It is as if magit had 17+ dependencies and failed to build.

grahamc commented 4 years ago

The problem appears to be something wrong in calculating the EMACSLOADPATH, where each plugin is included many times:

    unpackPhase = ''
      env | grep EMACSLOADPATH | tr : '\n' | sort | uniq -c | sort -nr
    '';

with 16 dependencies:

     63 /nix/store/z6vpclw5qy8mhshln5xqvx4lpa1a49lp-emacs-swiper-20200105.1958/share/emacs/site-lisp
     63 /nix/store/x044jrmzv37ymnm85mjg66gi62c5nfhf-emacs-avy-20191106.1234/share/emacs/site-lisp
     63 /nix/store/wjvyy9n1jhipk4dx59r43vbnhn39ixcd-emacs-ace-popup-menu-20190713.1337/share/emacs/site-lisp
     63 /nix/store/s369q2b6r2g2p44y84sdap3cfj8p939l-emacs-f-20191110.1357/share/emacs/site-lisp
     63 /nix/store/qm24rw44f5j7ysdj4px8h86h4yvc5sn9-emacs-ace-window-20191022.1203/share/emacs/site-lisp
     63 /nix/store/q8i9kxr0hcczwhccxpg3b66yrx698x5w-emacs-auctex-12.2.0/share/emacs/site-lisp
     63 /nix/store/q4wphgsh4cm2f7ixgrvjffhjbcv3xncq-emacs-direnv-20191016.1907/share/emacs/site-lisp
     63 /nix/store/pll7a87qw27xyipxvjfq0798wi4ndyia-emacs-26.3/share/emacs/site-lisp
     63 /nix/store/lxd3a0r5fvjr69g0iva6jvk70b9jzsn3-emacs-yaml-mode-20191127.2314/share/emacs/site-lisp
     63 /nix/store/lkk1b8c4rn1fzcbn14d2lxfldp172pfn-emacs-counsel-20200105.1947/share/emacs/site-lisp
     63 /nix/store/iglhwyjc6jpra6kx1hwr6kb6a73x2526-emacs-ztree-20191108.2234/share/emacs/site-lisp
     63 /nix/store/d6zp6kanhi36q4a7wq6qnnzvk22s11wg-emacs-cyphejor-20190713.1339/share/emacs/site-lisp
     63 /nix/store/bxbzc1widlzjbw9y5s9m3158zc9mqdch-emacs-aggressive-indent-20190828.1828/share/emacs/site-lisp
     63 /nix/store/9ja2rbl0apmb1q5slb9az7y3yn735aqx-emacs-zygospore-20140703.852/share/emacs/site-lisp
     63 /nix/store/58ralp4c2pnqwjjrxi0vm6d2nhh5j586-emacs-dash-20191109.1327/share/emacs/site-lisp
     63 /nix/store/4y35kqnrkaxcqfsa0rrq8lx2mhx54hi5-emacs-zenburn-theme-20200105.1010/share/emacs/site-lisp
     63 /nix/store/4s7qznw8q62p9dn84wy2l8wld9bpipg5-emacs-s-20180406.808/share/emacs/site-lisp
     63 /nix/store/3mf3xcr8r2qr8p5gfyigrxvdb96rhgz4-emacs-char-menu-20190713.1343/share/emacs/site-lisp
     63 /nix/store/21b3f1h6jrvw68f2i5ycc04gd5cbwkzp-emacs-avy-menu-20190713.1348/share/emacs/site-lisp
     63 /nix/store/1z74f3rp27bpdhl9hrb56v7pndyhcsgq-emacs-ivy-20200105.1946/share/emacs/site-lisp
     62 /nix/store/q7d8zj1lyqbldvzkdfq92lz6yw5dgqdw-emacs-zzz-to-char-20190713.1344/share/emacs/site-lisp

with 8:

     30 /nix/store/z6vpclw5qy8mhshln5xqvx4lpa1a49lp-emacs-swiper-20200105.1958/share/emacs/site-lisp
     30 /nix/store/x044jrmzv37ymnm85mjg66gi62c5nfhf-emacs-avy-20191106.1234/share/emacs/site-lisp
     30 /nix/store/wjvyy9n1jhipk4dx59r43vbnhn39ixcd-emacs-ace-popup-menu-20190713.1337/share/emacs/site-lisp
     30 /nix/store/qm24rw44f5j7ysdj4px8h86h4yvc5sn9-emacs-ace-window-20191022.1203/share/emacs/site-lisp
     30 /nix/store/q8i9kxr0hcczwhccxpg3b66yrx698x5w-emacs-auctex-12.2.0/share/emacs/site-lisp
     30 /nix/store/pll7a87qw27xyipxvjfq0798wi4ndyia-emacs-26.3/share/emacs/site-lisp
     30 /nix/store/lkk1b8c4rn1fzcbn14d2lxfldp172pfn-emacs-counsel-20200105.1947/share/emacs/site-lisp
     30 /nix/store/bxbzc1widlzjbw9y5s9m3158zc9mqdch-emacs-aggressive-indent-20190828.1828/share/emacs/site-lisp
     30 /nix/store/3mf3xcr8r2qr8p5gfyigrxvdb96rhgz4-emacs-char-menu-20190713.1343/share/emacs/site-lisp
     30 /nix/store/21b3f1h6jrvw68f2i5ycc04gd5cbwkzp-emacs-avy-menu-20190713.1348/share/emacs/site-lisp
     29 /nix/store/1z74f3rp27bpdhl9hrb56v7pndyhcsgq-emacs-ivy-20200105.1946/share/emacs/site-lisp

with 4:

     18 /nix/store/x044jrmzv37ymnm85mjg66gi62c5nfhf-emacs-avy-20191106.1234/share/emacs/site-lisp
     18 /nix/store/wjvyy9n1jhipk4dx59r43vbnhn39ixcd-emacs-ace-popup-menu-20190713.1337/share/emacs/site-lisp
     18 /nix/store/qm24rw44f5j7ysdj4px8h86h4yvc5sn9-emacs-ace-window-20191022.1203/share/emacs/site-lisp
     18 /nix/store/pll7a87qw27xyipxvjfq0798wi4ndyia-emacs-26.3/share/emacs/site-lisp
     18 /nix/store/bxbzc1widlzjbw9y5s9m3158zc9mqdch-emacs-aggressive-indent-20190828.1828/share/emacs/site-lisp
     18 /nix/store/21b3f1h6jrvw68f2i5ycc04gd5cbwkzp-emacs-avy-menu-20190713.1348/share/emacs/site-lisp
     17 /nix/store/q8i9kxr0hcczwhccxpg3b66yrx698x5w-emacs-auctex-12.2.0/share/emacs/site-lisp

with 2:

     15 /nix/store/x044jrmzv37ymnm85mjg66gi62c5nfhf-emacs-avy-20191106.1234/share/emacs/site-lisp
     15 /nix/store/wjvyy9n1jhipk4dx59r43vbnhn39ixcd-emacs-ace-popup-menu-20190713.1337/share/emacs/site-lisp
     15 /nix/store/pll7a87qw27xyipxvjfq0798wi4ndyia-emacs-26.3/share/emacs/site-lisp
     15 /nix/store/21b3f1h6jrvw68f2i5ycc04gd5cbwkzp-emacs-avy-menu-20190713.1348/share/emacs/site-lisp
     14 /nix/store/qm24rw44f5j7ysdj4px8h86h4yvc5sn9-emacs-ace-window-20191022.1203/share/emacs/site-lisp

with 1:

      6 /nix/store/qm24rw44f5j7ysdj4px8h86h4yvc5sn9-emacs-ace-window-20191022.1203/share/emacs/site-lisp
      6 /nix/store/pll7a87qw27xyipxvjfq0798wi4ndyia-emacs-26.3/share/emacs/site-lisp
      5 /nix/store/x044jrmzv37ymnm85mjg66gi62c5nfhf-emacs-avy-20191106.1234/share/emacs/site-lisp

mrkkrp commented 4 years ago

Could it be that if two plugins depend on the same package it's simply included several times?

mrkkrp commented 4 years ago

Perhaps some filtering here

https://github.com/NixOS/nixpkgs/blob/353c6c948e0db50cddac8eca4bffe979267b285d/pkgs/build-support/emacs/setup-hook.sh#L1-L15

is necessary. Although this differs from my local nixpkgs checkout and looks like it may already be correct.

mrkkrp commented 4 years ago

Appears to be fixed by #82604.

NixOS / nixpkgs

An Emacs package cannot have more than 17 dependencies #83230