pkgsBuildBuild.pkgsi686Linux in pkgsCross is build for cross platform

Cynerd commented 1 year ago

Steps To Reproduce

Steps to reproduce the behavior:

build nix build nixpkgs#legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.gcc_multi

Build log

From config.log for 32bit gcc.

/nix/store/49zl26cmymymzzh1xqahd6cp1g10l141-glibc-2.35-224-dev/include/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
    7 | # include <gnu/stubs-32.h>
      |           ^~~~~~~~~~~~~~~~

Additional context

The error is caused by missing file in glibc-multi. That is created by merging glibc and pkgsi686Linux.glibc. It turns out that issue is in pkgsi686Linux.glibc which for some reason uses some options from host platform. This is output of nix show-derivation nixpkgs#legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.glibc

{
  "/nix/store/27skn0aayk7xivdrsqz7sh4zr5v9c61i-glibc-armv7l-unknown-linux-gnueabihf-2.35-224.drv": {
    "args": [
      "-e",
      "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
    ],
    "builder": "/nix/store/pn5sy7dwndzpxbih7xjl7g7inkdphxzq-bash-5.2-p15/bin/bash",
    "env": {
      "BASH_SHELL": "/bin/sh",
      "NIX_CFLAGS_COMPILE": "-Wno-error=missing-attributes",
      "NIX_HARDENING_ENABLE": "pic strictoverflow format relro bindnow",
      "NIX_NO_SELF_RPATH": "1",
      "__structuredAttrs": "",
      "bin": "/nix/store/6xbpkixalf0s429qd7d5syv0gghwxxlh-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-bin",
      "buildInputs": "/nix/store/kmw32416a7ya2pk5kbrfyyk9b6szpsxk-linux-headers-6.1",
      "builder": "/nix/store/pn5sy7dwndzpxbih7xjl7g7inkdphxzq-bash-5.2-p15/bin/bash",
      "cmakeFlags": "-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=armv7l -DCMAKE_HOST_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_PROCESSOR=i686",
      "configureFlags": "-C --enable-add-ons --sysconfdir=/etc --enable-stack-protector=strong --enable-bind-now --with-headers=/nix/store/kmw32416a7ya2pk5kbrfyyk9b6szpsxk-linux-headers-6.1/include --disable-profile --enable-kernel=3.10.0 --without-fp --with-__thread --disable-crypt --build=i686-unknown-linux-gnu --host=armv7l-unknown-linux-gnueabihf",
      "debug": "/nix/store/9mwwaw0bi4pd6zyvr8ms73c677x1a159-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-debug",
      "depsBuildBuild": "/nix/store/2n1gi30hscji7s0j7lk76c2fh0zm657w-gcc-wrapper-11.3.0",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "dev": "/nix/store/ypnmzg2awsqnqd8bq60cjx42742f7inw-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-dev",
      "doCheck": "",
      "doInstallCheck": "",
      "enableParallelBuilding": "1",
      "enableParallelChecking": "1",
      "hardeningDisable": "fortify pie stackprotector",
      "installFlags": "sysconfdir=$(out)/etc",
      "is64bit": "",
      "linuxHeaders": "/nix/store/kmw32416a7ya2pk5kbrfyyk9b6szpsxk-linux-headers-6.1",
      "makeFlags": "OBJCOPY=armv7l-unknown-linux-gnueabihf-objcopy",
      "mesonFlags": "--cross-file=/nix/store/liv5rkz44z1cy77w4a92fg2ibnf852ng-cross-file.conf",
      "name": "glibc-armv7l-unknown-linux-gnueabihf-2.35-224",
      "nativeBuildInputs": "/nix/store/r10al970d8z0mkrwxww4lw0jkq9rhhpd-bison-3.8.2 /nix/store/0k4afl42b6hd0m8mjpl5mna5q5siraij-python3-minimal-3.10.9 /nix/store/g1669dakr0yxrx2w6c7czqih8cq3351r-separate-debug-info.sh",
      "out": "/nix/store/93b480qcg6kd29s1j6slfh03krqiq84a-glibc-armv7l-unknown-linux-gnueabihf-2.35-224",
      "outputs": "out bin dev static debug",
      "patches": "/nix/store/sxv7vwsanif38wri00hr2cg91ffagcjf-2.35-master.patch.gz /nix/store/8haph3ng4mgsqr6p4024vj8k6kg3mqc4-nix-locale-archive.patch /nix/store/b1w7zbvm39ff1i52iyjggyvw2rdxz104-dont-use-system-ld-so-cache.patch /nix/store/mnglr8rr7nl444h7p50ysyq8qd0fm1lm-dont-use-system-ld-so-preload.patch /nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch /nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch /nix/store/001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch /nix/store/l9yiy982yxkijaazijr741w36a97llcs-0001-Revert-Remove-all-usage-of-BASH-or-BASH-in-installed.patch",
      "pname": "glibc",
      "postConfigure": "# Hack: get rid of the `-static' flag set by the bootstrap stdenv.\n# This has to be done *after* `configure' because it builds some\n# test binaries.\nexport NIX_CFLAGS_LINK=\nexport NIX_LDFLAGS_BEFORE=\n\nexport NIX_DONT_SET_RPATH=1\nunset CFLAGS\n\n# Apparently --bindir is not respected.\nmakeFlagsArray+=(\"bindir=$bin/bin\" \"sbindir=$bin/sbin\" \"rootsbindir=$bin/sbin\")\n",
      "postInstall": "# This is based on http://www.linuxfromscratch.org/lfs/view/development/chapter06/glibc.html\n# Instead of using their patch to build a build-native localedef,\n# we simply use the one from buildPackages\npushd ../glibc-2*/localedata\nexport I18NPATH=$PWD GCONV_PATH=$PWD/../iconvdata\nmkdir -p $NIX_BUILD_TOP//nix/store/hnnmq5vp5y32imk3c4i2pdvsqxvw4chz-glibc-2.35-224/lib/locale\n/nix/store/sl93gr469wkjrngs46490qm9y2ma0din-glibc-2.35-224-bin/bin/localedef \\\n  --alias-file=../intl/locale.alias \\\n  -i locales/C \\\n  -f charmaps/UTF-8 \\\n  --prefix $NIX_BUILD_TOP \\\n  --little-endian \\\n  C.UTF-8\ncp -r $NIX_BUILD_TOP//nix/store/hnnmq5vp5y32imk3c4i2pdvsqxvw4chz-glibc-2.35-224/lib/locale $out/lib\npopd\n\ntest -f $out/etc/ld.so.cache && rm $out/etc/ld.so.cache\n\nif test -n \"$linuxHeaders\"; then\n    # Include the Linux kernel headers in Glibc, except the `scsi'\n    # subdirectory, which Glibc provides itself.\n    (cd $dev/include && \\\n     ln -sv $(ls -d $linuxHeaders/include/* | grep -v scsi\\$) .)\nfi\n\n# Fix for NIXOS-54 (ldd not working on x86_64).  Make a symlink\n# \"lib64\" to \"lib\".\nif test -n \"$is64bit\"; then\n    ln -s lib $out/lib64\nfi\n\n# Get rid of more unnecessary stuff.\nrm -rf $out/var $bin/bin/sln\n\n# Backwards-compatibility to fix e.g.\n# \"configure: error: Pthreads are required to build libgomp\" during `gcc`-build\n# because it's not actually needed anymore to link against `pthreads` since\n# it's now part of `libc.so.6` itself, but the gcc build breaks if\n# this doesn't work.\nln -sf $out/lib/libpthread.so.0 $out/lib/libpthread.so\nln -sf $out/lib/librt.so.1 $out/lib/librt.so\nln -sf $out/lib/libdl.so.2 $out/lib/libdl.so\nln -sf $out/lib/libutil.so.1 $out/lib/libutil.so\ntouch $out/lib/libpthread.a\n\n# Put libraries for static linking in a separate output.  Note\n# that libc_nonshared.a and libpthread_nonshared.a are required\n# for dynamically-linked applications.\nmkdir -p $static/lib\nmv $out/lib/*.a $static/lib\nmv $static/lib/lib*_nonshared.a $out/lib\n# Some of *.a files are linker scripts where moving broke the paths.\nsed \"/^GROUP/s|$out/lib/lib|$static/lib/lib|g\" \\\n  -i \"$static\"/lib/*.a\n\n# Work around a Nix bug: hard links across outputs cause a build failure.\ncp $bin/bin/getconf $bin/bin/getconf_\nmv $bin/bin/getconf_ $bin/bin/getconf\n",
      "postPatch": "# Needed for glibc to build with the gnumake 3.82\n# http://comments.gmane.org/gmane.linux.lfs.support/31227\nsed -i 's/ot \\$/ot:\\n\\ttouch $@\\n$/' manual/Makefile\n\n# nscd needs libgcc, and we don't want it dynamically linked\n# because we don't want it to depend on bootstrap-tools libs.\necho \"LDFLAGS-nscd += -static-libgcc\" >> nscd/Makefile\n\n# Ensure that `__nss_files_fopen` can still be wrapped by `libredirect`.\nsed -i -e '/libc_hidden_def (__nss_files_fopen)/d' nss/nss_files_fopen.c\nsed -i -e '/libc_hidden_proto (__nss_files_fopen)/d' include/nss_files.h\n",
      "preBuild": "",
      "preConfigure": "export PWD_P=$(type -tP pwd)\nfor i in configure io/ftwtest-sh; do\n    # Can't use substituteInPlace here because replace hasn't been\n    # built yet in the bootstrap.\n    sed -i \"$i\" -e \"s^/bin/pwd^$PWD_P^g\"\ndone\n\nmkdir ../build\ncd ../build\n\nconfigureScript=\"`pwd`/../$sourceRoot/configure\"\n\nmakeFlags=\"$makeFlags BUILD_LDFLAGS=-Wl,-rpath,/nix/store/1jb8f8yw7yvchsw3gpsi5bwkfa37anbv-glibc-armv7l-unknown-linux-gnueabihf-2.35-224/lib OBJDUMP=/nix/store/1cq6nm161b4jw0kqr0lm77qiqprb9znw-armv7l-unknown-linux-gnueabihf-binutils-2.39/bin/objdump\"\n\n\nsed -i s/-lgcc_eh//g \"../$sourceRoot/Makeconfig\"\n\ncat > config.cache << \"EOF\"\nlibc_cv_forced_unwind=yes\nlibc_cv_c_cleanup=yes\nlibc_cv_gnu89_inline=yes\nEOF\n\n# ./configure has logic like\n#\n#     AR=`$CC -print-prog-name=ar`\n#\n# This searches various directories in the gcc and its wrapper. In nixpkgs,\n# this returns the bare string \"ar\", which is build ar. This can result as\n# a build failure with the following message:\n#\n#     libc_pic.a: error adding symbols: archive has no index; run ranlib to add one\n#\n# (Observed cross compiling from aarch64-linux -> armv7l-linux).\n#\n# Nixpkgs passes a correct value for AR and friends, so to use the correct\n# set of tools, we only need to delete this special handling.\nsed -i \\\n  -e '/^AR=/d' \\\n  -e '/^AS=/d' \\\n  -e '/^LD=/d' \\\n  -e '/^OBJCOPY=/d' \\\n  -e '/^OBJDUMP=/d' \\\n  $configureScript\n",
      "preInstall": "",
      "profilingLibraries": "",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "separateDebugInfo": "1",
      "src": "/nix/store/hsji8dkz2azgn3ylyzx9898qyrf46axv-glibc-2.35.tar.xz",
      "static": "/nix/store/wqmz7xq2pw08rhhayyvg271wqpfwcna3-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-static",
      "stdenv": "/nix/store/9772prci5wkkqkjm4kj8n4m915ygkc0m-stdenv-linux",
      "strictDeps": "1",
      "system": "i686-linux",
      "version": "2.35-224",
      "withLibcrypt": ""
    },
    "inputDrvs": {
      "/nix/store/2gfgl2n7wxzr3w3z6qshgd8qyx45a8lj-glibc-2.35-224.drv": [
        "bin",
        "out"
      ],
      "/nix/store/5kj6nifh4080ym1kimcvb8ax1mzvkk5r-armv7l-unknown-linux-gnueabihf-binutils-2.39.drv": [
        "out"
      ],
      "/nix/store/70sabiv6cnaj943jszsib3ki9x3q6sg7-bison-3.8.2.drv": [
        "out"
      ],
      "/nix/store/9k3zwhzkzj7hbfs5a62imrpg58jhmdhv-bash-5.2-p15.drv": [
        "out"
      ],
      "/nix/store/bd67qkpfzdx6isqnrb46zvr4f8akwbly-stdenv-linux.drv": [
        "out"
      ],
      "/nix/store/pwb6lncqmal5nqwzwi73dv9grsn1hg8c-gcc-wrapper-11.3.0.drv": [
        "out"
      ],
      "/nix/store/qm4vj29kcp188mbbabk6ndpjb15711sz-linux-headers-6.1.drv": [
        "out"
      ],
      "/nix/store/varvf7907q8xm1p1ccil359l2zbcrnq3-glibc-armv7l-unknown-linux-gnueabihf-2.35-224.drv": [
        "out"
      ],
      "/nix/store/wpsjbgw43jyji44vralbnmylfqrrq4x7-python3-minimal-3.10.9.drv": [
        "out"
      ],
      "/nix/store/zxa99fi6bvk41r17xkqa4d6payipyxhx-glibc-2.35.tar.xz.drv": [
        "out"
      ]
    },
    "inputSrcs": [
      "/nix/store/001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch",
      "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh",
      "/nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch",
      "/nix/store/8haph3ng4mgsqr6p4024vj8k6kg3mqc4-nix-locale-archive.patch",
      "/nix/store/b1w7zbvm39ff1i52iyjggyvw2rdxz104-dont-use-system-ld-so-cache.patch",
      "/nix/store/g1669dakr0yxrx2w6c7czqih8cq3351r-separate-debug-info.sh",
      "/nix/store/l9yiy982yxkijaazijr741w36a97llcs-0001-Revert-Remove-all-usage-of-BASH-or-BASH-in-installed.patch",
      "/nix/store/liv5rkz44z1cy77w4a92fg2ibnf852ng-cross-file.conf",
      "/nix/store/mnglr8rr7nl444h7p50ysyq8qd0fm1lm-dont-use-system-ld-so-preload.patch",
      "/nix/store/sxv7vwsanif38wri00hr2cg91ffagcjf-2.35-master.patch.gz",
      "/nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch"
    ],
    "outputs": {
      "bin": {
        "path": "/nix/store/6xbpkixalf0s429qd7d5syv0gghwxxlh-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-bin"
      },
      "debug": {
        "path": "/nix/store/9mwwaw0bi4pd6zyvr8ms73c677x1a159-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-debug"
      },
      "dev": {
        "path": "/nix/store/ypnmzg2awsqnqd8bq60cjx42742f7inw-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-dev"
      },
      "out": {
        "path": "/nix/store/93b480qcg6kd29s1j6slfh03krqiq84a-glibc-armv7l-unknown-linux-gnueabihf-2.35-224"
      },
      "static": {
        "path": "/nix/store/wqmz7xq2pw08rhhayyvg271wqpfwcna3-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-static"
      }
    },
    "system": "i686-linux"
  }
}

The system is according to it i686-linux but it doesn't look like it. I was unable to find the real reason for this. It might be something in stage.nix. I also tried other packages in pkgsi686Linux; they all seem to contain paths to armv7l instead of i686.

Notify maintainers

@johnae @matthewbauer @danbst

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.3, NixOS, 23.05 (Stoat), 23.05.20230122.6c582bd`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.12.0`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Cynerd commented 1 year ago

Eh...

$ nix eval nixpkgs#legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.stdenv.hostPlatform.system
"x86_64-linux"
$ nix eval nixpkgs#legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.stdenv.hostPlatform.system
"armv7l-linux"
$ nix eval nixpkgs#legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsMusl.stdenv.hostPlatform.system
"armv7l-linux"

Cynerd commented 1 year ago

It turns out that having armv7l there is not actually something completely invalid, but rather just strange and non-intuitive. I thought that the following chain is true but no (I am ignoring target there for simplicity)..:

legacyPackages.x86_64-linux
-> build: x86_64-linux, host: x86_64-linux
legacyPackages.x86_64-linux.pkgsi686Linux
-> build: i686-linux, host: i686-linux
legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform
-> build: x86_64-linux, host: i686-linux
legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild
-> build: x86_64-linux, host: x86_64-linux
legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux
-> build: i686-linux, host: i686-linux

This would make sense to me but no... It turns out it is instead:

legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux
-> build: i686-linux, host: armv7l-linux
legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.buildPackages
-> build: i686-linux, host: i686-linux

What makes it confusing is that this is the same package set, but it doesn't feel like it should be:

legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux
legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.pkgsBuildHost.pkgsi686Linux

The core issue is then probably that glibc_multi should not be possible to be built when the host is not i686 as it results in gcc_multi build failure.

It might also be desirable to resolve the confusion I fell into here.

Cynerd commented 1 year ago

For completeness, my original entry point to this was stdenv_32bit which uses gcc_multi. The typical approach in the packaging when software needs a 32-bit host compiler, I would expect, is to use stdenv_32bit. The same applies when I need it to compile some build tool during cross-compilation. I would simply use buildPackages.stdenv_32bit but that doesn't work, and neither does buildPackages.buildPackages.stdenv_32bit. Those environments consider a local system to be i686 but a host system to be the cross-compilation target.

Cynerd commented 1 year ago

I have more discoveries because nobody else seems to care about this... It turns out that behaviour, where pkgsi686Linux has host and target set to the target of the original package set, is a limitation of package sets instantiation. There is only possible to pass localSystem and crossSystem. The pkgsi686Linux set is generated by reevaluating the package set with these variables set to something other than the original set. The issue is that when the original set has the same host and build platform but a different target, it can only set the build and host platform. There is just no possibility of getting set where the target is different from the host directly. The only option is to use pkgsi686Linux.buildPackages to get the correct combination of platforms.

wucke13 commented 1 year ago

@Cynerd I ran into this on multiple cross compilation attempts. What do you think would be a way forward?

Cynerd commented 1 year ago

@wucke13 I think the core issue is that we can't set a target platform. We can only set localSystem and crossSystem. The neighbor package set like the ones discussed here sometimes requires a custom target platform that can't be directly archived through the standard nixpkgs set evaluation. It might be possible to evaluate it with the correct crossPlatform but then take buildPackages of that set instead. Honestly, I am not experienced enough to fix this right now, that is also why I am writing this here and not fixing it.

wolfgangwalther commented 5 months ago

I have more discoveries because nobody else seems to care about this... It turns out that behaviour, where pkgsi686Linux has host and target set to the target of the original package set, is a limitation of package sets instantiation. There is only possible to pass localSystem and crossSystem. The pkgsi686Linux set is generated by reevaluating the package set with these variables set to something other than the original set. The issue is that when the original set has the same host and build platform but a different target, it can only set the build and host platform. There is just no possibility of getting set where the target is different from the host directly. The only option is to use pkgsi686Linux.buildPackages to get the correct combination of platforms.

I didn't fully understand this whole issue - but I fixed the fact that pkgsi686Linux changes the localSystem instead of the crossSystem when "you are already cross" in #303849, specifically in 1028e73e8428f659f2c462c950355cd221b82107.

Can you try whatever you did here ;) on my branch again and see whether that helps?

wolfgangwalther commented 5 months ago

On my branch mentioned above:

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.stdenv.hostPlatform.system
"x86_64-linux"

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.stdenv.hostPlatform.system
"i686-linux"

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsMusl.stdenv.hostPlatform.system
"x86_64-linux"

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.stdenv.buildPlatform.system
"x86_64-linux"

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.buildPackages.stdenv.buildPlatform.system
"x86_64-linux"

nix-repl> pkgs.pkgsCross.armv7l-hf-multiplatform.pkgsBuildBuild.pkgsi686Linux.buildPackages.stdenv.hostPlatform.system
"x86_64-linux"

This looks a lot more what you'd expect, I guess?

NixOS / nixpkgs