Nix Installer: "cannot set host name: Operation not permitted" in WSL2 + podman + QEMU while building alpine-based minimal nix image

philipp-baumann commented 3 weeks ago

Platform

[ ] Linux:
[ ] macOS
[x] WSL2

Additional information

While trying to build a minimal alpine container with multiuser and nix-daemon setup via podman, the official installer fails when I try to build this image on WSL2 Ubuntu via QEMU, but not on aarch64-darwin via vfkit, where it perfectly builds until the installer step (only nix-shell currently doesn't run because of permissions issue, but that is not relevant here).

I have this nix-shell env:

let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/30c9efeef01e2ad4880bff6a01a61dd99536b3c9.tar.gz") {};

  system_packages = builtins.attrValues {
    inherit (pkgs) 
      glibcLocales
      nix
      podman
  } ++ pkgs.lib.optionals pkgs.stdenv.isDarwin [ pkgs.vfkit ]
    ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ pkgs.qemu ];

in

pkgs.mkShell {
  LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";
  buildInputs = [ system_packages ];
}

podman machine set --user-mode-networking
podman machine init --user-mode-networking
podman build --network host --dns 8.8.8.8 -t alpine-nix .

Part of this build script.

build_container.sh

#! /usr/bin/env nix-shell
#! nix-shell default.nix -i bash 

# Check if both arguments are provided
if [ $# -ne 2 ]; then
    echo "Usage: $0 <image_name> <directory>"
    exit 1
fi

# Assign arguments to variables
image_name=$1
directory=$2

if [ -n "$WSL_INTEROP" ]; then
    echo "Running in WSL2"
    podman machine set --user-mode-networking
    podman machine init --user-mode-networking
    # https://github.com/containers/podman/issues/20921
fi

# Run podman build with the provided arguments
podman build --network host --dns 8.8.8.8 -t "${image_name}" "${directory}"

This is the image I'm trying to build:

FROM alpine:3.20.3

# Add DNS configuration
RUN echo "nameserver 8.8.8.8" > /etc/resolv.conf && \
    echo "nameserver 8.8.4.4" >> /etc/resolv.conf

RUN apk update && \
    apk add --no-cache \
        bash \
        curl \
        sudo \
        shadow \
        xz \
        openrc && \
        mkdir -p /run/openrc && \
        touch /run/openrc/softlevel

# Set environment variables
ENV NIX_FIRST_BUILD_UID=1000
ENV NIX_BUILD_GROUP_ID=101

RUN mkdir -m 0755 /nix && \
    chown root /nix && \
    addgroup -S nixbld && \
    for n in $(seq 1 10); do \
        adduser -D -H -g "Nix build user $n" \
            -G nixbld -s "$(command -v nologin)" "nixbld$n"; \
    done

# Install Nix
RUN sh <(curl -L https://nixos.org/nix/install) \
    --daemon --yes

RUN echo 'extra-experimental-features = flakes nix-command' >> /etc/nix/nix.conf

# nix rc service script
COPY nix-daemon.sh /etc/init.d/nix-daemon
# Make the script executable
RUN chmod a+rx /etc/init.d/nix-daemon && \
    cp /root/.nix-profile/bin/nix-daemon /usr/sbin # && \
    rc-update add nix-daemon

WORKDIR /app
COPY . /app

VOLUME [ "/sys/fs/cgroup" ]

# Alpine docker images allow only a single process to be launched
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

ENV PATH="${PATH}:/nix/var/nix/profiles/default/bin"

# CMD ["nix-shell", "/app/"]
CMD ["nix-shell", "-p", "nix-info", "--run", "'nix-info -m'"]

entrypoint.sh

#!/bin/sh

openrc default

rc-update add nix-daemon default

rc-service nix-daemon start

# exec nix-shell /app/

exec "$@"

nix-daemon.sh

#!/sbin/openrc-run
description="Nix multi-user support daemon"

command="/usr/sbin/nix-daemon"
command_background="yes"
pidfile="/run/$RC_SVCNAME.pid"

#!/sbin/openrc-run
description="Nix multi-user support daemon"

command="/usr/sbin/nix-daemon"
command_background="yes"
pidfile="/run/$RC_SVCNAME.pid"

Output

```log installing 'nix-2.24.10' error: … while setting up the build environment error: cannot set host name: Operation not permitted ```

Linux HYBDk5ilQDH1in3 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 GNU/Linux Windows 10 Enterprise Version 10.0.19045 Build 19045

Guesses

Networking stack complications. Do we need nested hyper-V virtualization? https://github.com/MicrosoftDocs/WSL/issues/899
Is it a cgroups issue on WSL2? Where and why is the offical nix installer setting the host name?

Priorities

My goal is to develop a multi-user nix container setup via classical installer. Currently, the determinate/lix seems to not support this (rust debugging and also the README's).

Add :+1: to issues you find important.

abathur commented 3 weeks ago

Where and why is the official nix installer setting the host name?

The error stack in your output is from Nix itself, so I'm fairly sure the answer is that the installer itself isn't directly doing this.

It looks like the error comes from here: https://github.com/NixOS/nix/blob/dfd0033afbbb12e6578ab3f1f026d15ff9dec132/src/libstore/unix/build/local-derivation-goal.cc#L1840-L1843

philipp-baumann commented 3 weeks ago

Where and why is the official nix installer setting the host name?

The error stack in your output is from Nix itself, so I'm fairly sure the answer is that the installer itself isn't directly doing this.

It looks like the error comes from here:

https://github.com/NixOS/nix/blob/dfd0033afbbb12e6578ab3f1f026d15ff9dec132/src/libstore/unix/build/local-derivation-goal.cc#L1840-L1843

Thanks a lot for this hint. When I omit the nix-shell runtime command, It still fails with the same message. I tested again on macOS in the same default.nix nix-shell (but with vfkit and podman), there it installs and i can build my derivations in default.nix in the container. On WSL2 Ubuntu 22.04, virtualized on Win10, and with podman and qemu combo, it fails.

If it would be nix-daemon launched as openRC service complaining, it would not be the installer message, or I am missing something obvious or very internal?:

"installing 'nix-2.24.10'
error:
       … while setting up the build environment

       error: cannot set host name: Operation not permitted

---- oh no! --------------------------------------------------------------------
Oh no, something went wrong. If you can take all the output and open
an issue, we'd love to fix the problem so nobody else has this issue.

:(

We'd love to help if you need it.

You can open an issue at
https://github.com/NixOS/nix/issues/new?labels=installer&template=installer.md

Or get in touch with the community: https://nixos.org/community
Error: building at STEP "RUN sh <(curl -L https://nixos.org/nix/install)     --daemon --yes": while running runtime: exit status 1"

philipp-baumann commented 2 weeks ago

For some more context and a repo with reproducible example, https://github.com/rstats-on-nix/r-minimal-alpine

NixOS / nix