nix-daemon socket failure on Ubuntu 22.04

[kquick]

We are in the process of upgrading our github CI workflow from Ubuntu 18 and 20.04 to 22.04, but are getting failures on the nix portion.
Our configuration: https://github.com/GaloisInc/crucible/blob/5676b291af0134ffe2e1fcfb7d01d27992563660/.github/workflows/crux-llvm-build.yml#L68-L73 When the CI runs "nix shell github:nixos/nixpkgs/21.11#cabal-install github:nixos/nixpkgs/21.11#haskell.compiler.ghc8107 nixpkgs#gmp nixpkgs#zlib nixpkgs#zlib.dev", we get the error:

error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted

This sounds like the situation where the nix-daemon isn't working.

The tail of the installation appears successful:

2022-08-15T15:21:35.5270585Z ~~> Setting up the nix-daemon systemd service
2022-08-15T15:21:35.5414014Z Created symlink /etc/systemd/system/nix-daemon.service → /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.service.
2022-08-15T15:21:35.8133420Z Created symlink /etc/systemd/system/sockets.target.wants/nix-daemon.socket → /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.socket.
2022-08-15T15:21:35.8134521Z Created symlink /etc/systemd/system/nix-daemon.socket → /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.socket.
2022-08-15T15:21:36.3839685Z Alright! We're done!
2022-08-15T15:21:36.3866717Z Try it! Open a new terminal, and type:

Adding some systemctl status requests to the build reveals that indeed the nix-daemon wasn't started:

○ nix-daemon.service - Nix Daemon
     Loaded: loaded (/etc/systemd/system/nix-daemon.service; linked; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ○ nix-daemon.socket
  Condition: start condition failed at Mon 2022-08-15 16:22:33 UTC; 9s ago
             └─ ConditionPathIsReadWrite=/nix/var/nix/daemon-socket was not met

Aug 15 16:22:33 fv-az440-827 systemd[1]: Condition check resulted in Nix Daemon being skipped.
Error: Process completed with exit code 3.

○ nix-daemon.socket - Nix Daemon Socket
     Loaded: loaded (/etc/systemd/system/nix-daemon.socket; enabled; vendor preset: enabled)
     Active: inactive (dead)
   Triggers: ● nix-daemon.service
  Condition: start condition failed at Mon 2022-08-15 16:52:43 UTC; 12s ago
             └─ ConditionPathIsReadWrite=/nix/var/nix/daemon-socket was not met
     Listen: /nix/var/nix/daemon-socket/socket (Stream)

Aug 15 16:52:43 fv-az564-989 systemd[1]: Condition check resulted in Nix Daemon Socket being skipped.
Error: Process completed with exit code 3.

and an ls -lh /nix/var/nix/daemon-socket fails with ls: cannot access '/nix/var/nix/daemon-socket': No such file or directory.

I find it hard to believe that there's a nix issue with Ubuntu 22.04 at this late date (and don't see any such complaints anywhere), but I'm at a loss to know what we might be doing wrong. And this works fine under Ubuntu 20.04, with the only real change in the CI workflow being changing the OS to 22.04. Any suggestions or help would be appreciated.

[RyanGlScott]

I've created a repo here for testing purposes that demonstrates the issue. The key part is this GitHub Actions configuration:

name: GHA-Sandbox
on:
  - push
  - pull_request
jobs:
  linux:
    name: install-nix-action with ${{ matrix.runs-on }}
    runs-on: ${{ matrix.runs-on }}
    strategy:
      matrix:
        include:
          - runs-on: ubuntu-20.04
          - runs-on: ubuntu-22.04
      fail-fast: false
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Install Nix
        uses: cachix/install-nix-action@v16
        with:
          nix_path: nixpkgs=channel:21.11
          install_url: https://releases.nixos.org/nix/nix-2.4/install

      - name: Test nix shell
        run: |
          nix shell nixpkgs#gmp -c ls

As can be seen in this workflow, the ubuntu-20.04 job succeeds and the ubuntu-22.04 job fails. There are no other differences between the jobs besides the Ubuntu version.

[RyanGlScott]

The culprit appears to be the fact that install_url is pinning a somewhat old version of Nix (2.4). If I upgrade to the latest version:

@@ -20,7 +20,7 @@ jobs:
         uses: cachix/install-nix-action@v16
         with:
           nix_path: nixpkgs=channel:21.11
-          install_url: https://releases.nixos.org/nix/nix-2.4/install
+          install_url: https://releases.nixos.org/nix/nix-2.10.3/install

       - name: Test nix shell
         run: |

Then this works on both Ubuntu 20.04 and 22.04. I'm unclear what the bugfix was, but in any case, this demonstrates a pretty clear path forward.

[domenkozar]

2.4 is quite old at this point, can we close this issue?

[kquick]

Yup, sorry we didn't identify that earlier.