Error: could not set permissions on '...' to 755: Operation not permitted when using experimental local overlay store

[kevinh-canva]

Describe the bug

I'm trying to use the new local overlay store feature to combine a user-specific upper layer store (residing on an external EBS volume) with a lower /nix/store were pre-baked in the AMI.

The idea is: the external EBS volume will be persistent, while the underlying AMI might be rotated from time to time, but the packages from the upper and lower store will be merged together seamlessly thanks to the local overlay store.

However, when trying to use a command like nix-env/nix-store, I got hit with this error: error: could not set permissions on '/mnt/nix-overlay/nix/var/nix/profiles/per-user' to 755: Operation not permitted

Steps To Reproduce

1. Install nix 2.22.0 in multi-user mode (this is the 1st version the local overlay store feature is available I believe).
2. Mount the external EBS volume at /mnt/nix.
3. Create 2 sub-directories for the overlayfs upper and work dir: /mnt/nix/upper-store and /mnt/nix/work
4. Create the directory for the merged overlay store dir: /mnt/nix-overlay/nix/store.
5. Mount an OverlayFS using the following command:

mount -t overlay overlay \
  -o lowerdir="/nix/store" \
  -o upperdir="/mnt/nix/upper-store" \
  -o workdir="/mnt/nix/work" \
  "/mnt/nix-overlay/nix/store"

5. Add the following to /etc/nix/nix.conf (lower-store is omitted from the documentation, because we just use the default /nix/store).

extra-experimental-features = local-overlay-store
store = local-overlay://?root=/mnt/nix-overlay&upper-layer=/mnt/nix/upper-store

6. Reload and restart nix-daemon.socket and nix-daemon.service.

Expected behavior

nix commands like nix-env/nix-store should work.

nix --version output

2.22.0

Additional context

Running sudo --login nix-store/nix-env works, but I expect these commands to be runnable by normal users as well.

Also, strangely, the directory permission is already 0755

Priorities

Add :+1: to issues you find important.