search

DavHau / nix-portable

Nix - Static, Permissionless, Installation-free, Pre-configured
MIT License
779 stars 29 forks source link

cannot use `nix --store` runtime which is selected by default #98

Open orzFly opened 2 months ago

orzFly commented 2 months ago

Using nix-portable v011

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

By the way, bwrap runs fine.

$ NP_RUNTIME=bwrap nix-portable nix run nixpkgs#htop

Diagnostics

$ zgrep CONFIG_USER_NS /proc/config.gz
CONFIG_USER_NS=y
CONFIG_USER_NS_UNPRIVILEGED=y

$ cat /proc/sys/kernel/unprivileged_userns_clone
1

$ unshare -r -n echo YES
YES

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux
$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop 2>&1 | tee np_debug.log

np_debug.log

$ NP_DEBUG=2 strace nix-portable nix run nixpkgs#htop 2>&1 | tee strace.log

strace.log

$ grep unshare strace.log
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

$ strace unshare -r -m echo YES 2>&1 | egrep 'unshare|YES'
execve("/usr/bin/unshare", ["unshare", "-r", "-m", "echo", "YES"], 0x7ffc6e302bd0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/home/user/perl5/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/opt/asdf-vm/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = 0
write(1, "YES\n", 4YES
DavHau commented 2 months ago

Thanks for the detailed report. I'm not sure at the moment why the nix local store doesn't work but bubblewrap does.

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

orzFly commented 2 months ago

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

Still no luck :crying_cat_face: same error

I updated nix-portable, nuked ~/.nix-portable and tried:

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

Removing last_auto_runtime for a completer log

$ rm /home/user/.nix-portable/conf/last_auto_runtime
$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop >np_debug_v012.log 2>&1

np_debug_v012.log

orzFly commented 2 months ago

Worth noting in np_debug_v012.log: 

+ /home/user/.nix-portable/bin/nix --store /home/user/.nix-portable/tmp/__store shell -f /home/user/.nix-portable/mini-drv.nix -c /home/user/.nix-portable/bin/nix store add-file --store /home/user/.nix-portable/tmp/__store /home/user/.nix-portable/tmp/testfile
this derivation will be built:
  /nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv
building '/nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv'...
+ chmod -R +w /home/user/.nix-portable/tmp/__store
+ rm -r /home/user/.nix-portable/tmp/__store
+ debug 'nix --store works on this system -> will use nix as runtime'
+ echo nix --store works on this system '->' will use nix as runtime
nix --store works on this system -> will use nix as runtime

I am not sure why the second call to unshare is failed. I am not familiar with unshare-y things.

$ strace nix-portable nix run nixpkgs#htop 2>&1 | grep unshare
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

This command can success. I am not sure if this helps.

$ strace /usr/bin/unshare -r -m /usr/bin/unshare -r -m echo YES 2>&1 | grep unshare
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffcfe206b78 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffce18616e0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
DavHau commented 2 months ago

Which version of which distro are you running? Maybe I can add a test for it to the CI pipeline.

orzFly commented 2 months ago

Arch Linux. Up-to-dated (with an old kernel for ZFS).

However, this is a little embarrassing to say... Actually, I am new to nix. Due to my oversatisfaction to bundling applications with nix-portable as single executable, I just installed Nix (multiple users install)! With the real /nix directory, things has changed.

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux

$ sudo zfs unmount rpool/nix
$ mount | grep /nix
$ ls -alh /nix
total 10K
drwxr-xr-x  2 root root  2 Apr 16 02:10 .
drwxr-xr-x 18 root root 24 Apr 16 02:10 ..
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted

$ sudo zfs mount rpool/nix
$ mount | grep /nix
rpool/nix on /nix type zfs (rw,relatime,xattr,posixacl,casesensitive)
$ ls -alh /nix
total 1.1M
drwxr-xr-x   4 root root      4 Apr 16 02:11 .
drwxr-xr-x  18 root root     24 Apr 16 02:10 ..
drwxrwxr-t 461 root nixbld 4.2K Apr 16 02:27 store
drwxr-xr-x   4 root root      4 Apr 16 02:14 var
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
write(1, "Hello, world!\n", 14Hello, world!
orzFly commented 2 months ago

Reproducible with ArchLinux Netboot Release 2024.04.01. Here's the rough session transcript:

# uname -a
Linux archiso 6.8.2-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 28 Mar 2024 17:06:35 +0000 x86_64 GNU/Linux
# mount -o remount,size=50% /run/archiso/cowspace
# pacman -Sy wget strace
# useradd user
# mkdir /home/user
# chown -R user:user /home/user
# sudo -uuser -Hs
$ wget https://github.com/DavHau/nix-portable/releases/download/v012/nix-portable-x86_64
$ chmod +x nix-portable-x86_64
$ ./nix-portable-x86_64 nix run nixpkgs\#hello                          # let it download and fail
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs\#hello 2>&1 | grep -E unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted
$ NP_RUNTIME=bwrap ./nix-portable-x86_64 nix run nixpkgs\#hello
Hello, world!
orzFly commented 2 months ago

Maybe I can add a test for it to the CI pipeline.

Maybe you should try adding test for nix run nixpkgs#hello to the CI pipeline...

Producible on Debian GNU/Linux 11 (bullseye) ```shellSession orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix-shell -p hello --run hello Hello, world! orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello Hello, world! orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix-shell -p hello --run hello Hello, world! orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello error: setting up a private mount namespace: Operation not permitted orzfly@media-sorter:~$ uname -a Linux media-sorter 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03) x86_64 GNU/Linux orzfly@media-sorter:~$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye ```
ccornix commented 2 months ago

Reproducible on Debian GNU/Linux 12 (bookworm) as well, using v012.

ccornix@debian:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello
Hello, world!

ccornix@debian:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello
error: setting up a private mount namespace: Operation not permitted

ccornix@debian:~$ uname -a
Linux debian 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

ccornix@debian:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm
chaserhkj commented 1 month ago

Considering NixOS/nix#6853 is present and probably will be around for some time as well. Maybe we should just switch to bwrap runtime as default in nix-portable ?