netblue30 / firejail

Linux namespaces and seccomp-bpf sandbox
https://firejail.wordpress.com
GNU General Public License v2.0
5.82k stars 567 forks source link

dnsmasq: libvirtd cannot activate virtual network: PATH environment variable not set #6121

Open marek22k opened 11 months ago

marek22k commented 11 months ago

Description

I cannot activate the virtual network when firejail is activated.

Steps to reproduce the behavior

  1. Install Arch Linux
  2. Install qemu/kvm
  3. Run sudo virsh net-start default

Expected behavior

The network starts.

Actual behavior

The network does not start.

Behavior without a profile

Since dnsmasq is called by libvirt, it is difficult to do this manually. However, running it after firecfg --clean works.

Additional context

$ sudo virsh net-start default
error: Failed to start network default
error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

$ sudo firecfg --clean
Removing all firejail symlinks:
   alacarte removed
   atril removed
   atril-previewer removed
   atril-thumbnailer removed
   calibre removed
   chromium removed
   claws-mail removed
   codium removed
   com.github.tchx84.Flatseal removed
   conplay removed
   cvlc removed
   dig removed
   display removed
   dnsmasq removed
   drill removed
   ebook-convert removed
   ebook-edit removed
   ebook-meta removed
   ebook-polish removed
   ebook-viewer removed
   enchant-2 removed
   enchant-lsmod-2 removed
   ffplay removed
   ffprobe removed
   filezilla removed
   ftp removed
   gajim removed
   gapplication removed
   geany removed
   gimp removed
   gimp-2.10 removed
   gpa removed
   hexchat removed
   host removed
   img2txt removed
   inkscape removed
   inkview removed
   keepassxc removed
   keepassxc-cli removed
   keepassxc-proxy removed
   libreoffice removed
   librewolf removed
   lobase removed
   localc removed
   lodraw removed
   loffice removed
   lofromtemplate removed
   loimpress removed
   lomath removed
   loweb removed
   lowriter removed
   man removed
   mate-color-select removed
   meld removed
   mousepad removed
   mpg123 removed
   mpg123-id3dump removed
   mpg123-strip removed
   mpv removed
   nslookup removed
   out123 removed
   parole removed
   patch removed
   pdftotext removed
   ping removed
   pluma removed
   qt-faststart removed
   qtox removed
   ristretto removed
   seahorse removed
   secret-tool removed
   smplayer removed
   soffice removed
   sqlitebrowser removed
   ssh removed
   strings removed
   telnet removed
   tshark removed
   unbound removed
   vlc removed
   vscodium removed
   wget removed
   whois removed
   wireshark removed
   xfburn removed
   xfce4-dict removed
   xfce4-notes removed
   xfce4-screenshooter removed
   yt-dlp removed
   zeal removed

$ sudo virsh net-start default
Network default started

Already reported several times, but apparently not yet resolved:

Workaround:

sudo sed -i 's/^dnsmasq/# dnsmasq/' /etc/firejail/firecfg.config

Environment

$firejail --version
firejail version 0.9.72

Compile time support:
    - always force nonewprivs support is disabled
    - AppArmor support is enabled
    - AppImage support is enabled
    - chroot support is enabled
    - D-BUS proxy support is enabled
    - file transfer support is enabled
    - firetunnel support is disabled
    - IDS support is disabled
    - networking support is enabled
    - output logging is enabled
    - overlayfs support is disabled
    - private-home support is enabled
    - private-cache and tmpfs as user enabled
    - SELinux support is disabled
    - user namespace support is enabled
    - X11 sandboxing support is enabled

Checklist

Log

Output of LC_ALL=C firejail /path/to/program

``` $sudo LC_ALL=C firejail /usr/bin/virsh net-start default Reading profile /etc/firejail/server.profile Reading profile /etc/firejail/disable-common.inc Reading profile /etc/firejail/disable-programs.inc Reading profile /etc/firejail/disable-write-mnt.inc Reading profile /etc/firejail/disable-xdg.inc ** Note: you can use --noprofile to disable server.profile ** Parent pid 9679, child pid 9680 The new log directory is /proc/9680/root/var/log Warning: An abstract unix socket for session D-BUS might still be available. Use --net or remove unix from --protocol set. Child process initialized in 25.35 ms Error registering authentication agent: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: Cannot determine user of subject (polkit-error-quark, 0) error: Failed to start network default error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set Parent is shutting down, bye... ```

Output of LC_ALL=C firejail --debug /path/to/program

https://gist.github.com/marek22k/53c067d5a7e23121984dd8b6b74ebb5a

glitsj16 commented 11 months ago

sudo sed -i 's/^dnsmasq/# dnsmasq/' /etc/firejail/firecfg.config

The upcoming Firejail release will have override support for firecfg.config:

https://github.com/netblue30/firejail/blob/b02a7a337c759c130455956d5e9420c5ce3b6108/src/man/firecfg.1.in#L142-L187

If you use firejail-git ftom the AUR you can have that functionality now. Dropping a file like the below will disable dnsmasq sandboxing persistently:

$ cat /etc/firejail/firecfg.d/10-disabled.conf
!dnsmasq
ShellCode33 commented 11 months ago

@glitsj16 ideally I'd like to keep using the dnsmasq profile. While this can be a temporary workaround, it does not solve the underlying issue

glitsj16 commented 11 months ago

@ShellCode33 Agreed, the underlying issue is still not very clear (to me).

I've zero experience with libvirt/dnsmasq. Going over the referenced issue threads, I did notice https://github.com/netblue30/firejail/issues/5089#issuecomment-1094276371 mentions caps.keep chown,dac_override,net_admin,net_bind_service,net_raw,setgid,setuid might be needed (besides whitelist /var/lib/libvirt/dnsmasq and whitelist /var/run). OP's response seems to suggest that fixes things, but the resulting commits https://github.com/netblue30/firejail/commit/ce6f792efd0af09b95050864b71f79c46359fa49 and https://github.com/netblue30/firejail/commit/f3de2e37fd0bb3eb18393961f8382ff08fe3c3fb don't touch caps.keep.

Have you tried using dnsmasq.profile with caps.keep chown,dac_override,net_admin,net_bind_service,net_raw,setgid,setuid yet?

glitsj16 commented 11 months ago

Follow-up

I've installed libvirt/dnsmasq on my Arch Linux box to get a better understanding of this issue. With the below it works here, without the firecfg workaround:

$ cat ~/.config/firejail/dnsmasq.local
# Firejail profile for dnsmasq
# Persistent local customizations

allusers
caps.keep chown,net_admin,net_bind_service,net_raw,setgid,setuid
ignore caps.keep

Caveats:

@marek22k @ShellCode33 Can you try again with the above dnsmasq.local and report back here please? Hopefully we're closer to fixing this properly...

ShellCode33 commented 11 months ago

Thanks for taking the time to look at it @glitsj16 !

I still have the same PATH-related error:

$ sudo virsh net-start default
error: Failed to start network default
error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

You said that for you virsh was just hanging, that sounds odd. Make sure the libvirtd service is running

Here's my /etc/firejail/dnsmasq.local :

noblacklist /run/libvirt
noblacklist /usr/lib/libvirt
noblacklist /usr/local/bin/dnsmasq
noblacklist /usr/bin/dnsmasq
noblacklist /usr/bin/libvirtd

whitelist /usr/lib/libvirt
whitelist /run/libvirt
whitelist /usr/local/bin/dnsmasq
whitelist /usr/bin/dnsmasq
whitelist /usr/bin/libvirtd

noblacklist /usr/lib
noblacklist /usr/bin
noblacklist /usr/local/bin/
noblacklist /run

allusers
caps.keep chown,net_admin,net_bind_service,net_raw,setgid,setuid
ignore caps.keep

(I'm deliberately trying to be very permissive to narrow it down after, but that still doesn't work)

marek22k commented 11 months ago
$ sudo virsh net-start default
error: Failed to start network default
error: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/local/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 1: Error: PATH environment variable not set

$ cat /etc/firejail/dnsmasq.local
allusers
caps.keep chown,net_admin,net_bind_service,net_raw,setgid,setuid
ignore caps.keep
glitsj16 commented 11 months ago

You said that for you virsh was just hanging, that sounds odd. Make sure the libvirtd service is running

@ShellCode33 I did start libvirtd.service and virtlogd.service via systemd. Nothing special here. If you don't start those you'll indeed see errors:

$ sudo virsh net-start default
error: failed to connect to the hypervisor
error: Operation not supported: Cannot use direct socket mode if no URI is set

@ShellCode33 @marek22k Did you re-enable the dnsmasq symlink in /usr/local/bin (via firecfg or manually)? To make absolutely sure I created a wrapper script:

$ cat /usr/local/bin/dnsmasq
#!/bin/sh
#
## wrapper for dnsmasq
#+ sandbox support via firejail

### vars
_app="dnsmasq"
_bin="/usr/bin/${_app}"

# sandboxing
_bin="firejail --name=${_app}-6121 --quiet ${_bin}"

### logic
${_bin} "$@"

exit 0

Mind the --name=${_app}-6121 part. It's another assisting param to double-check if sandboxing dnsmasq is or isn't working. After issueing the virsh command you can run:

$ firejail --list | grep dnsmasq
11943:root:dnsmasq:firejail --name=dnsmasq-6121 --quiet /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

You should see the same if you added name dnsmasq-6121 (or anything you choose really) to dnsmasq.local.

Did any of you both added his user to the libvirt group? Any polkit stuff we're missing eyes on in this context? Check these docs for details: https://wiki.archlinux.org/title/Libvirt#Using_libvirt_group https://wiki.archlinux.org/title/Libvirt#Using_polkit https://wiki.archlinux.org/title/Polkit#Bypass_password_prompt https://wiki.archlinux.org/title/Polkit#Globally https://wiki.archlinux.org/title/Polkit#For_specific_actions

ShellCode33 commented 11 months ago

Did you re-enable the dnsmasq symlink in /usr/local/bin (via firecfg or manually)?

Yes it is currently enabled

Did any of you both added his user to the libvirt group?

Yes my user is part of this group, but I guess it doesn't matter considering we are running virsh using sudo, therefore polkit shouldn't be at play here


I tried to put your script in place of the /usr/local/bin/dnsmasq symlink, now virsh runs fine without error (at least it confirms this is not a PATH issue).

But now I have an apparmor denial :smiling_face_with_tear:

image

So I tried to disable this particular AppArmor profile, and now I observe the same behavior as you: virsh hangs.

While it's still hanging, I can see it's running within firejail:

 firejail --list | grep dnsmasq
63207:root:dnsmasq-6121:firejail --name=dnsmasq-6121 --quiet /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

EDIT: even if I CTRL+C virsh, dnsmasq is still running, but it's a bit annoying because all virsh commands hang, even sudo virsh net-list

EDIT2: my dnsmasq.locale is completely empty and it's still working. I'm starting to wonder if this is a bug in libvirt which does not resolve the symlink properly

EDIT3:

CTRL+C doesn't work, the libvirtd daemon will error after some time.

And when it's automatically restarted by systemd, the following errors/warning are emitted:

systemd logs ``` Dec 11 23:13:15 laptop libvirtd[66091]: End of file while reading data: Input/output error Dec 11 23:15:45 laptop libvirtd[66091]: Make forcefull daemon shutdown Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Main process exited, code=exited, status=1/FAILURE ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ An ExecStart= process belonging to unit libvirtd.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit libvirtd.service has entered the 'failed' state with result 'exit-code'. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Unit process 66475 (dnsmasq) remains running after unit stopped. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Unit process 66476 (firejail) remains running after unit stopped. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Unit process 66477 (firejail) remains running after unit stopped. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Unit process 66490 (dnsmasq) remains running after unit stopped. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Unit process 66491 (dnsmasq) remains running after unit stopped. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Consumed 1.201s CPU time, 12.4M memory peak, 0B memory swap peak. ░░ Subject: Resources consumed by unit runtime ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit libvirtd.service completed and consumed the indicated resources. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 1. ░░ Subject: Automatic restarting of a unit has been scheduled ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ Automatic restarting of the unit libvirtd.service has been scheduled, as the result for ░░ the configured Restart= setting for the unit. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Found left-over process 66475 (dnsmasq) in control group while starting unit. Ignoring. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Found left-over process 66476 (firejail) in control group while starting unit. Ignoring. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Found left-over process 66477 (firejail) in control group while starting unit. Ignoring. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Found left-over process 66490 (dnsmasq) in control group while starting unit. Ignoring. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: Found left-over process 66491 (dnsmasq) in control group while starting unit. Ignoring. Dec 11 23:15:45 laptop systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Dec 11 23:15:45 laptop systemd[1]: Starting libvirt legacy monolithic daemon... ░░ Subject: A start job for unit libvirtd.service has begun execution ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit libvirtd.service has begun execution. ░░ ░░ The job identifier is 4850. Dec 11 23:15:45 laptop systemd[1]: Started libvirt legacy monolithic daemon. ░░ Subject: A start job for unit libvirtd.service has finished successfully ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit libvirtd.service has finished successfully. ░░ ░░ The job identifier is 4850. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Deactivated successfully. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit libvirtd.service has successfully entered the 'dead' state. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Unit process 66475 (dnsmasq) remains running after unit stopped. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Unit process 66476 (firejail) remains running after unit stopped. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Unit process 66477 (firejail) remains running after unit stopped. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Unit process 66490 (dnsmasq) remains running after unit stopped. Dec 11 23:17:45 laptop systemd[1]: libvirtd.service: Unit process 66491 (dnsmasq) remains running after unit stopped. ```
rieje commented 3 weeks ago

FWIW I'm having the same issue on Arch. I don't use AppArmor.

Utini2000 commented 1 week ago

Same issue here on Arch. Disabled dnsmasq profile in firejail. Otherwise it won't function.