ganto / copr-lxc4

RPM spec files for building the latest stable lxc/lxd/incus releases on Fedora COPR
MIT License
34 stars 6 forks source link

Incus blocked by SELinux #40

Closed ganto closed 8 months ago

ganto commented 9 months ago

Various operations of Incus violate the default SELinux policies present in Fedora.

dejlek commented 8 months ago

I can't even start the daemon... When I do try, this is what I get in the system logs:

Dec 28 21:15:16 mini.server.lan setroubleshoot[77166]: SELinux is preventing systemd from create access on the sock_file unix.socket. For complete SELinux messages run: sealert -l e1d45937-7747-4bba-b67b-a3f0fde5dcb4
Dec 28 21:15:16 mini.server.lan setroubleshoot[77166]: SELinux is preventing systemd from create access on the sock_file unix.socket.

                                                      *****  Plugin catchall_labels (83.8 confidence) suggests   *******************

                                                      If you want to allow systemd to have create access on the unix.socket sock_file
                                                      Then you need to change the label on unix.socket
                                                      Do
                                                      # semanage fcontext -a -t FILE_TYPE 'unix.socket'
                                                      where FILE_TYPE is one of the following: NetworkManager_dispatcher_console_var_run_t, NetworkManager_var_run_t, abrt_retrace_spool_t, abrt_var_run_t, aiccu_var_run_t, ajaxterm_var_run_t, alsa_var_run_t, antivirus_var_run_t, apcupsd_var_run_t, apmd_var_run_t, arpwatch_var_run_t, asterisk_spool_t, asterisk_var_run_t, audisp_var_run_t, audit_spool_t, auditd_var_run_t, automount_var_run_t, avahi_var_run_t, bacula_var_run_t, bcfg2_var_run_t, bitlbee_var_run_t, blkmapd_var_run_t, blktap_var_run_t, blueman_var_run_t, bluetooth_var_run_t, boltd_var_lib_t, boltd_var_run_t, boothd_var_run_t, bootloader_var_run_t, bootupd_var_run_t, brltty_var_run_t, bumblebee_var_run_t, cache_home_t, cachefilesd_var_run_t, callweaver_var_run_t, canna_var_run_t, cardmgr_var_run_t, ccs_var_run_t, certmaster_var_run_t, certmonger_var_run_t, cgred_var_run_t, chronyd_var_run_t, cinder_var_run_t, clogd_var_run_t, cluster_var_run_t, clvmd_var_run_t, cmirrord_var_run_t, collectd_var_run_t, comsat_var_run_t, condor_var_run_t, config_home_t, conman_var_run_t, conntrackd_var_run_t, consolekit_var_run_t, container_kvm_var_run_t, container_plugin_var_run_t, container_var_run_t, couchdb_var_run_t, courier_spool_t, courier_var_run_t, cpuplug_var_run_t, cpuspeed_var_run_t, cron_spool_t, cron_var_run_t, crond_var_run_t, ctdbd_var_run_t, cupsd_config_var_run_t, cupsd_lpd_var_run_t, cupsd_var_run_t, cvs_var_run_t, cyphesis_var_run_t, cyrus_var_run_t, data_home_t, dbskkd_var_run_t, dbus_home_t, dcc_var_run_t, dccd_var_run_t, dccifd_var_run_t, dccm_var_run_t, dcerpcd_var_run_t, ddclient_var_run_t, deltacloudd_var_run_t, devicekit_var_run_t, devlog_t, dhcpc_var_run_t, dhcpd_var_run_t, dictd_var_run_t, dirsrv_snmp_var_run_t, dirsrv_var_run_t, dkim_milter_data_t, dlm_controld_var_run_t, dnsmasq_var_run_t, dnssec_trigger_var_run_t, dovecot_spool_t, dovecot_var_run_t, drbd_var_run_t, dspam_var_run_t, entropyd_var_run_t, eventlogd_var_run_t, evtchnd_var_run_t, exim_spool_t, exim_var_run_t, fail2ban_var_run_t, fcoemon_var_run_t, fenced_var_run_t, fetchmail_var_run_t, fingerd_var_run_t, firewalld_var_run_t, foghorn_var_run_t, freeipmi_bmc_watchdog_var_run_t, freeipmi_ipmidetectd_var_run_t, freeipmi_ipmiseld_var_run_t, fsadm_var_run_t, fsdaemon_var_run_t, ftpd_var_run_t, games_srv_var_run_t, gconf_home_t, gdomap_var_run_t, getty_var_run_t, gfs_controld_var_run_t, gkeyringd_gnome_home_t, glance_var_run_t, glusterd_var_run_t, gnome_home_t, gnome_initial_setup_var_run_t, gpm_var_run_t, gpsd_var_run_t, greylist_milter_data_t, groupd_var_run_t, gssproxy_var_run_t, gstreamer_home_t, haproxy_var_run_t, hostapd_var_run_t, httpd_var_run_t, hwloc_var_run_t, ibacm_var_run_t, icc_data_home_t, icecast_var_run_t, ifconfig_var_run_t, inetd_child_var_run_t, inetd_var_run_t, init_tmp_t, init_var_lib_t, init_var_run_t, initrc_var_run_t, innd_var_run_t, insights_client_var_run_t, install_var_run_t, ipmievd_var_run_t, ipsec_mgmt_var_run_t, ipsec_var_run_t, iptables_var_lib_t, iptables_var_run_t, irqbalance_var_run_t, iscsi_var_run_t, isnsd_var_run_t, iwhd_var_run_t, jetty_var_run_t, kadmind_var_run_t, keepalived_var_run_t, keystone_var_run_t, kismet_var_run_t, klogd_var_run_t, kmod_var_run_t, krb5kdc_var_run_t, ksmtuned_var_run_t, l2tpd_var_run_t, lircd_var_run_t, lldpad_var_run_t, locate_var_run_t, logwatch_var_run_t, lpd_var_run_t, lsassd_var_run_t, lsmd_var_run_t, lttng_sessiond_var_run_t, lvm_var_run_t, lwiod_var_run_t, lwregd_var_run_t, lwsmd_var_run_t, mail_spool_t, mailman_var_run_t, mcelog_var_run_t, mdadm_var_run_t, memcached_var_run_t, minidlna_var_run_t, minissdpd_var_run_t, mirrormanager_var_run_t, mock_var_run_t, mon_statd_var_run_t, mongod_var_run_t, motion_var_run_t, mount_var_run_t, mpd_var_run_t, mqueue_spool_t, mrtg_var_run_t, mscan_var_run_t, munin_var_run_t, mysqld_var_run_t, mysqlmanagerd_var_run_t, naemon_var_run_t, nagios_spool_t, nagios_var_run_t, named_var_run_t, netlogond_var_run_t, neutron_var_run_t, news_spool_t, ninfod_run_t, nmbd_var_run_t, nova_var_run_t, nrpe_var_run_t, nscd_var_run_t, nsd_var_run_t, nslcd_var_run_t, ntop_var_run_t, ntpd_var_run_t, numad_var_run_t, nut_var_run_t, nx_server_var_run_t, oddjob_var_run_t, opafm_var_run_t, openct_var_run_t, opendnssec_var_run_t, openhpid_var_run_t, openshift_var_run_t, openvpn_var_run_t, openvswitch_var_run_t, openwsman_run_t, osad_var_run_t, pads_var_run_t, pam_var_console_t, pam_var_run_t, passenger_var_run_t, pasta_pid_t, pcp_var_run_t, pcscd_var_run_t, pdns_var_run_t, pegasus_openlmi_storage_var_run_t, pegasus_var_run_t, pesign_var_run_t, piranha_fos_var_run_t, piranha_lvs_var_run_t, piranha_pulse_var_run_t, piranha_web_var_run_t, pkcs11proxyd_var_run_t, pkcs_slotd_var_run_t, pki_ra_var_run_t, pki_tomcat_var_run_t, pki_tps_var_run_t, plymouthd_spool_t, plymouthd_var_run_t, policykit_var_run_t, polipo_pid_t, portmap_var_run_t, portreserve_var_run_t, postfix_spool_bounce_t, postfix_spool_t, postfix_var_run_t, postgresql_var_run_t, postgrey_spool_t, postgrey_var_run_t, pppd_var_run_t, pptp_var_run_t, prelude_audisp_var_run_t, prelude_lml_var_run_t, prelude_spool_t, prelude_var_run_t, print_spool_t, privoxy_var_run_t, prosody_var_run_t, psad_var_run_t, ptal_var_run_t, pulseaudio_var_run_t, puppet_var_run_t, pwauth_var_run_t, pyicqt_var_run_t, pyicqt_var_spool_t, qatlib_var_run_t, qdiskd_var_run_t, qemu_var_run_t, qmail_spool_t, qpidd_var_run_t, quota_nld_var_run_t, rabbitmq_var_run_t, radiusd_var_run_t, radvd_var_run_t, readahead_var_run_t, redis_var_run_t, regex_milter_data_t, restorecond_var_run_t, rhcd_var_run_t, rhev_agentd_var_run_t, rhnsd_var_run_t, rhsmcertd_var_run_t, ricci_modcluster_var_run_t, ricci_var_run_t, rlogind_var_run_t, rngd_var_run_t, roundup_var_run_t, rpcbind_var_run_t, rpcd_var_run_t, rpm_var_run_t, rrdcached_var_run_t, rsync_var_run_t, rtas_errd_var_run_t, rwho_spool_t, sanlock_var_run_t, saslauthd_var_run_t, sbd_var_run_t, sblim_var_run_t, screen_var_run_t, sendmail_var_run_t, sensord_var_run_t, setrans_var_run_t, setroubleshoot_var_run_t, slapd_var_run_t, slpd_var_run_t, smbd_var_run_t, smokeping_var_run_t, smsd_var_run_t, snmpd_var_run_t, snort_var_run_t, sosreport_var_run_t, soundd_var_run_t, spamass_milter_data_t, spamd_spool_t, spamd_var_run_t, spc_var_run_t, squid_var_run_t, squirrelmail_spool_t, srvsvcd_var_run_t, sshd_var_run_t, sslh_var_run_t, sssd_public_t, sssd_var_lib_t, sssd_var_run_t, stalld_var_run_t, stapserver_var_run_t, stratisd_var_run_t, stunnel_var_run_t, svnserve_var_run_t, swat_var_run_t, swift_var_run_t, syslogd_var_run_t, system_cron_spool_t, system_cronjob_var_run_t, system_dbusd_var_run_t, systemd_bootchart_var_run_t, systemd_importd_var_run_t, systemd_logind_inhibit_var_run_t, systemd_logind_sessions_t, systemd_logind_var_run_t, systemd_machined_var_run_t, systemd_networkd_var_run_t, systemd_passwd_var_run_t, systemd_resolved_var_run_t, systemd_timedated_var_run_t, systemd_userdbd_runtime_t, tangd_cache_t, targetclid_var_run_t, telnetd_var_run_t, tftpd_var_run_t, tgtd_var_run_t, thin_aeolus_configserver_var_run_t, thin_var_run_t, timemaster_var_run_t, tlp_var_run_t, tmpfs_t, tomcat_var_run_t, tor_var_run_t, tuned_var_run_t, udev_var_run_t, uml_switch_var_run_t, usbmuxd_var_run_t, user_cron_spool_t, user_tmp_t, useradd_var_run_t, uucpd_spool_t, uucpd_var_run_t, uuidd_var_run_t, var_run_t, var_spool_t, varnishd_var_run_t, varnishlog_var_run_t, vdagent_var_run_t, vhostmd_var_run_t, virt_common_var_run_t, virt_lxc_var_run_t, virt_qemu_ga_var_run_t, virt_var_run_t, virtlogd_var_run_t, vmware_host_pid_t, vmware_pid_t, vnstatd_var_run_t, vpnc_var_run_t, watchdog_var_run_t, wdmd_var_run_t, winbind_rpcd_var_run_t, winbind_var_run_t, xdm_spool_t, xdm_var_run_t, xenconsoled_var_run_t, xend_var_run_t, xenstored_var_run_t, xserver_var_run_t, ypbind_var_run_t, yppasswdd_var_run_t, ypserv_var_run_t, ypxfr_var_run_t, zabbix_var_run_t, zarafa_deliver_var_run_t, zarafa_gateway_var_run_t, zarafa_ical_var_run_t, zarafa_indexer_var_run_t, zarafa_monitor_var_run_t, zarafa_server_var_run_t, zarafa_spooler_var_run_t, zebra_var_run_t, zoneminder_var_run_t.
                                                      Then execute:
                                                      restorecon -v 'unix.socket'

                                                      *****  Plugin catchall (17.1 confidence) suggests   **************************

                                                      If you believe that systemd should be allowed create access on the unix.socket sock_file by default.
                                                      Then you should report this as a bug.
                                                      You can generate a local policy module to allow this access.
                                                      Do
                                                      allow this access for now by executing:
                                                      # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
                                                      # semodule -X 300 -i my-systemd.pp
ganto commented 8 months ago

Yes, I ran my analysis with SELinux in "permissive" mode. Your log is the more verbose version of the first alert that I posted for the startup.

m2Giles commented 8 months ago

As a starting point. We can relocate the socket to /run/ and set the INCUS_SOCKET environment variable to it. That will get the socket started and seems to be the workaround sorted out for lxd a few years ago

https://github.com/containers/container-selinux

Seems to be where lxd has the file contexts defined. Lxc has file contexts in their repo. How does lxd get properly labeled in the build process?

m2Giles commented 8 months ago

I seem to have incus working with SELinux right now.

I had to use the same workarounds for LXD socket.

Using the following file_contexts:

/usr/s?bin/incus  --      gen_context(system_u:object_r:container_runtime_exec_t:s0)
/usr/s?bin/incus-.*  --      gen_context(system_u:object_r:container_runtime_exec_t:s0)
/usr/lib/incus/.*  --      gen_context(system_u:object_r:container_runtime_exec_t:s0)
/usr/lib/systemd/system/incus.* --  gen_context(system_u:object_r:container_unit_file_t:s0)
/var/lib/incus(/.*)?      gen_context(system_u:object_r:container_var_lib_t:s0)
/var/log/incus(/.*)?      gen_context(system_u:object_r:container_log_t:s0)

I have a working alpine container created. It has networking once modifying firewalld.

m2Giles commented 8 months ago

VM's also appear to work fine.

ganto commented 8 months ago

First I tried to create a SELinux policy with the file context provided by @m2Giles then I tried to summarize them as:

/usr/s?bin/incus(.*)?           gen_context(system_u:object_r:container_runtime_exec_t:s0)
/usr/lib/systemd/system/incus.* gen_context(system_u:object_r:container_unit_file_t:s0)
/var/lib/incus(/.*)?            gen_context(system_u:object_r:container_var_lib_t:s0)
/var/log/incus(/.*)?            gen_context(system_u:object_r:container_log_t:s0)

Unfortunately in both cases I still get an error during the %selinux_modules_install of an incus-selinux package:

[...]
  Running scriptlet: container-selinux-2:2.226.0-1.fc39.noarch                         6/27 
  Installing       : container-selinux-2:2.226.0-1.fc39.noarch                         6/27 
  Running scriptlet: container-selinux-2:2.226.0-1.fc39.noarch                         6/27 
  Running scriptlet: incus-selinux-0.4-0.4.fc39.noarch                                 7/27 
  Installing       : incus-selinux-0.4-0.4.fc39.noarch                                 7/27 
  Running scriptlet: incus-selinux-0.4-0.4.fc39.noarch                                 7/27 
Failed to resolve filecon statement at /var/lib/selinux/targeted/tmp/modules/200/incus/cil:2                                                                                                                                                                                                                                  
Failed to resolve AST                                                                                                                                                                                                                                                                                                         
/usr/sbin/semodule:  Failed!

[...]

container-selinux is installed as dependency. Will continue investigating next time.

Source code used for testing can be found in the incus-selinux branch.

m2Giles commented 8 months ago

For your summarized one of them is missing.

/usr/lib/incus/(.*)

Unsure why it's failing. I was able to use semanage fcontext to set each of these.

This is what I have set via semanage context:

$ sudo semanage fcontext -l | grep "incus"
/usr/bin/incus                                     all files          system_u:object_r:container_runtime_exec_t:s0 
/usr/lib/incus/.*                                  all files          system_u:object_r:container_runtime_exec_t:s0 
/usr/lib/systemd/system/incus.*                    all files          system_u:object_r:container_unit_file_t:s0 
/usr/s?bin/incus                                   all files          system_u:object_r:container_runtime_exec_t:s0 
/usr/s?bin/incus-.*                                all files          system_u:object_r:container_runtime_exec_t:s0 
/var/lib/incus/(/.*)?                              all files          system_u:object_r:container_var_lib_t:s0 
/var/log/incus/(/.*)?                              all files          system_u:object_r:container_log_t:s0

So far I've yet to hit an SELinux denials.

ganto commented 8 months ago

The SELinux should now be fixed. Still have to update the documentation and fix the Rawhide builds. But feedback is welcome.

m2Giles commented 8 months ago

Thank you for getting this working. Besides a small workaround needed on my silverblue derived system (issue with how ostree manages file contexts), it works! I'm especially grateful you got the socket working in the correct location as well.

ganto commented 8 months ago

Any chance that you would share this workaround? I think there are also other people using Silverblue (e.g. #37) that might profit from it.

m2Giles commented 8 months ago

The issue is that /usr stuff won't have the right file contexts following install due to a bug with how ostree manages file contexts. They will exist in SELinux's compiled list so a restorecon will fix the problem but /usr is read-only. I work around this using a systemd service that will fix it on each boot.

# /etc/systemd/system/incus-workaround.service
[Unit]
Description=Workaround SELinux issues with Incus...
ConditionPathExists=/usr/bin/incus
ConditionPathExists=/usr/bin/incus-agent
ConditionPathExists=/usr/lib/incus
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/bin/mount --bind -o rw /usr/bin/incus /usr/bin/incus
ExecStart=/usr/bin/mount --bind -o rw /usr/bin/incus-agent /usr/bin/incus-agent
ExecStart=/usr/bin/mount --bind -o rw /usr/lib/incus /usr/lib/incus
ExecStart=/usr/sbin/restorecon -R -v /usr/bin/incus
ExecStart=/usr/sbin/restorecon -R -v /usr/bin/incus-agent
ExecStart=/usr/sbin/restorecon -R -v /usr/lib/incus
ExecStart=/usr/bin/umount /usr/bin/incus
ExecStart=/usr/bin/umount /usr/bin/incus-agent
ExecStart=/usr/bin/umount /usr/lib/incus
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Something similar is necessary for swtpm due to the same issue with ostree if you are using virtual machines thaopent need the tpm device.

Additionally, silverblue won't relabel things by default, if a user already was using incus, they will need to relabel /var/{lib,log}/incus. This would only be necessary if they already existed.

# restorecon -R -v /var/lib/incus
# restorecon -R -v /var/log/incus

The socket works with no workarounds needed which is awesome.

dejlek commented 8 months ago
shell» incus admin init --minimal
Error: Failed to connect to local daemon: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: permission denied

I can see that the service is running:

shell» systemctl status incus
● incus.service - Incus - Daemon
     Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Wed 2024-01-17 14:11:45 GMT; 5min ago
TriggeredBy: ● incus.socket
       Docs: man:incusd(1)
    Process: 18382 ExecStartPost=/usr/lib/incus/incusd waitready --timeout=600 (code=exited, status=0/SUCCESS)
   Main PID: 18381 (incusd)
      Tasks: 28
     Memory: 109.0M
        CPU: 335ms
     CGroup: /system.slice/incus.service
             └─18381 /usr/lib/incus/incusd --group incus-admin

Jan 17 14:11:44 myserver.lan systemd[1]: Starting incus.service - Incus - Daemon...
Jan 17 14:11:44 myserver.lan incusd[18381]: time="2024-01-17T14:11:44Z" level=warning msg="AppArmor support has been disabled because of lack of kernel suppor>
Jan 17 14:11:44 myserver.lan incusd[18381]: time="2024-01-17T14:11:44Z" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of >
Jan 17 14:11:44 myserver.lan incusd[18381]: time="2024-01-17T14:11:44Z" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits wil>
Jan 17 14:11:44 myserver.lan incusd[18381]: time="2024-01-17T14:11:44Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instan>
Jan 17 14:11:45 myserver.lan systemd[1]: Started incus.service - Incus - Daemon.

Version:

shell» incus --version
0.4

Could it be that I just need to wait for the new package? :)

m2Giles commented 8 months ago

Are you a member of the incus-admin group?

dejlek commented 8 months ago

Ah, right... Tried with superuser:

» sudo incus admin init --minimal
[sudo] password for bmll: 
Error: Failed to create local member network "incusbr0" in project "default": Failed generating auto config: Failed to automatically find an unused IPv4 subnet, manual configuration required

I will try to do the manual configuration, as suggested. Thanks @m2Giles !

dejlek commented 8 months ago

OK. I found what caused the above (in case someone else is having similar problem). - Incus, just like lxd/lxc, tries to setup a 10.x.x.x network by default, when none is available, like it is in my case (10.0.0.0/8 via 172.2.2.1 dev tun0 .. long story why it is so), it fails. So, I created manually a "smaller" 172.16.xx.xx network (192.168.16.x would also work for an example), and it all worked, well kind of.

So, something like the following helped: incus network create incusbr0 --type bridge ipv4.address=192.168.10.1/24 However incus containers can't get IPv4 addresses. At least I could create containers... :)

dejlek commented 8 months ago

It turned out that firewalld was enabled and it caused all this...