Nix multi-user installer fails on Fedora Rawhide due to SELinux denial

[outergod]

I have just come across this trying to install Nix on Fedora 28 Workstation with SELinux enabled (default) using the ./install-multi-user script from the nix-2.0.4-x86_64-linux bundle.

---- sudo execution ------------------------------------------------------------
I am executing:

    $ sudo systemctl link /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.service                                                                                

to set up the nix-daemon service

Failed to link unit: Access denied

I can confirm this is an SELinux denial due to this AVC message in /var/log/audit/audit.log:

type=AVC msg=audit(1535267537.912:2983): avc:  denied  { read } for  pid=1 comm="systemd" name="default" dev="nvme0n1p7" ino=2095622 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=lnk_file permissive=0 

This issue seems to be the same as the one discussed in #nixos-dev.

Based on the nature of the error, I would naturally assume all SELinux-enabled systems would be affected by this.

[dezgeg]

Yeah, probably the installer should bail out if SELinux is enforcing (at least for now).

[grahamc]

This bug is especially important now, the default installer as of 2.1 will detect systemd and default to the multi-user installer. See the attached PDF for a demonstration of the problem. (caution: big) report-d-2.pdf

[grahamc]

Adding this logic:

        if /usr/sbin/sestatus && /usr/sbin/sestatus 2>&1 | grep -q enforcing; then
            echo "punting to no-daemon"
                curl "${installUrl}" | sh -s -- --no-daemon
        else 
                  curl "${installUrl}" | sh
        fi

worked in the test harness, but the following did not:

        if sestatus && sestatus 2>&1 | grep -q enforcing; then
            echo "punting to no-daemon"
                curl "${installUrl}" | sh -s -- --no-daemon
        else 
                  curl "${installUrl}" | sh
        fi

because the install script, when run over SSH, doesn't have /usr/sbin in the PATH.

I'm not sure the nicest way to do this, but we could try adding a few directories to the PATH (/sbin, /usr/sbin, others ...?) just for this check.

Another option is we do a naive check for sestatus and assume it is in the PATH and leave it up to the users to handle it otherwise.

Any ideas / recommendations?

[dezgeg]

The getenforce command should be better for scripting.

Perhaps:

getenforce=$(PATH=$PATH:/sbin:/usr/sbin command -v getenforce)
if [ -n "$getenforce" ] && [ "$($getenforce)" = Enforcing ]; then
    echo "No multi-user for you, sorry"
fi

[outergod]

Sorry for the stupid question, but why not just set proper SELinux file contexts for everything in /nix and restorecon those contexts on installation as well as after manipulating the store? I could just get a service in /nix/store to run using my system's systemd after executing

semanage fcontext -a -t bin_t '/nix/store/[^/]+/bin(/.*)?' 
semanage fcontext -a -t lib_t '/nix/store/[^/]+/lib(/.*)?' 
restorecon -R /nix/store

This is also akin to the default file contexts for /opt:

/opt/(.*/)?bin(/.*)?    system_u:object_r:bin_t:s0
/opt/(.*/)?lib(/.*)?    system_u:object_r:lib_t:s0
/opt/(.*/)?man(/.*)?    system_u:object_r:man_t:s0
(etc)

This could be easily integrated into the installation, and later more ideally with Nix in general and nix-store in particular.

Edit: After setting the profile symlinks in /nix/var/nix/profiles/ to usr_t, I was also able to run a service pointing to a profile directory because now systemd happily follows the symlinks, and it's easily arguable that user environments correspond to /usr.

[dezgeg]

Because I personally don't know anything about writing SELinux rules for Fedora (nor CentOS for that matter.

Don't you also need something to allow systemd to use socket activation for the Nix daemon? At least I ran into similar issue to https://github.com/ganto/copr-lxd/issues/12.

[outergod]

Since I need to be able to run Nix based services, I will try my best to provide patches for the installer and possibly nix-store to integrate with SELinux properly.

[dezgeg]

Yes, that sounds good. I think we still need some quick solution for Nix 2.1.

[outergod]

Newly filed, related issue: https://github.com/systemd/systemd/issues/9997

[outergod]

@dezgeg you might want to check out https://github.com/e-user/nix/commit/2d53311c90a195ba7674f09605eb74124adfbff0, which depends on https://github.com/NixOS/nixpkgs/pull/46028 to be built. You could easily argue the policy belongs directly into nixpkgs because it provides value beyond the installation on non-NixOS systems. But as a first step, this should solve the issue of running Nix in multi-user mode in SELinux-enabled systems, but will not work once Nix is updated after installation - that could be fixed as a second step but requires some discussion as to how file contexts are preserved/restored when writing out derivations into the store:

1. SELinux file contexts can be baked directly into all derivations - which would make SELinux an integral part of Nix and might not be desired but requires no further changes to nix-store.
2. nix-store could be compiled with optional, direct SELinux support and write out file labels according to current, active policies. This is what systemd does when e.g. creating listening sockets.
3. nix-env and nix-store could should invoke restorecon after operating on profiles and the store, respectively - it's cheap and requires no deep, further modifications to any other tool.

[outergod]

If anyone wants this issue fixed as much as I do, please help me test #2670.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[sonirico]

This is still happening, I have the same output when installing nix as the initial comment.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[teohhanhui]

Bad stale bot.

This is obviously still not fixed.

[yawaramin]

I feel like I should point out here that the point of the stale bot is not to care about whether or not something is fixed, but to make a guesstimate that something will not be fixed because no one has bothered to fix it till now. To put the issue to rest either way, perhaps a maintainer would be so kind as to close this issue as 'wontfix' and make a note that Nix doesn't support Fedora or other SELinux distros.

[cgoetzke]

I think I'm hitting this, too.

Fedora 34 Workstation (KDE Plasma Spin).

Tried a multi-user install with sh <(curl -L https://nixos.org/nix/install) --daemon and got (snipped for brevity):

---- sudo execution ------------------------------------------------------------
I am executing:

    $ sudo systemctl link /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.service

to set up the nix-daemon service

Failed to link unit: Access denied

---- oh no! --------------------------------------------------------------------
Jeeze, something went wrong. If you can take all the output and open
an issue, we'd love to fix the problem so nobody else has this issue.

:(

We'd love to help if you need it.

You can open an issue at https://github.com/nixos/nix/issues

Or feel free to contact the team:
 - Matrix: #nix:nixos.org
 - IRC: in #nixos on irc.libera.chat
 - twitter: @nixos_org
 - forum: https://discourse.nixos.org

Since this issue has been open for 3.5 years, I'm assuming this isn't likely to get fixed anytime soon.

If it's not going to get a proper fix, it would be nice if the multi-user install would at least check for SELinux at the top and bail before starting to make changes on the system.

I'm also open to suggestions on the best way to back out this partial install I have now.

EDIT: I apologize for the grumpy tone. I'm tired and out of time to fiddle with this anymore tonight. I was excited to give Nix a try, but my initial experience was... probably not what any of us would have hoped for.

Anyway, any help or suggestions are appreciated.

EDIT 2: I went back through the install output (which seems to be very thorough) and manually un-did everything the install script reported doing, so I've probably fixed my system/cleaned up the partial Nix install.

So at this point, just consider this post a request that the Nix multi-user install either:

1. Be improved to work under SELinux; or, failing that, at least
2. Check for SELinux at the beginning of the install and abort gracefully before making any changes if SELinux is in place.

[ryleu]

Can confirm with Fedora 35.

Snippet:

---- sudo execution ------------------------------------------------------------
I am executing:

    $ sudo systemctl link /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.service

to set up the nix-daemon service

Failed to link unit: Access denied

---- oh no! --------------------------------------------------------------------

full output

My declarative adventure has been put on hold, it seems.

[emlautarom1]

Any workaround for Fedora users?

[eoghanlawless]

I'm facing the same issue on CentOS Stream 9

 $ uname -a
Linux <hostname> 5.14.0-47.el9.x86_64 #1 SMP PREEMPT Sat Jan 22 02:21:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

 $ cat /etc/centos-release
CentOS Stream release 9

[ryleu]

It seems that this will not be fixed, considering that it’s 3 years old. I’ll just have to find another declarative package manager, I suppose.

[bjorn]

So, this just failed for me on Fedora 35 as well. Since the issue has been known for so long, I'm surprised that there isn't at least an early-out check in the installer, because now it has done a lot of things on my system, which I have no idea how to undo. :-(

Also a warning could be added to the installation instructions about this problem.

This message from the install script prompted me to go here, but it leaves me a bit speechless finding this old issue:

Jeeze, something went wrong. If you can take all the output and open
an issue, we'd love to fix the problem so nobody else has this issue.

How many people still need to run into this and are left with a half-finished Nix install on their system?

[ryleu]

At least three. I’m incredibly annoyed that it didn’t at least check before doing a ton on random stuff to my computer that may leave it in an insecure state, for all I know.

[kiteloopdesign]

Same issue here with Fedora 35. Many years later and not even a workaround has been provided.

I understand they must be swamped with so many github issues, but not properly documenting this is going to cost them so many more issues being submitted

[matthewpi]

I was able to get Nix on Fedora Silverblue installed by using these workarounds.

https://gist.github.com/matthewpi/08c3d652e7879e4c4c30bead7021ff73#file-readme-md

[stelcodes]

Ran into the same issue on Fedora 35. Install script failed halfway through due to SELinux. For now, you either have to permanently disable SELinux or write your own SELinux rules. I'm opting to turn off SELinux permanently, but do so at your own risk. There are security implications.

Turn off SELinux: https://tecadmin.net/how-to-disable-selinux-on-fedora/

Here are the commands I ran to clean up the installation enough to reinstall:

sudo rm /etc/bashrc
sudo mv /etc/bashrc.backup-before-nix /etc/bashrc
sudo rm -r /nix
sudo systemctl disable nix-daemon

The systemctl disable might not be necessary, I just had to do that because I tried to enable and start the service before realizing there was way more unfinished business left in the installation process.

[bjorn]

After this, rerun the installation script and it should work.

@stelcodes So you mean, if no cleanup is necessary, the only thing to make the install work is running sudo setenforce 0 beforehand? Is this change temporary? If not, should or can it be re-enabled after installation? Or would the more specific permission changes by @matthewpi be preferred?

[stelcodes]

@bjorn Sorry for the confusion, I just edited my comment to make it more clear. I'm deciding to turn off SELinux permanently. The more specific permission changes should definitely be preferred, and I will try that in the future. If you're disabling SELinux, I would highly recommend locking down your ports with a firewall like firewalld or ufw.

[ryleu]

The solution is to create SELinux policies, but the ability to do so is an exceptionally rare skill. I am learning how to do so, but it may take me multiple months. I'll send them over once they're ready. From there, the devs can work on getting it merged into distros that use SELinux.

[adisbladis]

I have created an alternate set of installers using distribution native packaging formats over https://nix-community.github.io/nix-installers/. Currently supported distributions are rpm-based, deb-based and pacman-based. All of these come with selinux policies and it's been tested on Fedora along with Ubuntu & Arch.

I hope that after more testing & polishing this can become our default recommended installation method for the supported distributions.

Please note that this doesn't work for rpm-ostree distros such as Fedora Silverblue, but it works for regular Fedora.

[RokeJulianLockhart]

@adisbladis, do you intend to add this to Fedora's repository so that users are able to install it via dnf? That would cause prospective users to be easily able to install the software, and provide significantly more trust than an external binary invoked by bash does. I dare state that after what "http://github.com/NixOS/nix/issues/6552" demonstrates that the usual installer did to my computer, subsequent trial of Nix worried me slightly.

If the file is intended to remain external, please provide a vanity-link so that I am able to automate the process of acquisition of the installer.

[adisbladis]

do you intend to add this to Fedora's repository so that users are able to install it via dnf?

No, and I do not think that they would permit a package built in this kind of way and used as a one-shot install rather than a fully distro managed package.

If the file is intended to remain external, please provide a vanity-link so that I am able to automate the process of acquisition of the installer.

What do you mean by vanity link in this context? A stable URL to the installer or something else? In any case I've added a -latest that works for every installer: https://nix-community.github.io/nix-installers/x86_64/nix-multi-user-latest.deb

[RokeJulianLockhart]

@adisbladis, I would nevertheless be incredibly grateful if you were to attempt to add it, not because I particularly desire it, but because recommendation of this is more easy if the distribution has verified that it is not malicious, intentional or not.

[RokeJulianLockhart]

@adisbladis, the "-latest" is approximately what I desire, but I would prefer a .noarch version so that my code does not require as much verification of the bit-depth of the operating-system of its host.

Additionally, I do not believe that it is specified within "http://nix-community.github.io/nix-installers", so please add that it if I am correct, so that other users learn of this ability.

I am thankful for your assistance.

[chdoc]

I just want to note that this it is now 2022 and SELinux on Fedora 35 is so unobtrusive that many people (including me) may not even realize their system is using SELinux. So modifying a host of things all over the system and then bailing at the very end due to an issue that's been known for almost four years is about the gravest sin I have ever seen an installer perform in over a decade of working in IT.

Once I've cleaned up this mess, I'll probably refrain from installing nix on anything other than throw-away VMs for the foreseeable future. :angry:

[stelcodes]

@chdoc The installer really needs to display a huge warning with an option to bail on a multi-user install if SELinux is turned on. It should be a relatively simple addition to the installer, I'm really not sure why it hasn't been added. I might try working on this. Cleaning up the mess isn't too hard:

sudo rm /etc/bashrc
sudo mv /etc/bashrc.backup-before-nix /etc/bashrc
sudo rm -r /nix

Then turn off SELinux: sudo setenforce 0 Make sure it's turned off: sudo getenforce Then install Nix again.

Of course if you're no longer interested in Nix then you will have to clean up the users as well.

[chdoc]

Well, let's assume for a moment that I would like to actually get this working the way it should be, that is without switching off SELinux. From the looks of it, the problem is that it tries to "install" nix-daemon.service through systemctl link (i.e., making the .service file available to systemd via a symbolic link). However SELinux forbids systemd from accessing /nix. So I suppose there are two options here:

1. add a SELinux rule enabling the systemctl link to succeed
2. Install the .service file to the default location (/lib/systemd/system), this should not require any SELinux rule.

Option (2) seems preferable to me.

Of course, there may still be an issue with the permissions of the sockets used by the service. But for that I'd need to know more about how that works and what rules (Fedora) SELinux imposes here by default.

[ahayzen]

2. Install the `.service` file to the default location (`/lib/systemd/system`), this should not require any SELinux rule.

Note that with Fedora Silverblue /lib is read-only. And if you disable SELinux it is currently possible to install Nix on Fedora Silverblue with nix-daemon.service being linked into /etc.

If a .service was installed into /lib could there be a fallback to install into /etc as well if /lib isn't writeable ?

[chdoc]

Note that with Fedora Silverblue /lib is read-only.

Do you mean read only in the sense that not even a software install (which anyway requires root privileges) cannot install new files in /lib on Fedora Silverblue? Or is this about updating the .service file, which again (for good reason) shouldn't be possible without root privileges. I guess that's part of the point of forbidding linked systemd files that can be modified uncontrollably.

But yes, I guess one could also install a service to /etc/systemd/system but that's not exactly clean, as IIUC this is for configuration (linking disabled services to /dev/null and the likes).

Note that one of the main issues with https://github.com/NixOS/nix/pull/2670 seems to be to find a clean way to deal with the nix store and the services modifying it. But skimming over the discussion I wasn't able to identify the problem clearly enough to judge the merits of the proposed solutions.

So I acknowledge that proper SELinux support is not entirely trivial. But that means that until there is a tested satisfactory solution, the installer should check for SELinux on multi-user installs and abort rather than adding dozens of users and leaving files scattered across the system.

[ahayzen]

Do you mean read only in the sense that not even a software install (which anyway requires root privileges) cannot install new files in /lib on Fedora Silverblue?

Yes, Fedora Silverblue is an emerging variant of Fedora which uses ostree and is immutable, this means that /bin /lib /usr are a read-only filesystem, only /etc /var are writeable (/home is stored in /var). You can see this below

$ sudo touch /lib/systemd/system/test
[sudo] password for andrew: 
touch: cannot touch '/lib/systemd/system/test': Read-only file system

So if /lib is a read-only filesystem it would be good to have a fallback of either linking or writing directly to /etc if this becomes the route to install without SELinux denials.

Note that if a rpm of nix was in the fedora repository then Silverblue does allow for "layering" this package into the read-only part of ostree, so in that case /lib could be used, but I believe you were describing using the Nix installer.

[chdoc]

Thank you for the explanation regarding the Silverblue specifics.

Note that if a rpm of nix was in the fedora repository then Silverblue does allow for "layering" this package into the read-only part of ostree, so in that case /lib could be used, but I believe you were describing using the Nix installer.

I (semi-purposefully) wasn't specific about the mechanism used for the install. However, I think that - from the user perspective - installing software through the distributions own package manager is always preferable to running an install script. In particular the package manager usually records which installed files belong to which installed package (e.g., by running rpm -q --whatprovides <file>) and can therefore simply and cleanly uninstall packages. Of course, I acknowledge that providing (good) installers requires knowledge about both the software to be installed as well as the quirks of the particular distribution (or even release). This is something neither the Nix developers nor the Fedora maintainers may be willing to engage in. :man_shrugging:

[Artturin]

Made a pr to check if selinux is enabled and then abort if it is https://github.com/NixOS/nix/pull/6639

[RokeJulianLockhart]

@outergod should modify the title to demonstrate that this affects Rawhide rather than any specific release.

[thomasjcf21]

I've raised #6881 as the solution provided in #6639 prevents install on a system with SELinux set to permissive. For all intents and purposes disabled and permissive should be treated the same (as far as nix is concerned).

I've gone into further detail on the PR.

[RokeJulianLockhart]

@thomasjcf29, you might want to link your comments rather than manually duplicate them. Makes it easier to maintain them when there's a spelling-mistake, etcetera.

[pmoieni]

Is there any fix for this or we should use the single-user install?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1

[uxdiin]

I have created an alternate set of installers using distribution native packaging formats over https://nix-community.github.io/nix-installers/. Currently supported distributions are rpm-based, deb-based and pacman-based. All of these come with selinux policies and it's been tested on Fedora along with Ubuntu & Arch.

I hope that after more testing & polishing this can become our default recommended installation method for the supported distributions.

Please note that this doesn't work for rpm-ostree distros such as Fedora Silverblue, but it works for regular Fedora.

anyone having xdg-desktop-portal error after installing this

[dnkmmr69420]

Try out this guide it works on silverblue, workstation, and RHEL (tested on rocky linux)

[cgevans]

6639 is useful in preventing failed multi-user installations because of this bug, but when requesting a single-user installation, the installer still (incorrectly) states "Note: a multi-user installation is possible".