Spawning singleuser.service fails permission denied by SELinux

warrenson commented 1 month ago

Spawner service fails to start server

Starting the single server fails on RHEL9 (5.14.0-427.24.1.el9_4.x86_64) with SELinux enabled (enforcing). It seems SELinux is preventing the services spawned by the main service from writing into /run/systemd/.

Disabling SElinux fixes the problem.

Is there a configuration that is required, or a pre-defined SELinux rule, that will allow the systemdspawner to work with SELinux enabled?

How to reproduce

Start a single user server on RHEL9 with SElinux enabled. Default configuration with systemdspawner config added:

c.JupyterHub.spawner_class = 'systemd'
c.SystemdSpawner.default_shell = '/bin/bash'
c.SystemdSpawner.cpu_limit = 1
c.SystemdSpawner.mem_limit = '4G'

Expected behaviour

Once SELinux is disabled (set to permissive) everything works fine, the jupyter-USERNAME-singleuser.service can create the files it needs in /run/systemd/. The server starts and the user is able to use notebooks etc as expected.

Actual behaviour

The server fails to start, with error below:

Jul 19 15:29:23 HOSTNAME systemd[1]: Started /opt/jupyterhub/bin/jupyterhub-singleuser.
Jul 19 15:29:23 HOSTNAME systemd[208722]: jupyter-USERNAME-singleuser.service: Failed to set up special execution directory in /run: Permission denied
Jul 19 15:29:23 HOSTNAME systemd[208722]: jupyter-USERNAME-singleuser.service: Failed at step RUNTIME_DIRECTORY spawning /opt/jupyterhub/bin/jupyterhub-singleuser: Permission denied
Jul 19 15:29:23 HOSTNAME systemd[1]: jupyter-USERNAME-singleuser.service: Main process exited, code=exited, status=233/RUNTIME_DIRECTORY
Jul 19 15:29:23 HOSTNAME systemd[1]: jupyter-USERNAME-singleuser.service: Failed with result 'exit-code'.
Jul 19 15:32:30 HOSTNAME systemd[1]: jupyter-USERNAME-singleuser.service: Failed to open /run/systemd/transient/jupyter-USERNAME-singleuser.service: No such file or directory

Your set up

RHEL9 (5.14.0-427.24.1.el9_4.x86_64) with SELinux enabled.

Jupyterhub 4.1.5, Python 3.9.19, via conda/miniforge

Tested with both PAM (default) and LDAP auth, same outcome in both cases.

manics commented 1 month ago

Can you share the relevant bits of your audit.log log,output of audit2why and audit2allow, and your system logs?

warrenson commented 1 month ago

Sure. I have captured the output from the /var/log/audit.log for the single-server spawning event which fails. Then created the outputs of audit2why and audit2allow from that log. I've also included the preventing lines from /var/log/messages.

(The text files attached have the user and hostname redacted.)

SELinux_messages_preventing.txt audit.audit2allow.txt audit.log audit.audit2why.txt

manics commented 1 month ago

Jul 22 15:08:30 myhostname setroubleshoot[386683]: SELinux is preventing /usr/lib/systemd/systemd from setattr access on the file jupyter-myuser-singleuser.env.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that systemd should be allowed setattr access on the jupyter-myuser-singleuser.env file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c '(ngleuser)' --raw | audit2allow -M my-ngleuser#012# semodule -X 300 -i my-ngleuser.pp#012

type=AVC msg=audit(1721619379.891:6866): avc:  denied  { setattr } for  pid=388340 comm="(ngleuser)" name="jupyter-myuser-singleuser.env" dev="tmpfs" ino=8620 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=0

#============= init_t ==============
allow init_t var_run_t:file setattr;

This must be caused by the environment file that's written by the spawner: https://github.com/jupyterhub/systemdspawner/blob/f0018337cc1b22c991407cb9b07659b81d077b35/systemdspawner/systemd.py#L47-L66

If this is the only problem you could try following the suggestion to generate a custom policy module, though it's broader than ideal since it applies to all init_t processes, not just JupyterHub.

If you want to be really secure you could perhaps generate a custom policy (not something I've ever tried!) http://redhatgov.io/workshops/selinux_policy/ Or perhaps try asking on the Jupyter community forum.

warrenson commented 1 month ago

@manics thanks for the suggestions. I won't get a chance to try it out until next week, I'll report back then.

warrenson commented 1 month ago

I've applied the policy suggested by audit2allow, which seems to have worked fine. Thanks for the advice.

jupyterhub / systemdspawner