nix-daemon systemd unit security

[evrim]

Hello,

Are there any plans for 21.11 to secure nix-daemon unit? Current situation is as follows:

$ systemd-analyze security nix-daemon
  NAME                                                        DESCRIPTION                                                             EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                                     0.5
✗ User=/DynamicUser=                                          Service runs as root user                                                    0.4
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                           0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                         0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                                     0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                        0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                           0.3
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                          0.3
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unrestricted      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                         0.2
  NAME                                                        DESCRIPTION                                                             EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                                     0.5
✗ User=/DynamicUser=                                          Service runs as root user                                                    0.4
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                           0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                         0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                                     0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                        0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                           0.3
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                          0.3
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unrestricted      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                         0.2
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                                 0.2
✗ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service may load kernel modules                                              0.2
✗ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has raw I/O access                                                   0.2
✗ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes may change the system clock                                0.2
✗ DeviceAllow=                                                Service has no device ACL                                                    0.2
✗ IPAddressDeny=                                              Service does not define an IP address allow list                             0.2
✓ KeyringMode=                                                Service doesn't share key material with other services                          
✗ NoNewPrivileges=                                            Service processes may acquire new privileges                                 0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state                              
✗ PrivateDevices=                                             Service potentially has access to hardware devices                           0.2
✗ PrivateMounts=                                              Service may install system mounts                                            0.2
✗ PrivateTmp=                                                 Service has access to other software's temporary files                       0.2
✗ PrivateUsers=                                               Service has access to other users                                            0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                      0.2
✗ ProtectControlGroups=                                       Service may modify the control group file system                             0.2
✗ ProtectHome=                                                Service has full access to home directories                                  0.2
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer                 0.2
✗ ProtectKernelModules=                                       Service may load or read kernel modules                                      0.2
✗ ProtectKernelTunables=                                      Service may alter kernel tunables                                            0.2
✗ ProtectProc=                                                Service has full access to process tree (/proc hidepid=)                     0.2
✗ ProtectSystem=                                              Service has full access to the OS file hierarchy                             0.2
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                          0.2
✗ RestrictSUIDSGID=                                           Service may create SUID/SGID files                                           0.2
✗ SystemCallArchitectures=                                    Service may execute system calls with all ABIs                               0.2
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                         0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities                           
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                           0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                         0.1
✗ CapabilityBoundingSet=~CAP_MKNOD                            Service may create device nodes                                              0.1
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                                   0.1
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters                     0.1
✗ RestrictNamespaces=~CLONE_NEWCGROUP                         Service may create cgroup namespaces                                         0.1
✗ RestrictNamespaces=~CLONE_NEWIPC                            Service may create IPC namespaces                                            0.1
✗ RestrictNamespaces=~CLONE_NEWNET                            Service may create network namespaces                                        0.1
✗ RestrictNamespaces=~CLONE_NEWNS                             Service may create file system namespaces                                    0.1
✗ RestrictNamespaces=~CLONE_NEWPID                            Service may create process namespaces                                        0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                      0.1
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                         0.1
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                         0.1
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                         0.1
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                                0.1
  SupplementaryGroups=                                        Service runs as root, option does not matter                                    
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                                 0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                                   0.1
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree               
✗ LockPersonality=                                            Service may change ABI personality                                           0.1
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                       0.1
  RemoveIPC=                                                  Service runs as root, option does not apply                                     
✗ RestrictNamespaces=~CLONE_NEWUTS                            Service may create hostname namespaces                                       0.1
✗ UMask=                                                      Files created by service are world-readable by default                       0.1
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                             0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                             0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                                   0.1
✗ ProtectHostname=                                            Service may change system host/domainname                                    0.1
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                             0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                               0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                       0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                                  0.1
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                           0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                           0.1
✗ ProcSubset=                                                 Service has full access to non-process /proc files (/proc subset=)           0.1

→ Overall exposure level for nix-daemon.service: 9.6 UNSAFE 😨

nix-info:

$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 5.14.10, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11pre323079.2cf9db0e3d4"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[rnhmjoj]

This tool is pretty cool, but declaring a service "safe" because it uses a bunch of hardening options is quite a stretch. The nix-daemon needs root access for setting up build sandboxes, running builds as separate users and write access to /nix/store. Enabling some of these options and running it as an unprivileged user with capabilitiess is probably feasible, but it won't make it safer, imho. For example, if the nix-daemon is compromised, it can just hijack any binary, shell script or service inside the /nix/store and regain whatever level of privilege it lost due to the hardening

[evrim]

SystemD Documentation: https://www.freedesktop.org/software/systemd/man/systemd.exec.html

Currently, most of the services on NixOS are UNSAFE or EXPOSED. Safe in this context means score <4.0 (OK). (I couldn't find any doc on that, man doesn't say which score is OK, just says scores between 0 and 10, my system has a services with 4.0-OK). Anyway, nix-daemon is a single point of failure for all deployed instances, drawing a perimeter is a wise action. I believe its better to start hardening mostly used services.

[7c6f434c]

Anyway, nix-daemon is a single point of failure for all deployed instances, drawing a perimeter is a wise action. I believe its better to start hardening mostly used services.

What is the threat model here? The report doesn't even seem to care to distinguish protect-the-service and protect-against-the service, and a significant compromise of Nix daemon is a game over because of store access. Even if we fool naïve audit by tricks, Nix daemon can alter the installed glibc package. And data leaks from daemon do not mean that much as long as paths built are world-readable. Nix daemon launches things that should indeed be sandboxed (builds), but it sandboxes them itself.

[evrim]

@7c6f434c Raskin, you are right. In a normal distro, users wont be able install packages. Maybe it is possible start with implying those kind of restrictions. Another idea is restricting the packages that users can install. More, separating system packages from users, deploying user packages as separate. Nevertheless, these are for nix.

For NixOS, I believe, defining which Capabilities nix-daemon requires, which Socket Families it uses, what kind of Namespace functions it utilizes will carry us to the next step.

[7c6f434c]

For NixOS, I believe, defining which Capabilities nix-daemon requires, which Socket Families it uses, what kind of Namespace functions it utilizes will carry us to next step.

Next step towards what? What is the threat model? Nix bugs with direct consequences? Insufficient builder sandboxing that you propose to improve? Something else?

Note that service-level restrictions are also restricting all the builds Nix can ever run.

[evrim]

@7c6f434c Honestly, I am not sure, never met such an omnipotent daemon. Nevertheless, I believe it is possible to start with an adversary with rw access to nix-daemon.socket similar to docker.

I have doubts, is that true? I mean the capabilities of nix-daemon should be separate, no?

[7c6f434c]

@.*** Honestly, I am not sure, never met such an omnipotent daemon.

Doesn't any distribution have an option to install some updates automatically? Whatever daemon manages this, has the same issue of effective omnipotence. Which is indeed often abused in supply chain security attacks…

Nevertheless, I believe it is possible to start with an adversary with rw access to nix-daemon.socket similar to docker.

Presumably, if the attacker manages to create a socket of type Nix should not create, i.e. make a syscall with parameters that cannot be passed by simply taking a wrong branch, we have a code injection of some kind (which is most likely a game over), what am I missing here? But sure, taking wrong branch / opening file in a wrong way / something like that could happen, measures that protect about that could be of some use.

I have doubts, is that true? I mean the capabilities of nix-daemon should be separate, no?

Separate from what? It surely restricts what is available to builds, but as it spawns them they cannot have more than the daemon has. I guess you could fork Nix and make a daemon that writes to the store and the daemon that starts the builds separate processes with different sandboxing, but this is a lot of work and not on the level of the service file. Protocol audit and fuzzing could surely be useful in any case, no question there.

[evrim]

@7c6f434c

Doesn't any distribution have an option to install some updates automatically?

My prev experiences with auto-update daemons were terrible since config files are left intact, services being disrupted by updates. With nix, the experience is awesome. My point is, if you say because auto-update needs omnipotence, than it should be a separate service without any attack surfaces.

Presumably, if the attacker manages to create a socket of type Nix should not create, ...

I was thinking of someone fuzzing nix finding a type confusing etc doing ROP over nix-daemon.socket.

Separate from what? It surely restricts what is available to builds, but as it spawns them they cannot have more than the daemon has. I guess you could fork Nix and make a daemon that writes to the store and the daemon that starts the builds separate processes with different sandboxing, but this is a lot of work and not on the level of the service file. Protocol audit and fuzzing could surely be useful in any case, no question there.

This mite be interesting though it should be evident what we would gain if builder and system manager are separate entities.

Another point I am curious about is what are the proper auditd rules? It would be wonderful if we can build up a default auditd ruleset for nixos so that we can track whats going on on the system overall.

[7c6f434c]

Doesn't any distribution have an option to install some updates automatically?

My prev experiences with the auto-update daemons were terrible since config files are left intact, services being disrupted by updates. With nix, the experience is awesome. My point is, if you say because auto-update need omnipotence, than it should be a separate service without any attack surfaces.

Auto-update without any network attack surface sounds not very useful.

Nix is more than auto-update because it is a general package manager daemon allowing some user control. Which might have non-sudo implementations in other distributions, but I am not sure how they would be called. Something that would include PolicyKit and DBus, I guess, making sandboxing even more complicated.

Presumably, if the attacker manages to create a socket of type Nix should not create, ...

I was thinking of someone fuzzing nix finding a type confusing etc doing ROP over nix-daemon.socket.

Given how few sockets and how many files Nix opens, I would still expect some more file-based attack to be more likely to be found. Although I guess only a succesful fuzzing will tell…

Separate from what? It surely restricts what is available to builds, but as it spawns them they cannot have more than the daemon has. I guess you could fork Nix and make a daemon that writes to the store and the daemon that starts the builds separate processes with different sandboxing, but this is a lot of work and not on the level of the service file. Protocol audit and fuzzing could surely be useful in any case, no question there. This mite be interesting though it should be evident what we would gain if builder and system manager are separate entities.

Well, if you want systemd service sandboxing, then you want components with different activity and risk profile to be different services so they can be sandboxed according to the least authority sufficient to perform the duties…

Another point I am curious about is what are proper auditd rules? It would be wonderful if we can build up a default auditd ruleset for nixos so that we can track whats going on on the overall system.

[mohe2015]

I guess you could fork Nix and make a daemon that writes to the store and the daemon that starts the builds separate processes with different sandboxing

I don't even know if this would help much as attacking the build daemon itself probably could lead to full compromise and attacking the daemon that writes to the store would probably also lead to a full compromise.

Maybe it would reduce the attack surface minimally but I don't really know. Also as you said it would increase the complexity.

Actually I think this is a pretty hard problem as the attack surface seems comparably large. Fuzzing / security audits would probably be really worth it.

[7c6f434c]

I don't even know if this would help much as attacking the build daemon itself probably could lead to full compromise and attacking the daemon that writes to the store would probably also lead to a full compromise.

Well, if you split things, there is protocol handling daemon (exposed to untrusted input, needs version negotiation for upgrade path reasons, can do nothing on its own), store writing daemon (can in principle damage preexisting paths, has quite restricted protocol and no longer needs version negotiation, sandboxed from network stuff and such), build spawning daemon (at least no more version negotiation, can only write temporary things and not alter the past, maybe hashing the entire input of build-spawning daemon could be done by yet another very simple daemon to exclude wrongly-building-critical-paths without changing hash).

You probably gain something by sufficient amount of work. But it's quite a lot of work.

Maybe it would reduce the attack surface minimally but I don't really know. Also as you said it would increase the complexity.

Actually I think this is a pretty hard problem as the attack surface seems comparably large.