search

desbma / shh

Systemd Hardening Helper
GNU General Public License v3.0
13 stars 0 forks source link

Detect mount point propagation #6

Closed Jeroen0494 closed 1 week ago

Jeroen0494 commented 1 month ago

Hi,

First of all, I really like this tool. It allows me to get a basic set of hardening options for services.

One point I run into is that this tool recommends settings like ProtectSystem for services that require mount point propagation.

From the systemd.exec manual:

Note that these settings will disconnect propagation of mounts from the unit's processes to the host. This means that this setting may not be used for services which shall be able to install mount points in the main mount namespace.

For example, the smartmontools service. This is what shh recommends:

INFO  [shh] Detected versions: Systemd 249.11, Linux kernel 6.5.0, strace 6.9
INFO  [shh::systemd::service] stop smartmontools.service
INFO  [shh::systemd::service] "/run/systemd/system/smartmontools.service.d/zz_shh-profile.conf" removed
INFO  [shh] Resolved systemd options: ProtectSystem=strict, ProtectHome=tmpfs, PrivateTmp=true, ProtectKernelTunables=true, ProtectKernelModules=true, ProtectKernelLogs=true, ProtectControlGroups=true, ProtectProc=ptraceable, MemoryDenyWriteExecute=true, RestrictAddressFamilies=AF_UNIX, SocketBindDeny=ipv4:tcp, SocketBindDeny=ipv4:udp, SocketBindDeny=ipv6:tcp, SocketBindDeny=ipv6:udp, LockPersonality=true, RestrictRealtime=true, ProtectClock=true, SystemCallFilter=~@aio:EPERM @chown:EPERM @clock:EPERM @cpu-emulation:EPERM @debug:EPERM @io-event:EPERM @ipc:EPERM @keyring:EPERM @memlock:EPERM @module:EPERM @mount:EPERM @obsolete:EPERM @pkey:EPERM @privileged:EPERM @raw-io:EPERM @reboot:EPERM @resources:EPERM @sandbox:EPERM @setuid:EPERM @swap:EPERM @sync:EPERM @timer:EPERM
INFO  [shh::systemd::service] start smartmontools.service

However, when applying this the service won't start. All of the options that require setting up a mount namespace don't work (all of the ProtectX options).

Separate from this, all of the :EPERM options don't work and need to be removed (Ubuntu 22.04).

desbma commented 1 month ago

I can't reproduce this, smartd.service (for some reason Debian has renamed the service smartmontools but it seem to be the same thing) starts successfully with ProtectSystem=strict.

Can you provide more details about what fails when it restarts (journalctl log)?

Jeroen0494 commented 1 month ago

Thanks for the quick response. There is some output:

jeroen@jeroen-XPS-13-9370:~$ sudo systemctl restart smartmontools.service
Job for smartmontools.service failed because a timeout was exceeded.
See "systemctl status smartmontools.service" and "journalctl -xeu smartmontools.service" for details.
jeroen@jeroen-XPS-13-9370:~$ systemctl status smartmontools.service
× smartmontools.service - Self Monitoring and Reporting Technology (SMART) Daemon
     Loaded: loaded (/lib/systemd/system/smartmontools.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/smartmontools.service.d
             └─override.conf
     Active: failed (Result: timeout) since Sun 2024-07-14 22:27:09 CEST; 23s ago
       Docs: man:smartd(8)
             man:smartd.conf(5)
    Process: 95872 ExecStart=/usr/sbin/smartd -n $smartd_opts (code=exited, status=0/SUCCESS)
   Main PID: 95872 (code=exited, status=0/SUCCESS)
         IP: 0B in, 0B out
        CPU: 105ms

jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Starting Self Monitoring and Reporting Technology (SMART) Daemon...
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: start operation timed out. Terminating.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Failed with result 'timeout'.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.
jeroen@jeroen-XPS-13-9370:~$ systemctl status smartmontools.service
× smartmontools.service - Self Monitoring and Reporting Technology (SMART) Daemon
     Loaded: loaded (/lib/systemd/system/smartmontools.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/smartmontools.service.d
             └─override.conf
     Active: failed (Result: timeout) since Sun 2024-07-14 22:27:09 CEST; 2min 12s ago
       Docs: man:smartd(8)
             man:smartd.conf(5)
    Process: 95872 ExecStart=/usr/sbin/smartd -n $smartd_opts (code=exited, status=0/SUCCESS)
   Main PID: 95872 (code=exited, status=0/SUCCESS)
         IP: 0B in, 0B out
        CPU: 105ms

jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Starting Self Monitoring and Reporting Technology (SMART) Daemon...
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: start operation timed out. Terminating.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Failed with result 'timeout'.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.
jeroen@jeroen-XPS-13-9370:~$ journalctl -xeu smartmontools.service
jul 14 21:32:23 jeroen-XPS-13-9370 smartd[88637]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
jul 14 21:32:23 jeroen-XPS-13-9370 smartd[88637]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
jul 14 21:32:23 jeroen-XPS-13-9370 smartd[88637]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
jul 14 21:32:23 jeroen-XPS-13-9370 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
░░ Subject: A start job for unit smartmontools.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit smartmontools.service has finished successfully.
░░ 
░░ The job identifier is 13209.
jul 14 22:10:04 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:11:55 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:15:42 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:17:32 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:17:49 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:18:21 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:20:30 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:21:17 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:21:54 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:24:17 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:25:35 jeroen-XPS-13-9370 systemd[1]: /etc/systemd/system/smartmontools.service.d/override.conf:5: Unknown system call group, ignoring: @sandbox
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
░░ Subject: A stop job for unit smartmontools.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit smartmontools.service has begun execution.
░░ 
░░ The job identifier is 16357.
jul 14 22:25:38 jeroen-XPS-13-9370 smartd[88637]: smartd received signal 15: Terminated
jul 14 22:25:38 jeroen-XPS-13-9370 smartd[88637]: smartd is exiting (exit status 0)
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit smartmontools.service has successfully entered the 'dead' state.
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Stopped Self Monitoring and Reporting Technology (SMART) Daemon.
░░ Subject: A stop job for unit smartmontools.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit smartmontools.service has finished.
░░ 
░░ The job identifier is 16357 and the job result is done.
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Starting Self Monitoring and Reporting Technology (SMART) Daemon...
░░ Subject: A start job for unit smartmontools.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit smartmontools.service has begun execution.
░░ 
░░ The job identifier is 16357.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: start operation timed out. Terminating.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit smartmontools.service has entered the 'failed' state with result 'timeout'.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.
░░ Subject: A start job for unit smartmontools.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit smartmontools.service has finished with a failure.
░░ 
░░ The job identifier is 16357 and the job result is failed.
jeroen@jeroen-XPS-13-9370:~$ sudo journalctl -ru smartmontools
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Failed with result 'timeout'.
jul 14 22:27:09 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: start operation timed out. Terminating.
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Starting Self Monitoring and Reporting Technology (SMART) Daemon...
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Stopped Self Monitoring and Reporting Technology (SMART) Daemon.
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Deactivated successfully.
jul 14 22:25:38 jeroen-XPS-13-9370 smartd[88637]: smartd is exiting (exit status 0)
jul 14 22:25:38 jeroen-XPS-13-9370 smartd[88637]: smartd received signal 15: Terminated
jul 14 22:25:38 jeroen-XPS-13-9370 systemd[1]: Stopping Self Monitoring and Reporting Technology (SMART) Daemon...
jeroen@jeroen-XPS-13-9370:~$ sudo cat /etc/systemd/system/smartmontools.service.d/override.conf
[Service]
NoNewPrivileges=yes
RestrictSUIDSGID=yes
LockPersonality=yes
SystemCallFilter=~@aio @chown @clock @cpu-emulation @debug @io-event @ipc @keyring @memlock @module @mount @obsolete @pkey @reboot @resources @sandbox @setuid @swap @sync @timer
ProtectClock=yes
#ProtectKernelLogs=yes
#ProtectKernelModules=yes
#ProtectKernelTunables=yes
#ProtectProc=invisible

ProtectSystem=strict
#ProtectHome=yes

MemoryDenyWriteExecute=true

PrivateNetwork=yes
RestrictAddressFamilies=AF_UNIX
SocketBindDeny=any
desbma commented 1 month ago

smartmontools.service: start operation timed out. Terminating.

Did you try increasing the start timeout (TimeoutStart=)?

If you leave ProtectSystem=strict, and comment out all other hardening options, does it still fail?

What makes you think the issue is related to mount propagation? If that was the case, the service would not fail. You would however miss some mount points in the root namespace. By the way, why would smartmontools add mount points? Isn't it just to run periodic SMART tests?

Jeroen0494 commented 1 month ago

When not using the ProtectSystem=strict option, the service restart immediately. When using the option (and only this option for testing), the service gives a timeout on restart.

I assumed the issue is mount propagation, because that's what the systemd.exec guide says is incompatible. No other knowledge.

desbma commented 1 month ago

To help me investigate, can you please:

  1. Remove all non default hardening in /etc/systemd/system/smartmontools.service.d/*
  2. Create a file /etc/systemd/system/smartmontools.service.d/log.conf, with: 
    [Service] 
ExecStart=
ExecStart=/usr/bin/shh run -l /run/strace.log -- /usr/sbin/smartd -n $smartd_opts
NotifyAccess=all
KillMode=control-group
  3. Reload config and restart service: 
    systemctl daemon-reload && systemctl restart smartmontools
  4. Once it has started, stop the service with systemctl stop smartmontools, and post here the content of /run/strace.log (log for case that works) and remove the file.
  5. Create a file /etc/systemd/system/smartmontools.service.d/hardening.conf, with: 
    [Service]  
ProtectSystem=strict
ReadWritePaths=/run/
  6. Reload config and restart service (which should fail): 
    systemctl daemon-reload && systemctl restart smartmontools
  7. Once the timeout error has triggered, post here the content of /run/strace.log (log for case that fails).
desbma commented 1 month ago

Additionally, you can try adding ProtectSystem=true and then ProtectSystem=full as the only additional hardening option to see if it still fails, that would give me some hints.

Jeroen0494 commented 1 month ago

There is an error in your example.

jul 16 21:34:42 jeroen-XPS-13-9370 systemd[1]: Starting Self Monitoring and Reporting Technology (SMART) Daemon...
jul 16 21:34:42 jeroen-XPS-13-9370 shh[65954]: INFO  [shh] Detected versions: Systemd 249.11, Linux kernel 6.5.0, strace 6.9
jul 16 21:34:42 jeroen-XPS-13-9370 shh[65954]: error: unexpected argument '-l' found
jul 16 21:34:42 jeroen-XPS-13-9370 shh[65954]:   tip: to pass '-l' as a value, use '-- -l'
jul 16 21:34:42 jeroen-XPS-13-9370 shh[65954]: Usage: shh run [OPTIONS] <COMMAND>...
jul 16 21:34:42 jeroen-XPS-13-9370 shh[65954]: For more information, try '--help'.
jul 16 21:34:42 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
jul 16 21:34:42 jeroen-XPS-13-9370 systemd[1]: smartmontools.service: Failed with result 'exit-code'.
jul 16 21:34:42 jeroen-XPS-13-9370 systemd[1]: Failed to start Self Monitoring and Reporting Technology (SMART) Daemon.

Am I running another version of shh?

# dpkg -l | grep shh
ii  shh                                              2024.4.5-1                                                   amd64        Automatic systemd service hardening guided by strace profiling
desbma commented 1 month ago

The strace parser was completely rewritten between versions 2024.4.5 and 2024.6.4, so maybe check if the issue is still here with 2024.6.4 before doing any more tests.

Jeroen0494 commented 1 month ago

Error is still there.

Error: Read-only file system (os error 30) at path "/tmp/.tmp57QAmq" Had to add PrivateTmp=true for the service to start.

Log before hardening: strace.log

Log after hardening: strace-hardened.log

desbma commented 1 month ago

Does that fail with ProtectSystem=true or ProtectSystem=full, instead of ProtectSystem=strict?

Jeroen0494 commented 1 month ago

When using ProtectSystem=true the service also gives a timeout. So with true, /usr and /boot and /efi are read-only. Let's try something out:

ReadOnlyPaths="/boot"

Result: timeout

ReadOnlyPaths="/usr"

Result: timeout

ReadOnlyPaths creates a filesystem namespace, maybe smartmontools doesn't like this? Let's try a bind mount instead:

BindReadOnlyPaths="/boot"

Result: timeout

Hmm, weird.

desbma commented 1 month ago

By looking at your two logs they seem to start differing when smartd sends a notification to systemd through the socket in /run/systemd/notify. The logs only contains the successful syscalls, and the sendmsg call for the socket is missing in the second log. That explains the service timeout: systemd waits for a notification that never comes and ends up killing the service process.

If you try Type=simple instead of Type=notify, does the service start and keep running?

Jeroen0494 commented 1 month ago

Setting type to simple seems to work, the service works now and the process keeps running. Any idea why the message never get's through / send?

desbma commented 1 month ago

I had a look both at the systemd and smartmontools source code, no luck so far. To understand: first is the systemd socket reachable from the service namespace?

You can add ProtectSystem=strict and Type=notify again to reproduce the error, but also add ExecStartPre=/usr/bin/stat /run/systemd/notify. When restarting the service, does stat return an error?

Jeroen0494 commented 1 month ago

It's almost 23:00 where I live, I'll try more options tomorrow. Thanks so far.

desbma commented 1 month ago

I'm on the same timezone. :wink: I have other ideas of things to look at to investigate further. To be continued

desbma commented 1 week ago

Any update on this?

Jeroen0494 commented 1 week ago

Sorry, I'm currently unable to work on this.

desbma commented 1 week ago

Since I can't reproduce this issue, and I unable to investigate further without an answer to https://github.com/desbma/shh/issues/6#issuecomment-2231789899, I am closing this issue.

If you have time in the future to spend on this, feel free to comment here and I will reopen it.