pyinfra may run a filesystem check that triggers a machine reboot

[julienlavergne]

Describe the bug

While running this task:

server.service(
        "sshd",
        restarted=True,
        _sudo=True,
    )

the following command may be run on target host : sudo /etc/rc.d/rc.sysinit check, which triggers a host to reboot in order to perform a filesystem check, which cannot be performed on mounted devices.

To Reproduce

I am not so sure what kind of host you need that, But I think it has to be a bit old, typically RHEL 6.

Task is trying to use systemctl, rc-service and initctl which are all not available. To get a service status on this machine, the required command is sudo service sshd status.

The relevant log with --debug and -vvv:

[host03] >>> sudo -H -n sh -c 'which systemctl || true'
[host03] which: no systemctl in ($PATH)
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [host03] Loaded fact server.Which (command=systemctl)
    [pyinfra.api.facts] Getting fact: server.Which (command=rc-service) (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on host03: (pty=None) sudo -H -n sh -c 'which rc-service || true'
[host03] >>> sudo -H -n sh -c 'which rc-service || true'
[host03] which: no rc-service in ($PATH)
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [host03] Loaded fact server.Which (command=rc-service)
    [pyinfra.api.facts] Getting fact: server.Which (command=initctl) (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on host03: (pty=None) sudo -H -n sh -c 'which initctl || true'
[host03] >>> sudo -H -n sh -c 'which initctl || true'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] which: no initctl in ($PATH)
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] >>> sudo -H -n sh -c '! (test -e /etc/init.d || test -L /etc/init.d ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/init.d 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /etc/init.d )'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] user=root group=root mode=lrwxrwxrwx atime=1653225241 mtime=1568120828 ctime=1568120828 size=11 `/etc/init.d' -> `rc.d/init.d'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [host03] Loaded fact files.Directory (path=/etc/init.d)
    [pyinfra.api.facts] Getting fact: files.Directory (path=/etc/rc.d) (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on host03: (pty=None) sudo -H -n sh -c '! (test -e /etc/rc.d || test -L /etc/rc.d ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/rc.d 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /etc/rc.d )'
[host03] >>> sudo -H -n sh -c '! (test -e /etc/rc.d || test -L /etc/rc.d ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/rc.d 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /etc/rc.d )'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] user=root group=root mode=drwxr-xr-x atime=1653271634 mtime=1578644701 ctime=1578644701 size=4096 `/etc/rc.d'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [host03] Loaded fact files.Directory (path=/etc/rc.d)
    [pyinfra.api.facts] Getting fact: server.Os () (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on host03: (pty=None) sudo -H -n sh -c 'uname -s'
[host03] >>> sudo -H -n sh -c 'uname -s'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] Linux
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [host03] Loaded fact server.Os
    [pyinfra.api.facts] Getting fact: bsdinit.RcdStatus () (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on host03: (pty=None) sudo -H -n sh -c '
        for SERVICE in `find /etc/rc.d /usr/local/etc/rc.d -type f`; do
            $SERVICE status 2> /dev/null || $SERVICE check 2> /dev/null
            echo "`basename $SERVICE`=$?"
        done
    '
[host03] >>> sudo -H -n sh -c '
        for SERVICE in `find /etc/rc.d /usr/local/etc/rc.d -type f`; do
            $SERVICE status 2> /dev/null || $SERVICE check 2> /dev/null
            echo "`basename $SERVICE`=$?"
        done
    '
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] find: `/usr/local/etc/rc.d': No such file or directory
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] abrt-dump-oops is stopped
[host03] Usage: /etc/rc.d/init.d/abrt-oops {start|stop|status|restart|condrestart|reload|force-reload}
[host03] abrt-oops=0
[host03] dmeventd is stopped
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
[host03] Usage: /etc/rc.d/init.d/lvm2-monitor {start|stop|restart|status|force-stop}
...
[host03] saslauthd is stopped
[host03] Usage: /etc/rc.d/init.d/saslauthd {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
[host03] saslauthd=0
[host03] Entering non-interactive startup
[host03] rc=0
[host03]                 Welcome to Red Hat Enterprise Linux Server
[host03] Setting hostname host03:  [  OK  ]
[host03] Setting up Logical Volume Management:   2 logical volume(s) in volume group "host03" now active
[host03]   3 logical volume(s) in volume group "vg0" now active
[host03] [  OK  ]
[host03] Checking filesystems
[host03] Checking all file systems.
[host03] [/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/mapper/vg0-root
[host03] /dev/mapper/vg0-root is mounted.
[host03] [FAILED]
[host03]
[host03] *** An error occurred during the file system check.
[host03] *** Dropping you to a shell; the system will reboot
[host03] *** when you leave the shell.
[host03] Unmounting file systems
[host03] Automatic reboot in progress.

Expected behavior

I do not whish any command run by pyinfra to have the side effect of rebooting the machine without my consent.

Meta

v2.1

[sysadmin75]

What OS is host03?

[julienlavergne]

RHEL 6

[Fizzadar]

Wondering what the best fix for this is - the BSD service checks use:

for SERVICE in `find /etc/rc.d /usr/local/etc/rc.d -type f`; do
    $SERVICE status 2> /dev/null || $SERVICE check 2> /dev/null
    echo "`basename $SERVICE`=$?"
done

The server.service information will run through available init systems and RHEL6 is sysvinit based nothing newer (IIRC). I'm not sure why this would be executed though as /etc/init.d/ should exist on a RHEL 6 system? Which means it should match this check.

One option could be to check for "check" or "status" existing within the file, but that also feels very hacky.

A quick solution here is probably to make the above referenced check also check the OS is not Linux, because /etc/rc.d is dangerous to look through on Linux machines.

[sysadmin75]

Good question. What's wrong with doing something like this?

    elif host.get_fact(Which, command="service"):
        service_operation = sysvinit.service

Otherwise, I don't see where command is set to the correct command.

[Fizzadar]

I’ve expanded the check in https://github.com/Fizzadar/pyinfra/commit/1017a618a09d267234fee353c00586d59f39314b - relying on the service command alone could miss systems without it. I have also explicitly blocked loading the bsd init fact in https://github.com/Fizzadar/pyinfra/commit/fb5edc89af863677c326bfc5264dd7ec7164f531.