search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

Service beacons fails with Stacktraces #30117

Closed MadsRC closed 8 years ago

MadsRC commented 8 years ago

While trying to set up a service beacon and reading the documentation (https://docs.saltstack.com/en/latest/ref/beacons/all/salt.beacons.service.html) I noticed that the following beacon fails and leaves stack traces:

beacons:
  service:
    salt-minion:

Here's the salt-minion debug log:

[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt.REDACTED
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Minion "salt.REDACTED"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[INFO    ] The salt minion is starting up
[INFO    ] Minion is starting as user 'root'
[DEBUG   ] AsyncEventPublisher PUB socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pub.ipc
[DEBUG   ] AsyncEventPublisher PULL socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pull.ipc
[INFO    ] Starting pub socket on ipc:///var/run/salt/minion/minion_event_88c05f9029_pub.ipc
[INFO    ] Starting pull socket on ipc:///var/run/salt/minion/minion_event_88c05f9029_pull.ipc
[DEBUG   ] Minion 'salt.REDACTED' trying to tune in
[DEBUG   ] sync_connect_master
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (1185)
[DEBUG   ] Setting zmq_reconnect_ivl to '1185ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506', 'aes')
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] LazyLoaded timezone.get_offset
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command ['date', '+%z'] in directory '/root'
[DEBUG   ] output: -0500
[DEBUG   ] LazyLoaded config.merge
[DEBUG   ] LazyLoaded mine.update
[INFO    ] Added mine.update to scheduler
[INFO    ] Updating job settings for scheduled job: __mine_interval
[DEBUG   ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2016-01-01T19:04:54.865828', 'complete': True, 'schedule': {'__mine_interval': {'function': 'mine.update', 'jid_include': True, 'minutes': 60, 'maxrunning': 2}}}
[DEBUG   ] Persisting schedule
[DEBUG   ] Handling event '/salt/minion/minion_schedule_add_complete\n\n\x83\xa6_stamp\xba2016-01-01T19:04:54.865828\xa8complete\xc3\xa8schedule\x81\xaf__mine_interval\x84\xa8function\xabmine.update\xabjid_include\xc3\xa7minutes<\xaamaxrunning\x02'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506')
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506')
[INFO    ] Minion is ready to receive requests!
[DEBUG   ] LazyLoaded service.beacon
[DEBUG   ] Error loading module.dockerng: Docker module could not get imported
[DEBUG   ] LazyLoaded service.status
[INFO    ] Executing command 'systemctl --all --full --no-legend --no-pager list-units' in directory '/root'
[DEBUG   ] stdout: proc-sys-fs-binfmt_misc.automount                                                                              loaded    active   waiting   Arbitrary Executable File Formats File System Automount Point
dev-block-8:2.device                                                                                           loaded    active   plugged   LVM PV apg4aj-WUYz-GFdE-6tvu-o86V-rQXn-1b3ZGf on /dev/sda2 2
dev-cdrom.device                                                                                               loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
dev-centos-root.device                                                                                         loaded    active   plugged   /dev/centos/root
dev-centos-swap.device                                                                                         loaded    active   plugged   /dev/centos/swap
dev-disk-by\x2did-ata\x2dVMware_Virtual_IDE_CDROM_Drive_10000000000000000001.device                            loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
dev-disk-by\x2did-dm\x2dname\x2dcentos\x2droot.device                                                          loaded    active   plugged   /dev/disk/by-id/dm-name-centos-root
dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.device                                                          loaded    active   plugged   /dev/disk/by-id/dm-name-centos-swap
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2diTlJh7K9CuzdCqvXMCZfRv9i04mog5KXaPyz3lhHKAvAFbfsmTDrxcHxU0z1rNRW.device loaded    active   plugged   /dev/disk/by-id/dm-uuid-LVM-iTlJh7K9CuzdCqvXMCZfRv9i04mog5KXaPyz3lhHKAvAFbfsmTDrxcHxU0z1rNRW
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2diTlJh7K9CuzdCqvXMCZfRv9i04mog5KXcfs51q4gRnBBAV8X9KcrE8dldO2lP9IF.device loaded    active   plugged   /dev/disk/by-id/dm-uuid-LVM-iTlJh7K9CuzdCqvXMCZfRv9i04mog5KXcfs51q4gRnBBAV8X9KcrE8dldO2lP9IF
dev-disk-by\x2did-lvm\x2dpv\x2duuid\x2dapg4aj\x2dWUYz\x2dGFdE\x2d6tvu\x2do86V\x2drQXn\x2d1b3ZGf.device         loaded    active   plugged   LVM PV apg4aj-WUYz-GFdE-6tvu-o86V-rQXn-1b3ZGf on /dev/sda2 2
dev-disk-by\x2dlabel-CDROM.device                                                                              loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
dev-disk-by\x2dpath-pci\x2d0000:03:00.0\x2dscsi\x2d0:0:0:0.device                                              loaded    active   plugged   Virtual_disk
dev-disk-by\x2dpath-pci\x2d0000:03:00.0\x2dscsi\x2d0:0:0:0\x2dpart1.device                                     loaded    active   plugged   Virtual_disk 1
dev-disk-by\x2dpath-pci\x2d0000:03:00.0\x2dscsi\x2d0:0:0:0\x2dpart2.device                                     loaded    active   plugged   LVM PV apg4aj-WUYz-GFdE-6tvu-o86V-rQXn-1b3ZGf on /dev/sda2 2
dev-disk-by\x2duuid-2015\x2d12\x2d05\x2d06\x2d08\x2d32\x2d00.device                                            loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
dev-disk-by\x2duuid-b593b878\x2dc36b\x2d4778\x2d8258\x2d56251cca3188.device                                    loaded    active   plugged   /dev/disk/by-uuid/b593b878-c36b-4778-8258-56251cca3188
dev-disk-by\x2duuid-f977f94a\x2dfdd5\x2d4780\x2d95ce\x2dd25958cd7c26.device                                    loaded    active   plugged   Virtual_disk 1
dev-disk-by\x2duuid-fdafe805\x2d871b\x2d4828\x2da62f\x2df48aee25ca48.device                                    loaded    active   plugged   /dev/disk/by-uuid/fdafe805-871b-4828-a62f-f48aee25ca48
dev-dm\x2d0.device                                                                                             loaded    active   plugged   /dev/dm-0
dev-dm\x2d1.device                                                                                             loaded    active   plugged   /dev/dm-1
dev-fd0.device                                                                                                 loaded    active   plugged   /dev/fd0
dev-mapper-centos\x2droot.device                                                                               loaded    active   plugged   /dev/mapper/centos-root
dev-mapper-centos\x2dswap.device                                                                               loaded    active   plugged   /dev/mapper/centos-swap
dev-sda.device                                                                                                 loaded    active   plugged   Virtual_disk
dev-sda1.device                                                                                                loaded    active   plugged   Virtual_disk 1
dev-sda2.device                                                                                                loaded    active   plugged   LVM PV apg4aj-WUYz-GFdE-6tvu-o86V-rQXn-1b3ZGf on /dev/sda2 2
dev-sr0.device                                                                                                 loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
dev-ttyS0.device                                                                                               loaded    active   plugged   /dev/ttyS0
dev-ttyS1.device                                                                                               loaded    active   plugged   /dev/ttyS1
dev-ttyS2.device                                                                                               loaded    active   plugged   /dev/ttyS2
dev-ttyS3.device                                                                                               loaded    active   plugged   /dev/ttyS3
sys-devices-pci0000:00-0000:00:07.1-ata2-host2-target2:0:0-2:0:0:0-block-sr0.device                            loaded    active   plugged   VMware_Virtual_IDE_CDROM_Drive CDROM
sys-devices-pci0000:00-0000:00:11.0-0000:02:01.0-net-ens33.device                                              loaded    active   plugged   PRO/1000 MT Single Port Adapter
sys-devices-pci0000:00-0000:00:15.0-0000:03:00.0-host0-target0:0:0-0:0:0:0-block-sda-sda1.device               loaded    active   plugged   Virtual_disk 1
sys-devices-pci0000:00-0000:00:15.0-0000:03:00.0-host0-target0:0:0-0:0:0:0-block-sda-sda2.device               loaded    active   plugged   LVM PV apg4aj-WUYz-GFdE-6tvu-o86V-rQXn-1b3ZGf on /dev/sda2 2
sys-devices-pci0000:00-0000:00:15.0-0000:03:00.0-host0-target0:0:0-0:0:0:0-block-sda.device                    loaded    active   plugged   Virtual_disk
sys-devices-platform-floppy.0-block-fd0.device                                                                 loaded    active   plugged   /sys/devices/platform/floppy.0/block/fd0
sys-devices-platform-serial8250-tty-ttyS2.device                                                               loaded    active   plugged   /sys/devices/platform/serial8250/tty/ttyS2
sys-devices-platform-serial8250-tty-ttyS3.device                                                               loaded    active   plugged   /sys/devices/platform/serial8250/tty/ttyS3
sys-devices-pnp0-00:09-tty-ttyS0.device                                                                        loaded    active   plugged   /sys/devices/pnp0/00:09/tty/ttyS0
sys-devices-pnp0-00:0a-tty-ttyS1.device                                                                        loaded    active   plugged   /sys/devices/pnp0/00:0a/tty/ttyS1
sys-devices-virtual-block-dm\x2d0.device                                                                       loaded    active   plugged   /sys/devices/virtual/block/dm-0
sys-devices-virtual-block-dm\x2d1.device                                                                       loaded    active   plugged   /sys/devices/virtual/block/dm-1
sys-module-configfs.device                                                                                     loaded    active   plugged   /sys/module/configfs
sys-subsystem-net-devices-ens33.device                                                                         loaded    active   plugged   PRO/1000 MT Single Port Adapter
-.mount                                                                                                        loaded    active   mounted   /
boot.mount                                                                                                     loaded    active   mounted   /boot
dev-hugepages.mount                                                                                            loaded    active   mounted   Huge Pages File System
dev-mqueue.mount                                                                                               loaded    active   mounted   POSIX Message Queue File System
proc-sys-fs-binfmt_misc.mount                                                                                  loaded    inactive dead      Arbitrary Executable File Formats File System
run-user-1000.mount                                                                                            loaded    active   mounted   /run/user/1000
run-user-995.mount                                                                                             loaded    active   mounted   /run/user/995
sys-fs-fuse-connections.mount                                                                                  loaded    inactive dead      FUSE Control File System
sys-kernel-config.mount                                                                                        loaded    active   mounted   Configuration File System
sys-kernel-debug.mount                                                                                         loaded    active   mounted   Debug File System
brandbot.path                                                                                                  loaded    active   waiting   Flexible branding
systemd-ask-password-console.path                                                                              loaded    inactive dead      Dispatch Password Requests to Console Directory Watch
systemd-ask-password-plymouth.path                                                                             loaded    active   waiting   Forward Password Requests to Plymouth Directory Watch
systemd-ask-password-wall.path                                                                                 loaded    active   waiting   Forward Password Requests to Wall Directory Watch
session-380.scope                                                                                              loaded    active   running   Session 380 of user mrc
session-c1.scope                                                                                               loaded    active   abandoned Session c1 of user rabbitmq
auditd.service                                                                                                 loaded    active   running   Security Auditing Service
avahi-daemon.service                                                                                           loaded    active   running   Avahi mDNS/DNS-SD Stack
brandbot.service                                                                                               loaded    inactive dead      Flexible Branding Service
cpupower.service                                                                                               loaded    inactive dead      Configure CPU power related settings
crond.service                                                                                                  loaded    active   running   Command Scheduler
dbus.service                                                                                                   loaded    active   running   D-Bus System Message Bus
display-manager.service                                                                                        not-found inactive dead      display-manager.service
dm-event.service                                                                                               loaded    inactive dead      Device-mapper event daemon
dracut-shutdown.service                                                                                        loaded    inactive dead      Restore /run/initramfs
emergency.service                                                                                              loaded    inactive dead      Emergency Shell
exim.service                                                                                                   not-found inactive dead      exim.service
getty@tty1.service                                                                                             loaded    active   running   Getty on tty1
ip6tables.service                                                                                              loaded    active   exited    IPv6 firewall with ip6tables
iptables.service                                                                                               loaded    active   exited    IPv4 firewall with iptables
irqbalance.service                                                                                             loaded    inactive dead      irqbalance daemon
kdump.service                                                                                                  loaded    failed   failed    Crash recovery kernel arming
kmod-static-nodes.service                                                                                      loaded    active   exited    Create list of required static device nodes for the current kernel
ldconfig.service                                                                                               loaded    inactive dead      Rebuild Dynamic Linker Cache
lvm2-activation.service                                                                                        not-found inactive dead      lvm2-activation.service
lvm2-lvmetad.service                                                                                           loaded    active   running   LVM2 metadata daemon
lvm2-lvmpolld.service                                                                                          loaded    inactive dead      LVM2 poll daemon
lvm2-monitor.service                                                                                           loaded    active   exited    Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
lvm2-pvscan@8:2.service                                                                                        loaded    active   exited    LVM2 PV scan on device 8:2
microcode.service                                                                                              loaded    inactive dead      Load CPU microcode update
network.service                                                                                                loaded    active   exited    LSB: Bring up/down networking
NetworkManager-wait-online.service                                                                             loaded    inactive dead      Network Manager Wait Online
NetworkManager.service                                                                                         loaded    inactive dead      Network Manager
plymouth-quit-wait.service                                                                                     loaded    inactive dead      Wait for Plymouth Boot Screen to Quit
plymouth-quit.service                                                                                          loaded    inactive dead      Terminate Plymouth Boot Screen
plymouth-read-write.service                                                                                    loaded    inactive dead      Tell Plymouth To Write Out Runtime Data
plymouth-start.service                                                                                         loaded    inactive dead      Show Plymouth Boot Screen
polkit.service                                                                                                 loaded    active   running   Authorization Manager
postfix.service                                                                                                loaded    active   running   Postfix Mail Transport Agent
rabbitmq-server.service                                                                                        not-found failed   failed    rabbitmq-server.service
rc-local.service                                                                                               loaded    active   exited    /etc/rc.d/rc.local Compatibility
rescue.service                                                                                                 loaded    inactive dead      Rescue Shell
rhel-autorelabel-mark.service                                                                                  loaded    inactive dead      Mark the need to relabel after reboot
rhel-autorelabel.service                                                                                       loaded    inactive dead      Relabel all filesystems, if necessary
rhel-configure.service                                                                                         loaded    inactive dead      Reconfigure the system on administrator request
rhel-dmesg.service                                                                                             loaded    active   exited    Dump dmesg to /var/log/dmesg
rhel-import-state.service                                                                                      loaded    active   exited    Import network configuration from initramfs
rhel-loadmodules.service                                                                                       loaded    inactive dead      Load legacy module configuration
rhel-readonly.service                                                                                          loaded    active   exited    Configure read-only root support
rsyslog.service                                                                                                loaded    active   running   System Logging Service
salt-master.service                                                                                            loaded    active   running   The Salt Master Server
salt-minion.service                                                                                            loaded    inactive dead      The Salt Minion
sendmail.service                                                                                               not-found inactive dead      sendmail.service
sshd-keygen.service                                                                                            loaded    inactive dead      OpenSSH Server Key Generation
sshd.service                                                                                                   loaded    active   running   OpenSSH server daemon
syslog.service                                                                                                 not-found inactive dead      syslog.service
systemd-ask-password-console.service                                                                           loaded    inactive dead      Dispatch Password Requests to Console
systemd-ask-password-plymouth.service                                                                          loaded    inactive dead      Forward Password Requests to Plymouth
systemd-ask-password-wall.service                                                                              loaded    inactive dead      Forward Password Requests to Wall
systemd-binfmt.service                                                                                         loaded    inactive dead      Set Up Additional Binary Formats
systemd-firstboot.service                                                                                      loaded    inactive dead      First Boot Wizard
systemd-fsck-root.service                                                                                      loaded    active   exited    File System Check on Root Device
systemd-fsck@dev-disk-by\x2duuid-f977f94a\x2dfdd5\x2d4780\x2d95ce\x2dd25958cd7c26.service                      loaded    active   exited    File System Check on /dev/disk/by-uuid/f977f94a-fdd5-4780-95ce-d25958cd7c26
systemd-hwdb-update.service                                                                                    loaded    inactive dead      Rebuild Hardware Database
systemd-initctl.service                                                                                        loaded    inactive dead      /dev/initctl Compatibility Daemon
systemd-journal-catalog-update.service                                                                         loaded    inactive dead      Rebuild Journal Catalog
systemd-journal-flush.service                                                                                  loaded    active   exited    Flush Journal to Persistent Storage
systemd-journald.service                                                                                       loaded    active   running   Journal Service
systemd-logind.service                                                                                         loaded    active   running   Login Service
systemd-machine-id-commit.service                                                                              loaded    inactive dead      Commit a transient machine-id on disk
systemd-modules-load.service                                                                                   loaded    inactive dead      Load Kernel Modules
systemd-random-seed-load.service                                                                               not-found inactive dead      systemd-random-seed-load.service
systemd-random-seed.service                                                                                    loaded    active   exited    Load/Save Random Seed
systemd-readahead-collect.service                                                                              loaded    inactive dead      Collect Read-Ahead Data
systemd-readahead-done.service                                                                                 loaded    inactive dead      Stop Read-Ahead Data Collection
systemd-readahead-replay.service                                                                               loaded    inactive dead      Replay Read-Ahead Data
systemd-reboot.service                                                                                         loaded    inactive dead      Reboot
systemd-remount-fs.service                                                                                     loaded    active   exited    Remount Root and Kernel File Systems
systemd-shutdownd.service                                                                                      loaded    inactive dead      Delayed Shutdown Service
systemd-sysctl.service                                                                                         loaded    active   exited    Apply Kernel Variables
systemd-sysusers.service                                                                                       not-found inactive dead      systemd-sysusers.service
systemd-tmpfiles-clean.service                                                                                 loaded    inactive dead      Cleanup of Temporary Directories
systemd-tmpfiles-setup-dev.service                                                                             loaded    active   exited    Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service                                                                                 loaded    active   exited    Create Volatile Files and Directories
systemd-udev-trigger.service                                                                                   loaded    active   exited    udev Coldplug all Devices
systemd-udevd.service                                                                                          loaded    active   running   udev Kernel Device Manager
systemd-update-done.service                                                                                    loaded    inactive dead      Update is Completed
systemd-update-utmp-runlevel.service                                                                           loaded    inactive dead      Update UTMP about System Runlevel Changes
systemd-update-utmp.service                                                                                    loaded    active   exited    Update UTMP about System Boot/Shutdown
systemd-user-sessions.service                                                                                  loaded    active   exited    Permit User Sessions
systemd-vconsole-setup.service                                                                                 loaded    active   exited    Setup Virtual Console
tuned.service                                                                                                  loaded    active   running   Dynamic System Tuning Daemon
-.slice                                                                                                        loaded    active   active    Root Slice
system-getty.slice                                                                                             loaded    active   active    system-getty.slice
system-lvm2\x2dpvscan.slice                                                                                    loaded    inactive dead      system-lvm2\x2dpvscan.slice
system-systemd\x2dfsck.slice                                                                                   loaded    inactive dead      system-systemd\x2dfsck.slice
system.slice                                                                                                   loaded    active   active    System Slice
user-1000.slice                                                                                                loaded    active   active    user-1000.slice
user-995.slice                                                                                                 loaded    active   active    user-995.slice
user.slice                                                                                                     loaded    active   active    User and Session Slice
avahi-daemon.socket                                                                                            loaded    active   running   Avahi mDNS/DNS-SD Stack Activation Socket
dbus.socket                                                                                                    loaded    active   running   D-Bus System Message Bus Socket
dm-event.socket                                                                                                loaded    active   listening Device-mapper event daemon FIFOs
lvm2-lvmetad.socket                                                                                            loaded    active   running   LVM2 metadata daemon socket
lvm2-lvmpolld.socket                                                                                           loaded    active   listening LVM2 poll daemon socket
sshd.socket                                                                                                    loaded    inactive dead      OpenSSH Server Socket
syslog.socket                                                                                                  loaded    inactive dead      Syslog Socket
systemd-initctl.socket                                                                                         loaded    active   listening /dev/initctl Compatibility Named Pipe
systemd-journald.socket                                                                                        loaded    active   running   Journal Socket
systemd-shutdownd.socket                                                                                       loaded    active   listening Delayed Shutdown Socket
systemd-udevd-control.socket                                                                                   loaded    active   running   udev Control Socket
systemd-udevd-kernel.socket                                                                                    loaded    active   running   udev Kernel Socket
dev-centos-swap.swap                                                                                           loaded    active   active    /dev/centos/swap
dev-disk-by\x2did-dm\x2dname\x2dcentos\x2dswap.swap                                                            loaded    active   active    /dev/disk/by-id/dm-name-centos-swap
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2diTlJh7K9CuzdCqvXMCZfRv9i04mog5KXaPyz3lhHKAvAFbfsmTDrxcHxU0z1rNRW.swap   loaded    active   active    /dev/disk/by-id/dm-uuid-LVM-iTlJh7K9CuzdCqvXMCZfRv9i04mog5KXaPyz3lhHKAvAFbfsmTDrxcHxU0z1rNRW
dev-disk-by\x2duuid-b593b878\x2dc36b\x2d4778\x2d8258\x2d56251cca3188.swap                                      loaded    active   active    /dev/disk/by-uuid/b593b878-c36b-4778-8258-56251cca3188
dev-dm\x2d0.swap                                                                                               loaded    active   active    /dev/dm-0
dev-mapper-centos\x2dswap.swap                                                                                 loaded    active   active    /dev/mapper/centos-swap
basic.target                                                                                                   loaded    active   active    Basic System
cryptsetup.target                                                                                              loaded    active   active    Encrypted Volumes
emergency.target                                                                                               loaded    inactive dead      Emergency Mode
final.target                                                                                                   loaded    inactive dead      Final Step
getty.target                                                                                                   loaded    active   active    Login Prompts
graphical.target                                                                                               loaded    inactive dead      Graphical Interface
local-fs-pre.target                                                                                            loaded    active   active    Local File Systems (Pre)
local-fs.target                                                                                                loaded    active   active    Local File Systems
multi-user.target                                                                                              loaded    active   active    Multi-User System
network-online.target                                                                                          loaded    active   active    Network is Online
network-pre.target                                                                                             loaded    inactive dead      Network (Pre)
network.target                                                                                                 loaded    active   active    Network
nss-user-lookup.target                                                                                         loaded    inactive dead      User and Group Name Lookups
paths.target                                                                                                   loaded    active   active    Paths
remote-fs-pre.target                                                                                           loaded    inactive dead      Remote File Systems (Pre)
remote-fs.target                                                                                               loaded    active   active    Remote File Systems
rescue.target                                                                                                  loaded    inactive dead      Rescue Mode
shutdown.target                                                                                                loaded    inactive dead      Shutdown
slices.target                                                                                                  loaded    active   active    Slices
sockets.target                                                                                                 loaded    active   active    Sockets
swap.target                                                                                                    loaded    active   active    Swap
sysinit.target                                                                                                 loaded    active   active    System Initialization
syslog.target                                                                                                  not-found inactive dead      syslog.target
time-sync.target                                                                                               loaded    inactive dead      System Time Synchronized
timers.target                                                                                                  loaded    active   active    Timers
umount.target                                                                                                  loaded    inactive dead      Unmount All Filesystems
systemd-readahead-done.timer                                                                                   loaded    inactive dead      Stop Read-Ahead Data Collection 10s After Completed Startup
systemd-tmpfiles-clean.timer                                                                                   loaded    active   waiting   Daily Cleanup of Temporary Directories
[INFO    ] Executing command 'systemctl --full --no-legend --no-pager list-unit-files' in directory '/root'
[DEBUG   ] stdout: proc-sys-fs-binfmt_misc.automount      static  
dev-hugepages.mount                    static  
dev-mqueue.mount                       static  
proc-sys-fs-binfmt_misc.mount          static  
sys-fs-fuse-connections.mount          static  
sys-kernel-config.mount                static  
sys-kernel-debug.mount                 static  
tmp.mount                              disabled
brandbot.path                          disabled
systemd-ask-password-console.path      static  
systemd-ask-password-plymouth.path     static  
systemd-ask-password-wall.path         static  
session-107.scope                      static  
session-380.scope                      static  
session-84.scope                       static  
session-c1.scope                       static  
auditd.service                         enabled 
autovt@.service                        disabled
avahi-daemon.service                   enabled 
blk-availability.service               disabled
brandbot.service                       static  
console-getty.service                  disabled
console-shell.service                  disabled
container-getty@.service               static  
cpupower.service                       disabled
crond.service                          enabled 
dbus-org.freedesktop.Avahi.service     enabled 
dbus-org.freedesktop.hostname1.service static  
dbus-org.freedesktop.locale1.service   static  
dbus-org.freedesktop.login1.service    static  
dbus-org.freedesktop.machine1.service  static  
dbus-org.freedesktop.network1.service  invalid 
dbus-org.freedesktop.timedate1.service static  
dbus.service                           static  
debug-shell.service                    disabled
dm-event.service                       disabled
dnsmasq.service                        disabled
dracut-cmdline.service                 static  
dracut-initqueue.service               static  
dracut-mount.service                   static  
dracut-pre-mount.service               static  
dracut-pre-pivot.service               static  
dracut-pre-trigger.service             static  
dracut-pre-udev.service                static  
dracut-shutdown.service                static  
ebtables.service                       disabled
emergency.service                      static  
firewalld.service                      disabled
fstrim.service                         static  
getty@.service                         enabled 
halt-local.service                     static  
initrd-cleanup.service                 static  
initrd-parse-etc.service               static  
initrd-switch-root.service             static  
initrd-udevadm-cleanup-db.service      static  
ip6tables.service                      enabled 
iprdump.service                        disabled
iprinit.service                        disabled
iprupdate.service                      disabled
iptables.service                       enabled 
irqbalance.service                     enabled 
kdump.service                          enabled 
kmod-static-nodes.service              static  
ldconfig.service                       static  
lvm2-lvmetad.service                   disabled
lvm2-lvmpolld.service                  disabled
lvm2-monitor.service                   enabled 
lvm2-pvscan@.service                   static  
messagebus.service                     static  
microcode.service                      enabled 
NetworkManager-dispatcher.service      disabled
NetworkManager-wait-online.service     disabled
NetworkManager.service                 disabled
plymouth-halt.service                  disabled
plymouth-kexec.service                 disabled
plymouth-poweroff.service              disabled
plymouth-quit-wait.service             disabled
plymouth-quit.service                  disabled
plymouth-read-write.service            disabled
plymouth-reboot.service                disabled
plymouth-start.service                 disabled
plymouth-switch-root.service           static  
polkit.service                         static  
postfix.service                        enabled 
quotaon.service                        static  
rc-local.service                       static  
rdisc.service                          disabled
rescue.service                         static  
rhel-autorelabel-mark.service          static  
rhel-autorelabel.service               static  
rhel-configure.service                 static  
rhel-dmesg.service                     disabled
rhel-domainname.service                disabled
rhel-import-state.service              static  
rhel-loadmodules.service               static  
rhel-readonly.service                  static  
rsyncd.service                         disabled
rsyncd@.service                        static  
rsyslog.service                        enabled 
salt-master.service                    enabled 
salt-minion.service                    enabled 
serial-getty@.service                  disabled
sshd-keygen.service                    static  
sshd.service                           enabled 
sshd@.service                          static  
systemd-ask-password-console.service   static  
systemd-ask-password-plymouth.service  static  
systemd-ask-password-wall.service      static  
systemd-backlight@.service             static  
systemd-binfmt.service                 static  
systemd-bootchart.service              disabled
systemd-firstboot.service              static  
systemd-fsck-root.service              static  
systemd-fsck@.service                  static  
systemd-halt.service                   static  
systemd-hibernate-resume@.service      static  
systemd-hibernate.service              static  
systemd-hostnamed.service              static  
systemd-hwdb-update.service            static  
systemd-hybrid-sleep.service           static  
systemd-initctl.service                static  
systemd-journal-catalog-update.service static  
systemd-journal-flush.service          static  
systemd-journald.service               static  
systemd-kexec.service                  static  
systemd-localed.service                static  
systemd-logind.service                 static  
systemd-machine-id-commit.service      static  
systemd-machined.service               static  
systemd-modules-load.service           static  
systemd-nspawn@.service                disabled
systemd-poweroff.service               static  
systemd-quotacheck.service             static  
systemd-random-seed.service            static  
systemd-readahead-collect.service      enabled 
systemd-readahead-done.service         static  
systemd-readahead-drop.service         enabled 
systemd-readahead-replay.service       enabled 
systemd-reboot.service                 static  
systemd-remount-fs.service             static  
systemd-rfkill@.service                static  
systemd-shutdownd.service              static  
systemd-suspend.service                static  
systemd-sysctl.service                 static  
systemd-timedated.service              static  
systemd-tmpfiles-clean.service         static  
systemd-tmpfiles-setup-dev.service     static  
systemd-tmpfiles-setup.service         static  
systemd-udev-settle.service            static  
systemd-udev-trigger.service           static  
systemd-udevd.service                  static  
systemd-update-done.service            static  
systemd-update-utmp-runlevel.service   static  
systemd-update-utmp.service            static  
systemd-user-sessions.service          static  
systemd-vconsole-setup.service         static  
tcsd.service                           disabled
teamd@.service                         static  
tuned.service                          enabled 
wpa_supplicant.service                 disabled
-.slice                                static  
machine.slice                          static  
system.slice                           static  
user.slice                             static  
avahi-daemon.socket                    enabled 
dbus.socket                            static  
dm-event.socket                        enabled 
lvm2-lvmetad.socket                    enabled 
lvm2-lvmpolld.socket                   enabled 
rsyncd.socket                          disabled
sshd.socket                            disabled
syslog.socket                          static  
systemd-initctl.socket                 static  
systemd-journald.socket                static  
systemd-networkd.socket                disabled
systemd-shutdownd.socket               static  
systemd-udevd-control.socket           static  
systemd-udevd-kernel.socket            static  
basic.target                           static  
bluetooth.target                       static  
cryptsetup-pre.target                  static  
cryptsetup.target                      static  
ctrl-alt-del.target                    disabled
default.target                         enabled 
emergency.target                       static  
final.target                           static  
getty.target                           static  
graphical.target                       static  
halt.target                            disabled
hibernate.target                       static  
hybrid-sleep.target                    static  
initrd-fs.target                       static  
initrd-root-fs.target                  static  
initrd-switch-root.target              static  
initrd.target                          static  
iprutils.target                        disabled
kexec.target                           disabled
local-fs-pre.target                    static  
local-fs.target                        static  
machines.target                        disabled
multi-user.target                      enabled 
network-online.target                  static  
network-pre.target                     static  
network.target                         static  
nss-lookup.target                      static  
nss-user-lookup.target                 static  
paths.target                           static  
poweroff.target                        disabled
printer.target                         static  
reboot.target                          disabled
remote-fs-pre.target                   static  
remote-fs.target                       enabled 
rescue.target                          disabled
rpcbind.target                         static  
runlevel0.target                       disabled
runlevel1.target                       disabled
runlevel2.target                       static  
runlevel3.target                       static  
runlevel4.target                       static  
runlevel5.target                       static  
runlevel6.target                       disabled
shutdown.target                        static  
sigpwr.target                          static  
sleep.target                           static  
slices.target                          static  
smartcard.target                       static  
sockets.target                         static  
sound.target                           static  
suspend.target                         static  
swap.target                            static  
sysinit.target                         static  
system-update.target                   static  
time-sync.target                       static  
timers.target                          static  
umount.target                          static  
fstrim.timer                           disabled
systemd-readahead-done.timer           static  
systemd-tmpfiles-clean.timer           static
[INFO    ] Legacy init script: "README".
[INFO    ] Legacy init script: "functions".
[INFO    ] Legacy init script: "netconsole".
[INFO    ] Legacy init script: "network".
[INFO    ] Executing command 'systemctl status salt-minion.service' in directory '/root'
[ERROR   ] Command 'systemctl status salt-minion.service' failed with return code: 3
[ERROR   ] output: * salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2016-01-01 14:03:44 EST; 1min 12s ago
  Process: 19189 ExecStart=/usr/bin/salt-minion (code=killed, signal=TERM)
 Main PID: 19189 (code=killed, signal=TERM)
   CGroup: /system.slice/salt-minion.service

Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:51 salt.REDACTED salt-minion[19189]: [ERROR   ] Command 'systemctl status firewalld.service' failed with return code: 3
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:51 salt.REDACTED salt-minion[19189]: [ERROR   ] output: * firewalld.service - firewalld - dynamic firewall daemon
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:51 salt.REDACTED salt-minion[19189]: Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:51 salt.REDACTED salt-minion[19189]: Active: inactive (dead)
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:53 salt.REDACTED salt-minion[19189]: [ERROR   ] Command 'systemctl status NetworkManager.service' failed with return code: 3
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:53 salt.REDACTED salt-minion[19189]: [ERROR   ] output: * NetworkManager.service - Network Manager
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:53 salt.REDACTED salt-minion[19189]: Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; vendor preset: enabled)
Jan 01 14:03:45 salt.REDACTED salt-minion[19189]: Jan 01 14:00:53 salt.REDACTED salt-minion[19189]: Active: inactive (dead)
Jan 01 14:03:46 salt.REDACTED salt-minion[19189]: Jan 01 14:03:44 salt.REDACTED systemd[1]: Stopping The Salt Minion...
Jan 01 14:03:46 salt.REDACTED salt-minion[19189]: Jan 01 14:03:44 salt.REDACTED systemd[1]: Stopped The Salt Minion.
[INFO    ] Executing command 'systemctl is-active salt-minion.service' in directory '/root'
[DEBUG   ] output: inactive
[CRITICAL] The beacon errored: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1699, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 359, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/service.py", line 78, in beacon
    if 'uncleanshutdown' in config[service] and not ret_dict[service]['running']:
TypeError: argument of type 'NoneType' is not iterable
[ERROR   ] Exception in callback <function handle_beacons at 0x249b488>
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 1019, in _run
    return self.callback()
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1702, in handle_beacons
    if beacons:
UnboundLocalError: local variable 'beacons' referenced before assignment
[INFO    ] Running scheduled job: __mine_interval
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: This job was scheduled with a max number of 2
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun': 'mine.update', 'jid': '20160101140456763224', 'pid': 21858, 'id': 'salt.REDACTED', 'schedule': '__mine_interval'}
[DEBUG   ] MinionEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pub.ipc
[DEBUG   ] MinionEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_88c05f9029_pull.ipc
[DEBUG   ] Sending event - data = {'clear': False, 'cmd': '_mine', 'data': {}, 'id': 'salt.REDACTED', '_stamp': '2016-01-01T19:04:56.840521'}
[DEBUG   ] Handling event '_minion_mine\n\n\x85\xa5clear\xc2\xa3cmd\xa5_mine\xa4data\x80\xa2id\xabsalt.REDACTED\xa6_stamp\xba2016-01-01T19:04:56.840521'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'salt.REDACTED', 'tcp://REDACTED.IP.WAS.HERE:4506')

System Info: CentOS 7 Salt Version 2015.8.3 from SaltStack's own repo

MadsRC commented 8 years ago

Reference: https://github.com/saltstack/salt/blob/develop/salt/beacons/service.py

Looks like it fails because the code, contrary to what the documentation says, expects 'onchangeonly' and 'uncleanshutdown' to be present. While simply setting 'onchangeonly' to False would work (it's implied that False is the default value, which it's not) it won't work for 'uncleanshutdown' as it expects a path to a PID file.

I will test it further, consume a tad more RedBull, and hopefully have a patch ready later tonight.

cachedout commented 8 years ago

@MadsRC Will this be resolved once #30121 is merged?

MadsRC commented 8 years ago

@cachedout PR #30121 will solve the service beacon problem atleast on my CentOS 7 servers. If you'd like, I can conduct a few more tests on more systems?

cachedout commented 8 years ago

@MadsRC Sure, more testing is always welcome!

MadsRC commented 8 years ago

I'm not able to reproduce the issue after implementing my patch, and I can't find any regression bugs either. So I think you can consider this issue closed as per #30121

Seems like I forgot to make the PR against the 2015.8.3 branch (as it was a bugfix), do you want me to back-port it so it can be part of the next release?

basepi commented 8 years ago

I'll mark it for backporting. Thanks @MadsRC !