search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.1k stars 5.47k forks source link

service.dead no longer works after systemd update #38842

Closed roskens closed 6 years ago

roskens commented 7 years ago

Description of Issue/Question

systemctl status returns different output/error codes after 219-23 update. This causes service.dead to produce a failure when it should pass.

Setup

test-case.sls:

non-existing-service:
  service.dead:
    - name: non-existing-service
    - enable: False

Steps to Reproduce Issue

[root@salt-master base]# salt $MINIONID state.sls  test-case
$MINIONID:
----------
          ID: non-existing-service
    Function: service.dead
      Result: False
     Comment: Failed to get information on unit 'non-existing-service'
     Started: 10:31:51.660029
    Duration: 26.562 ms
     Changes:

Summary for $MINIONID
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:  26.562 ms
ERROR: Minions returned with non-zero exit code

From the minions log file (in debug mode:)

2017-01-20 10:35:29,665 [salt.minion      ][INFO    ][1815] User root Executing command state.sls with jid 20170120103529637036
2017-01-20 10:35:29,666 [salt.minion      ][DEBUG   ][1815] Command details {'tgt_type': 'glob', 'jid': '20170120103529637036', 'tgt': 'tcqa598int1.bius.bi.corp', 'ret': '', 'to': 20, 'user': 'root', 'arg': ['test-case'], 'fun': 'state.sls'}
2017-01-20 10:35:29,696 [salt.minion      ][INFO    ][3292] Starting a new job with PID 3292
2017-01-20 10:35:29,707 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded state.sls
2017-01-20 10:35:29,709 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded direct_call.get
2017-01-20 10:35:29,723 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded saltutil.is_running
2017-01-20 10:35:29,725 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded grains.get
2017-01-20 10:35:29,735 [salt.transport.zeromq][DEBUG   ][3292] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'tcqa598int1.bius.bi.corp', 'tcp://172.16.79.100:4506', 'aes')
2017-01-20 10:35:29,735 [salt.crypt       ][DEBUG   ][3292] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'tcqa598int1.bius.bi.corp', 'tcp://172.16.79.100:4506')
2017-01-20 10:35:33,137 [salt.pillar      ][DEBUG   ][3292] Determining pillar cache
2017-01-20 10:35:33,138 [salt.transport.zeromq][DEBUG   ][3292] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'tcqa598int1.bius.bi.corp', 'tcp://172.16.79.100:4506', 'aes')
2017-01-20 10:35:33,138 [salt.crypt       ][DEBUG   ][3292] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'tcqa598int1.bius.bi.corp', 'tcp://172.16.79.100:4506')
2017-01-20 10:35:33,580 [salt.crypt       ][DEBUG   ][3292] Loaded minion key: /etc/salt/pki/minion/minion.pem
2017-01-20 10:35:33,591 [salt.state       ][INFO    ][3292] Loading fresh modules for state activity
2017-01-20 10:35:33,610 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded jinja.render
2017-01-20 10:35:33,610 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded yaml.render
2017-01-20 10:35:33,658 [salt.fileclient  ][DEBUG   ][3292] In saltenv 'base', looking at rel_path 'test-case.sls' to resolve 'salt://test-case.sls'
2017-01-20 10:35:33,658 [salt.fileclient  ][DEBUG   ][3292] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/test-case.sls' to resolve 'salt://test-case.sls'
2017-01-20 10:35:33,659 [salt.fileclient  ][INFO    ][3292] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://test-case.sls', mode up-to-date
2017-01-20 10:35:33,659 [salt.template    ][DEBUG   ][3292] compile template: /var/cache/salt/minion/files/base/test-case.sls
2017-01-20 10:35:33,659 [salt.utils.jinja ][DEBUG   ][3292] Jinja search path: ['/var/cache/salt/minion/files/base']
2017-01-20 10:35:33,663 [salt.template    ][PROFILE ][3292] Time (in seconds) to render '/var/cache/salt/minion/files/base/test-case.sls' using 'jinja' renderer: 0.0039210319519
2017-01-20 10:35:33,664 [salt.template    ][DEBUG   ][3292] Rendered data from file: /var/cache/salt/minion/files/base/test-case.sls:
non-existing-service:
  service.dead:
    - name: non-existing-service
    - enable: False

2017-01-20 10:35:29,665 [salt.minion      ][INFO    ][1815] User root Executing command state.sls with jid 20170120103529637036
2017-01-20 10:35:29,666 [salt.minion      ][DEBUG   ][1815] Command details {'tgt_type': 'glob', 'jid': '20170120103529637036', 'tgt': 'MINION', 'ret': '', 'to': 20, 'user': 'root', 'arg': ['test-case'], 'fun': 'state.sls'}
2017-01-20 10:35:29,696 [salt.minion      ][INFO    ][3292] Starting a new job with PID 3292
2017-01-20 10:35:29,707 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded state.sls
2017-01-20 10:35:29,709 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded direct_call.get
2017-01-20 10:35:29,723 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded saltutil.is_running
2017-01-20 10:35:29,725 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded grains.get
2017-01-20 10:35:29,735 [salt.transport.zeromq][DEBUG   ][3292] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506', 'aes')
2017-01-20 10:35:29,735 [salt.crypt       ][DEBUG   ][3292] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506')
2017-01-20 10:35:33,137 [salt.pillar      ][DEBUG   ][3292] Determining pillar cache
2017-01-20 10:35:33,138 [salt.transport.zeromq][DEBUG   ][3292] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506', 'aes')
2017-01-20 10:35:33,138 [salt.crypt       ][DEBUG   ][3292] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506')
2017-01-20 10:35:33,580 [salt.crypt       ][DEBUG   ][3292] Loaded minion key: /etc/salt/pki/minion/minion.pem
2017-01-20 10:35:33,591 [salt.state       ][INFO    ][3292] Loading fresh modules for state activity
2017-01-20 10:35:33,610 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded jinja.render
2017-01-20 10:35:33,610 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded yaml.render
2017-01-20 10:35:33,658 [salt.fileclient  ][DEBUG   ][3292] In saltenv 'base', looking at rel_path 'test-case.sls' to resolve 'salt://test-case.sls'
2017-01-20 10:35:33,658 [salt.fileclient  ][DEBUG   ][3292] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/test-case.sls' to resolve 'salt://test-case.sls'
2017-01-20 10:35:33,659 [salt.fileclient  ][INFO    ][3292] Fetching file from saltenv 'base', ** skipped ** latest already in cache 'salt://test-case.sls', mode up-to-date
2017-01-20 10:35:33,659 [salt.template    ][DEBUG   ][3292] compile template: /var/cache/salt/minion/files/base/test-case.sls
2017-01-20 10:35:33,659 [salt.utils.jinja ][DEBUG   ][3292] Jinja search path: ['/var/cache/salt/minion/files/base']
2017-01-20 10:35:33,663 [salt.template    ][PROFILE ][3292] Time (in seconds) to render '/var/cache/salt/minion/files/base/test-case.sls' using 'jinja' renderer: 0.0039210319519
2017-01-20 10:35:33,664 [salt.template    ][DEBUG   ][3292] Rendered data from file: /var/cache/salt/minion/files/base/test-case.sls:
non-existing-service:
  service.dead:
    - name: non-existing-service
    - enable: False

2017-01-20 10:35:33,666 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded config.get
2017-01-20 10:35:33,666 [salt.loaded.int.render.yaml][DEBUG   ][3292] Results of YAML rendering:
OrderedDict([('non-existing-service', OrderedDict([('service.dead', [OrderedDict([('name', 'non-existing-service')]), OrderedDict([('enable', False)])])]))])
2017-01-20 10:35:33,666 [salt.template    ][PROFILE ][3292] Time (in seconds) to render '/var/cache/salt/minion/files/base/test-case.sls' using 'yaml' renderer: 0.00234699249268
2017-01-20 10:35:34,040 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded service.start
2017-01-20 10:35:34,041 [salt.utils.lazy  ][DEBUG   ][3292] LazyLoaded service.dead
2017-01-20 10:35:34,041 [salt.state       ][INFO    ][3292] Running state [non-existing-service] at time 10:35:34.041282
2017-01-20 10:35:34,041 [salt.state       ][INFO    ][3292] Executing state service.dead for non-existing-service
2017-01-20 10:35:34,042 [salt.loaded.int.module.cmdmod][INFO    ][3292] Executing command ['systemctl', 'status', 'non-existing-service', '-n', '0'] in directory '/root'
2017-01-20 10:35:34,056 [salt.loaded.int.module.cmdmod][DEBUG   ][3292] retcode: 4
2017-01-20 10:35:34,065 [salt.state       ][ERROR   ][3292] Failed to get information on unit 'non-existing-service'
2017-01-20 10:35:34,066 [salt.state       ][INFO    ][3292] Completed state [non-existing-service] at time 10:35:34.066396 duration_in_ms=25.114
2017-01-20 10:35:34,066 [salt.state       ][DEBUG   ][3292] File /var/cache/salt/minion/accumulator/62136016 does not exist, no need to cleanup.
2017-01-20 10:35:34,067 [salt.minion      ][DEBUG   ][3292] Minion return retry timer set to 7 seconds (randomized)
2017-01-20 10:35:34,067 [salt.minion      ][INFO    ][3292] Returning information for job: 20170120103529637036
2017-01-20 10:35:34,068 [salt.transport.zeromq][DEBUG   ][3292] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506', 'aes')
2017-01-20 10:35:34,068 [salt.crypt       ][DEBUG   ][3292] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'MINION', 'tcp://salt-master:4506')

Versions Report

[root@salt-master base]# salt --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 2.5.3
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: 0.20.0
        libnacl: Not Installed
       M2Crypto: 0.20.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3c1
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: 0.20.3
         Python: 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 0.8.1
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: redhat 6.8 Santiago
        machine: x86_64
        release: 2.6.32-642.11.1.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.8 Santiago

[root@$MINIONID salt]# salt-call --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.8.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  2 2016, 04:20:16)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.3 Maipo
        machine: x86_64
        release: 3.10.0-514.2.2.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.3 Maipo

The minion has these systemd packages installed:

[root@$MINIONID salt]# rpm -qa |grep systemd
systemd-219-30.el7_3.6.x86_64
systemd-libs-219-30.el7_3.6.x86_64
systemd-python-219-30.el7_3.6.x86_64
systemd-sysv-219-30.el7_3.6.x86_64

The state succeeds on a minion with prior versions of systemd:

[root@$MINIONID ~]# rpm -qa |grep systemd
systemd-219-19.el7_2.13.x86_64
systemd-python-219-19.el7_2.13.x86_64
systemd-libs-219-19.el7_2.13.x86_64
systemd-sysv-219-19.el7_2.13.x86_64
gtmanfred commented 7 years ago

I am unable to replicate this issue.

[root@salt ~]# salt-call state.apply test --local
local:
----------
          ID: non-existing-service
    Function: service.dead
      Result: True
     Comment: The named service non-existing-service is not available
     Started: 18:29:08.216462
    Duration: 21.847 ms
     Changes:

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  21.847 ms
[root@salt ~]# rpm -qa systemd\*
systemd-python-219-30.el7_3.6.x86_64
systemd-219-30.el7_3.6.x86_64
systemd-sysv-219-30.el7_3.6.x86_64
systemd-libs-219-30.el7_3.6.x86_64
[root@salt ~]# cat /srv/salt/test.sls
non-existing-service:
  service.dead:
    - name: non-existing-service
    - enable: False
[root@salt ~]# salt-call --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  2 2016, 04:20:16)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.3 Maipo
        machine: x86_64
        release: 3.10.0-514.2.2.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.3 Maipo

Also working from the master.

[root@salt ~]# salt \* state.apply test
salt.c.gtmanfred-1263.internal:
----------
          ID: non-existing-service
    Function: service.dead
      Result: True
     Comment: The named service non-existing-service is not available
     Started: 18:30:38.167912
    Duration: 22.323 ms
     Changes:

Summary for salt.c.gtmanfred-1263.internal
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  22.323 ms

Can you make sure you don't have an extra systemd.py or service.py in your extmods for the minion?

Otherwise, I will need some more information to be able to replicate this issue.

Thanks, Daniel

roskens commented 7 years ago

The only extmods in place are one for some custom grains and another module.

# locate extmod
/usr/lib/python2.7/site-packages/salt/utils/extmods.py
/usr/lib/python2.7/site-packages/salt/utils/extmods.pyc
/usr/lib/python2.7/site-packages/salt/utils/extmods.pyo
/var/cache/salt/minion/extmods
/var/cache/salt/minion/extmods/grains
/var/cache/salt/minion/extmods/modules
/var/cache/salt/minion/extmods/grains/biw_host.py
/var/cache/salt/minion/extmods/grains/biw_host.pyc
/var/cache/salt/minion/extmods/modules/biw.py
/var/cache/salt/minion/extmods/modules/biw.pyc

What's weird is I get the error on my RHEL6 salt-master running 2016.11.1, but not when I apply the state locally on the RHEL7 salt-minion.

# rpm -qa |grep salt
salt-2016.11.1-1.el6.noarch
salt-master-2016.11.1-1.el6.noarch
salt-repo-latest-1.el6.noarch
salt-minion-2016.11.1-1.el6.noarch
stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.