sjorge
commented
8 years ago a broken 2016.3.3 minion
salt-call -l trace state.apply[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from '/etc/salt/minion.d/smtp.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/smtp.conf
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[DEBUG ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from '/etc/salt/minion.d/smtp.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/smtp.conf
[TRACE ] Loading core.hwaddr_interfaces grain
[TRACE ] Loading core.hostname grain
[TRACE ] Loading core.get_master grain
[TRACE ] Loading core.pythonversion grain
[TRACE ] Loading core.path grain
[TRACE ] Loading core.get_server_id grain
[TRACE ] Loading core.saltversion grain
[TRACE ] Loading core.saltpath grain
[TRACE ] Loading core.pythonexecutable grain
[TRACE ] Loading core.ip_fqdn grain
[TRACE ] Loading core.dns grain
[TRACE ] Loading core.ip6_interfaces grain
[TRACE ] Loading core.ip4_interfaces grain
[TRACE ] Loading core.append_domain grain
[TRACE ] Loading core.os_data grain
[TRACE ] 'lspci' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[DEBUG ] The `lspci` binary is not available on the system. GPU grains will not be available.
[TRACE ] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
[TRACE ] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
[TRACE ] 'virt-what' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[DEBUG ] Please install 'virt-what' to improve results of the 'virtual' grain.
[WARNING ] /usr/lib/python2.7/dist-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.
[TRACE ] Loading core.zmqversion grain
[TRACE ] Loading core.saltversioninfo grain
[TRACE ] Loading core.pythonpath grain
[TRACE ] Loading core.id_ grain
[TRACE ] Loading core.locale_info grain
[TRACE ] Loading core.get_machine_id grain
[TRACE ] Loading core.ip_interfaces grain
[DEBUG ] Connecting to master. Attempt 1 of 1
[DEBUG ] Initializing new AsyncAuth for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506')
[DEBUG ] Generated random reconnect delay between '1000ms' and '11000ms' (7921)
[DEBUG ] Setting zmq_reconnect_ivl to '7921ms'
[DEBUG ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'clear')
[TRACE ] Inserted key into loop_instance_map id 140737407999080 for key ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'clear') and process 74848
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: /salt/config/pki/minion/minion.pem
[DEBUG ] Loaded minion key: /salt/config/pki/minion/minion.pem
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506')
[TRACE ] Inserted key into loop_instance_map id 140737407996416 for key ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes') and process 74848
[DEBUG ] Loaded minion key: /salt/config/pki/minion/minion.pem
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded state.apply
[DEBUG ] LazyLoaded grains.get
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506')
[TRACE ] Inserted key into loop_instance_map id 140737407115848 for key ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes') and process 74848
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506')
[TRACE ] Inserted key into loop_instance_map id 140737407163352 for key ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes') and process 74848
[DEBUG ] Loaded minion key: /salt/config/pki/minion/minion.pem
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[ERROR ] No contents found in top file
[DEBUG ] Loaded minion key: /salt/config/pki/minion/minion.pem
[DEBUG ] LazyLoaded config.option
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506')
[TRACE ] Inserted key into loop_instance_map id 140737407115560 for key ('/salt/config/pki/minion', 'artemis', 'tcp://[2001:470:7ee7:30::130]:4506', 'aes') and process 74848
[DEBUG ] LazyLoaded highstate.output
[TRACE ] data = {'local': {'no_|-states_|-states_|-None': {'comment': u'No Top file or external nodes data matches found.', 'name': 'No States', 'start_time': u'', 'result': False, 'duration': u'', '__run_num__': 0, 'changes': {}}}}
local:
----------
ID: states
Function: no.None
Result: False
Comment: No Top file or external nodes data matches found.
Started:
Duration:
Changes:
Summary for local
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 0.000 msmaster.log during salt-call
[TRACE ] _get_event() waited 5 seconds and received nothing
[INFO ] Authentication request from artemis
[INFO ] Authentication accepted from artemis
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCServer: Handling connection to address:
[DEBUG ] Sending event: tag = salt/auth; data = {'id': 'artemis', '_stamp': '2016-10-08T21:15:00.976309', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyZHFb4gifJHx5SuIV22O\n3pUKq0vUNuH3Votp//yoC+DH92Pd/46k+V+7j3m4FZR9CNTur13lINePZ9Shm6aI\nqxJMoR2vR+L5LUbQLnmlO9agvkt4DYlt7+YOiCW6Z5nq+AUb9OmjfEhBVTuSKEzw\nQXOSBx4Ya5Uz6YTmMcJe+hF0EJc1VmvZcEA1XUnuNgMuv/6xrM+0faxIUueZ00sB\nAH/IZeLzZsVyeZHBnqPxQNqyQ4VzDEGlCuNlyPeD7+tvY5MJTbe4nBcAfGG8FNYn\nT72xInnLLfheOYNex8rrKmmI2+zB9K+/3t2ZuJ4mJ3NZwIM6X8NbaVGm7ljUWdCu\npQIDAQAB\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[TRACE ] get_event() received = {'tag': 'salt/auth', 'data': {'_stamp': '2016-10-08T21:15:00.976309', 'act': 'accept', 'id': 'artemis', 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyZHFb4gifJHx5SuIV22O\n3pUKq0vUNuH3Votp//yoC+DH92Pd/46k+V+7j3m4FZR9CNTur13lINePZ9Shm6aI\nqxJMoR2vR+L5LUbQLnmlO9agvkt4DYlt7+YOiCW6Z5nq+AUb9OmjfEhBVTuSKEzw\nQXOSBx4Ya5Uz6YTmMcJe+hF0EJc1VmvZcEA1XUnuNgMuv/6xrM+0faxIUueZ00sB\nAH/IZeLzZsVyeZHBnqPxQNqyQ4VzDEGlCuNlyPeD7+tvY5MJTbe4nBcAfGG8FNYn\nT72xInnLLfheOYNex8rrKmmI2+zB9K+/3t2ZuJ4mJ3NZwIM6X8NbaVGm7ljUWdCu\npQIDAQAB\n-----END PUBLIC KEY-----\n', 'result': True}}
[DEBUG ] Gathering reactors for tag salt/auth
[TRACE ] AES payload received with command _pillar
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] compile template: /salt/pillar/top.sls
[DEBUG ] Jinja search path: ['/salt/pillar']
[PROFILE ] Time (in seconds) to render '/salt/pillar/top.sls' using 'jinja' renderer: 0.0146918296814
[DEBUG ] Rendered data from file: /salt/pillar/top.sls:
######
## top pillar
## -----------------------------------
######
base:
## everything (last because minion config should win)
'*':
- salt
# vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2
[DEBUG ] LazyLoaded config.get
[DEBUG ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['salt'])]))])
[PROFILE ] Time (in seconds) to render '/salt/pillar/top.sls' using 'yaml' renderer: 0.113229990005
[DEBUG ] compound_match: artemis ? *
[DEBUG ] compound_match artemis ? "*" => "True"
[DEBUG ] compile template: /salt/pillar/salt.sls
[DEBUG ] Jinja search path: ['/salt/pillar']
[PROFILE ] Time (in seconds) to render '/salt/pillar/salt.sls' using 'jinja' renderer: 0.00413703918457
[DEBUG ] Rendered data from file: /salt/pillar/salt.sls:
######
## salt pillar data
## -----------------------------------
######
####
## Template: (required) [optional]
##
## salt:
## prefix: (/salt/config) # location for salt configuration files
## master: (master.example.org) # salt master host
## pkg_provider: [pkgin] # overwrite package provider
## schedule_apply: (0) # schedule auto highstate runs every X minutes
## log_level_logfile: (warning) # log level
## smtp:
## server: (smtp.example.org) # smtp server
## tls: (true) # use tls
####
salt:
master: cronos.acheron.be
schedule_apply: 20
smtp:
server: exosphere.acheron.be
tls: true
# vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2
[DEBUG ] Results of YAML rendering:
OrderedDict([('salt', OrderedDict([('master', 'cronos.acheron.be'), ('schedule_apply', 20), ('smtp', OrderedDict([('server', 'exosphere.acheron.be'), ('tls', True)]))]))])
[PROFILE ] Time (in seconds) to render '/salt/pillar/salt.sls' using 'yaml' renderer: 0.0097541809082
[DEBUG ] LazyLoaded localfs.store
[TRACE ] Master function call _pillar took 0.28388094902 seconds
[TRACE ] AES payload received with command _master_opts
[DEBUG ] LazyLoaded roots.envs
[TRACE ] Master function call _master_opts took 0.00274276733398 seconds
[TRACE ] AES payload received with command _file_envs
[TRACE ] Master function call _file_envs took 0.000266075134277 seconds
[TRACE ] AES payload received with command _file_envs
[TRACE ] Master function call _file_envs took 0.000289916992188 seconds
[TRACE ] AES payload received with command _file_list
[DEBUG ] LazyLoaded roots.envs
[TRACE ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w created
[TRACE ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w removed
[TRACE ] Master function call _file_list took 0.0466001033783 seconds
[TRACE ] AES payload received with command _pillar
[DEBUG ] Determining pillar cache
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] compile template: /salt/pillar/top.sls
[DEBUG ] Jinja search path: ['/salt/pillar']
[PROFILE ] Time (in seconds) to render '/salt/pillar/top.sls' using 'jinja' renderer: 0.00322294235229
[DEBUG ] Rendered data from file: /salt/pillar/top.sls:
######
## top pillar
## -----------------------------------
######
base:
## everything (last because minion config should win)
'*':
- salt
# vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2
[DEBUG ] LazyLoaded config.get
[DEBUG ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['salt'])]))])
[PROFILE ] Time (in seconds) to render '/salt/pillar/top.sls' using 'yaml' renderer: 0.00576901435852
[DEBUG ] compound_match: artemis ? *
[DEBUG ] compound_match artemis ? "*" => "True"
[DEBUG ] compile template: /salt/pillar/salt.sls
[DEBUG ] Jinja search path: ['/salt/pillar']
[PROFILE ] Time (in seconds) to render '/salt/pillar/salt.sls' using 'jinja' renderer: 0.00324702262878
[DEBUG ] Rendered data from file: /salt/pillar/salt.sls:
######
## salt pillar data
## -----------------------------------
######
####
## Template: (required) [optional]
##
## salt:
## prefix: (/salt/config) # location for salt configuration files
## master: (master.example.org) # salt master host
## pkg_provider: [pkgin] # overwrite package provider
## schedule_apply: (0) # schedule auto highstate runs every X minutes
## log_level_logfile: (warning) # log level
## smtp:
## server: (smtp.example.org) # smtp server
## tls: (true) # use tls
####
salt:
master: cronos.acheron.be
schedule_apply: 20
smtp:
server: exosphere.acheron.be
tls: true
# vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2
[DEBUG ] Results of YAML rendering:
OrderedDict([('salt', OrderedDict([('master', 'cronos.acheron.be'), ('schedule_apply', 20), ('smtp', OrderedDict([('server', 'exosphere.acheron.be'), ('tls', True)]))]))])
[PROFILE ] Time (in seconds) to render '/salt/pillar/salt.sls' using 'yaml' renderer: 0.0093297958374
[TRACE ] Master function call _pillar took 0.145024061203 seconds
[TRACE ] AES payload received with command _ext_nodes
[TRACE ] Master function call _ext_nodes took 0.0299818515778 seconds
[TRACE ] AES payload received with command _return
[DEBUG ] LazyLoaded local_cache.prep_jid
[DEBUG ] Adding minions for job 20161008231501739421: ['artemis']
[INFO ] Got return from artemis for job 20161008231501739421
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCServer: Handling connection to address:
[DEBUG ] Sending event: tag = salt/job/20161008231501739421/ret/artemis; data = {'tgt_type': 'glob', 'fun_args': [], 'jid': '20161008231501739421', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found.', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'tgt': 'artemis', 'cmd': '_return', '_stamp': '2016-10-08T21:15:01.747562', 'arg': [], 'fun': 'state.apply', 'id': 'artemis', 'out': 'highstate'}
[TRACE ] get_event() received = {'tag': 'salt/job/20161008231501739421/ret/artemis', 'data': {'tgt_type': 'glob', 'fun_args': [], 'jid': '20161008231501739421', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found.', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'tgt': 'artemis', 'cmd': '_return', '_stamp': '2016-10-08T21:15:01.747562', 'arg': [], 'fun': 'state.apply', 'id': 'artemis', 'out': 'highstate'}}
[DEBUG ] Gathering reactors for tag salt/job/20161008231501739421/ret/artemis
[TRACE ] Master function call _return took 0.0174200534821 seconds
[TRACE ] _get_event() waited 5 seconds and received nothing
[TRACE ] _get_event() waited 5 seconds and received nothing
[TRACE ] _get_event() waited 5 seconds and received nothing
kev009
Ch3LL
rallytime
terminalmage
meggiebot
ccrivelli
Description of Issue/Question
I updated most of my minions except a critical one. I also updated my master.
Now the critical minion kept back at 2016.3.3 is no longer working.
state.show_top shows an empty top file.
Setup
top.sls
test.sls
Steps to Reproduce Issue
running state.apply or state.show_top on oldminion salt oldminion state.show_top
Versions Report
minion:
master