search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

Issues with compound matching #39097

Closed Muzammilshus closed 7 years ago

Muzammilshus commented 7 years ago

Description of Issue/Question

I'm new to salt and im having difficulties targeting minion with grains. For some reason i keep getting errors when i push it to minions with salt -N group1 state.apply or salt -C 'G@osfinger:CentOS-5 and *data*' state.apply

Setup

/srv/salt/directories/init.sls

/bar/foo/java/:
  file.directory:
    - user: root
    - group: root
    - mode: 777
    - makedirs: True

/srv/salt/top.sls

base:
  'group1':
    - match: nodegroup
    - directories

  'G@osfinger:CentOS-5 and *data*'
    - match: compound
    - directories

Also deleted cache and master pub key and restarted minion service cd /var/cache/salt/master/; rm rf * ; service salt-minion restart rm -rf /etc/salt/pki/minion/minion_master.pub; service salt-master restart 

#####         Node Groups           ######
##########################################
Node groups allow for logical groupings of minion nodes. A group consists of
a group name and a compound target. Nodgroups can reference other nodegroups
with 'N@' classifier. Ensure that you do not have circular `references.`
  nodegroups:
    group1: 'L@21010storage,21070fldata.bnd.corp,lnxbksvr'

[root@SaltMaster salt]# salt-key -L 
Accepted Keys:
21010storage
21070fldata.bnd.corp
lnxbksvr
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[root@SaltMaster directories]# salt '*' state.apply -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/cd/9be06e216b20af4f9ec8f6443b9f8a82798de249892350faa7a218a4e3d8a3/.minions.p
[DEBUG   ] get_iter_returns for jid 20170131161801194650 sent to set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp']) will timeout at 16:18:06.210579
[DEBUG   ] jid 20170131161801194650 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    - The function "state.apply" is running as PID 23490 and was started at 2017, Jan 31 16:18:01.194650 with jid 20170131161801194650
[DEBUG   ] jid 20170131161801194650 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    - The function "state.apply" is running as PID 23490 and was started at 2017, Jan 31 16:18:01.194650 with jid 20170131161801194650
[DEBUG   ] jid 20170131161801194650 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    - The function "state.apply" is running as PID 23490 and was started at 2017, Jan 31 16:18:01.194650 with jid 20170131161801194650
[DEBUG   ] jid 20170131161801194650 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    ----------
    no_|-states_|-states_|-None:
        ----------
        __run_num__:
            0
        changes:
            ----------
        comment:
            No Top file or external nodes data matches found.
        name:
            No States
        result:
            False
[DEBUG   ] jid 20170131161801194650 return from 21010storage
[DEBUG   ] LazyLoaded highstate.output
21010storage:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 20439 and was started at 2017, Jan 31 16:18:01.194650 with jid 20170131161801194650
[DEBUG   ] jid 20170131161801194650 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21070fldata.bnd.corp
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] jid 20170131161801194650 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21070fldata.bnd.corp
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] jid 20170131161801194650 found all minions set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp'])
ERROR: Minions returned with non-zero exit code
[root@SaltMaster directories]# salt -N group1 state.apply -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] nodegroup_comp(group1) => ['(', 'L@21010storage,21070fldata.bnd.corp,lnxbksvr', ')']
[DEBUG   ] No nested nodegroups detected. Using original nodegroup definition: L@21010storage,21070fldata.bnd.corp,lnxbksvr
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/b2/04fbc41439cdf4a6926c6f5ee8a76c5a2a27ea394eadbcf26784af2dda8180/.minions.p
[DEBUG   ] get_iter_returns for jid 20170131161816239887 sent to set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp']) will timeout at 16:18:21.252679
[DEBUG   ] jid 20170131161816239887 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 5093 and was started at 2017, Jan 31 16:18:16.239887 with jid 20170131161816239887
[DEBUG   ] jid 20170131161816239887 return from 21010storage
[DEBUG   ] LazyLoaded highstate.output
21010storage:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 20467 and was started at 2017, Jan 31 16:18:16.239887 with jid 20170131161816239887
[DEBUG   ] jid 20170131161816239887 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21070fldata.bnd.corp
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] jid 20170131161816239887 return from 21010storage
[DEBUG   ] LazyLoaded highstate.output
21010storage:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21010storage
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] Checking whether jid 20170131161816239887 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG   ] LazyLoaded localfs.list
[DEBUG   ] Cache file "/var/cache/salt/master/minions/21070fldata.bnd.cor/data.p" does not exist
[DEBUG   ] LazyLoaded localfs.contains
[DEBUG   ] LazyLoaded no_return.output
lnxbksvr:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code
[root@SaltMaster directories]# salt -C   'G@osfinger:CentOS-5 and *data*' state.apply
21070fldata.bnd.corp:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 5129 and was started at 2017, Jan 31 16:18:51.527410 with jid 20170131161851527410
ERROR: Minions returned with non-zero exit code
[root@SaltMaster directories]# salt -C 'G@osfinger:CentOS-5 and *data*' state.apply
21070fldata.bnd.corp:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21070fldata.bnd.corp
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
21070fldata.bnd.corp:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found.
     Changes:   

Summary for 21070fldata.bnd.corp
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
ERROR: Minions returned with non-zero exit code
[root@SaltMaster directories]# salt -C 'G@osfinger:CentOS-5 and *data*' state.apply -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/72/61e48d4f26e2dc9b9092d3595be1653d21f6be3f7f56e3cac6de1cc65d8bc2/.minions.p
[DEBUG   ] get_iter_returns for jid 20170131161908065445 sent to set(['21070fldata.bnd.corp']) will timeout at 16:19:13.078224
[DEBUG   ] jid 20170131161908065445 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 5176 and was started at 2017, Jan 31 16:19:08.065445 with jid 20170131161908065445
[DEBUG   ] jid 20170131161908065445 found all minions set(['21070fldata.bnd.corp'])
ERROR: Minions returned with non-zero exit code

[root@SaltMaster salt]# salt '*' test.ping -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/be/ff897b11c8afa5465f3cda2d6071590a4331fccd0aee11786eb46ef32ec23c/.minions.p
[DEBUG   ] get_iter_returns for jid 20170131160713307193 sent to set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp']) will timeout at 16:07:18.312437
[DEBUG   ] jid 20170131160713307193 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    True
[DEBUG   ] jid 20170131160713307193 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    True
[DEBUG   ] jid 20170131160713307193 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    True
[DEBUG   ] jid 20170131160713307193 return from lnxbksvr
[DEBUG   ] LazyLoaded nested.output
lnxbksvr:
    True
[DEBUG   ] jid 20170131160713307193 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded nested.output
21070fldata.bnd.corp:
    True
[DEBUG   ] jid 20170131160713307193 return from 21010storage
[DEBUG   ] LazyLoaded nested.output
21010storage:
    True
[DEBUG   ] jid 20170131160713307193 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded nested.output
21070fldata.bnd.corp:
    True
[DEBUG   ] jid 20170131160713307193 return from 21010storage
[DEBUG   ] LazyLoaded nested.output
21010storage:
    True
[DEBUG   ] jid 20170131160713307193 found all minions set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp'])
[root@SaltMaster salt]# salt -N group1 test.ping -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] nodegroup_comp(group1) => ['(', 'L@21010storage,21070fldata.bnd.corp,lnxbksvr', ')']
[DEBUG   ] No nested nodegroups detected. Using original nodegroup definition: L@21010storage,21070fldata.bnd.corp,lnxbksvr
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/1a/2168c70596d02ab6ac9675c87c4e8bba4d069a1474fd0b2afdaf39bd0ab5d9/.minions.p
[DEBUG   ] get_iter_returns for jid 20170131160733496174 sent to set(['lnxbksvr', '21010storage', '21070fldata.bnd.corp']) will timeout at 16:07:38.501374
[DEBUG   ] jid 20170131160733496174 return from 21010storage
[DEBUG   ] LazyLoaded nested.output
21010storage:
    True
[DEBUG   ] jid 20170131160733496174 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded nested.output
21070fldata.bnd.corp:
    True
[DEBUG   ] jid 20170131160733496174 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded nested.output
21070fldata.bnd.corp:
    True
[DEBUG   ] jid 20170131160733496174 return from 21010storage
[DEBUG   ] LazyLoaded nested.output
21010storage:
    True
[DEBUG   ] Checking whether jid 20170131160733496174 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG   ] LazyLoaded localfs.list
[DEBUG   ] Cache file "/var/cache/salt/master/minions/21070fldata.bnd.cor/data.p" does not exist
[DEBUG   ] LazyLoaded localfs.contains
[DEBUG   ] LazyLoaded no_return.output
lnxbksvr:
    Minion did not return. [No response]
21070fldata pushed with salt -C 'G@osfinger:CentOS-5 and *data*' state.apply on master
[root@21070fldata minion]# salt-minion -l debug 
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: 21070fldata.bnd.corp
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Minion "21070fldata.bnd.corp"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
/usr/lib/python2.6/site-packages/salt/grains/core.py:1493: DeprecationWarning: The "osmajorrelease" will be a type of an integer.
[INFO    ] Creating minion process manager
[WARNING ] IMPORTANT: Do not use md5 hashing algorithm! Please set "hash_type" to sha256 in Salt Minion config!
[INFO    ] The Salt Minion is starting up
[INFO    ] Minion is starting as user 'root'
[DEBUG   ] AsyncEventPublisher PUB socket URI: /var/run/salt/minion/minion_event_82a3f87662_pub.ipc
[DEBUG   ] AsyncEventPublisher PULL socket URI: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[INFO    ] Starting pull socket on /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Minion '21070fldata.bnd.corp' trying to tune in
[DEBUG   ] sync_connect_master
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (4117)
[DEBUG   ] Setting zmq_reconnect_ivl to '4117ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Process Manager starting!
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Re-using AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] LazyLoaded timezone.get_offset
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command ['date', '+%z'] in directory '/root'
[DEBUG   ] output: -0500
[DEBUG   ] LazyLoaded config.merge
[DEBUG   ] LazyLoaded mine.update
[INFO    ] Updating job settings for scheduled job: __mine_interval
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_82a3f87662_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2017-02-01T00:27:07.472412', 'complete': True, 'schedule': {'__mine_interval': {'function': 'mine.update', 'enabled': True, 'return_job': False, 'jid_include': True, 'maxrunning': 2, 'minutes': 60}}}
[DEBUG   ] Persisting schedule
[INFO    ] Added mine.update to scheduler
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_82a3f87662_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2017-02-01T00:27:07.479609', 'complete': True, 'schedule': {'__mine_interval': {'function': 'mine.update', 'enabled': True, 'return_job': False, 'jid_include': True, 'maxrunning': 2, 'minutes': 60}}}
[DEBUG   ] Persisting schedule
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_82a3f87662_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/minion/minion_event_82a3f87662_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2017-02-01T00:27:07.485463', 'complete': True, 'schedule': {'__mine_interval': {'function': 'mine.update', 'enabled': True, 'return_job': False, 'jid_include': True, 'maxrunning': 2, 'minutes': 60}}}
[DEBUG   ] Persisting schedule
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[INFO    ] Minion is ready to receive requests!
[DEBUG   ] Handling event tag '/salt/minion/minion_schedule_add_complete'
[DEBUG   ] Handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG   ] Handling event tag '/salt/minion/minion_schedule_delete_complete'
[INFO    ] Running scheduled job: __mine_interval
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: This job was scheduled with a max number of 2
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun_args': [], 'jid': '20170131192709088760', 'schedule': '__mine_interval', 'pid': 5637, 'fun': 'mine.update', 'id': '21070fldata.bnd.corp'}
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20170131192709088760
[DEBUG   ] compound_match: 21070fldata.bnd.corp ? G@osfinger:CentOS-5 and *data*
[DEBUG   ] grains target: osfinger:CentOS-5
[DEBUG   ] Attempting to match 'CentOS-5' in 'osfinger' using delimiter ':'
[DEBUG   ] compound_match 21070fldata.bnd.corp ? "G@osfinger:CentOS-5 and *data*" => "True and True"
[INFO    ] User root Executing command state.apply with jid 20170131162731951718
[DEBUG   ] Command details {'tgt_type': 'compound', 'jid': '20170131162731951718', 'tgt': 'G@osfinger:CentOS-5 and *data*', 'ret': '', 'user': 'root', 'arg': [], 'fun': 'state.apply'}
[INFO    ] Starting a new job with PID 5657
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded direct_call.get
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'base', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/top.sls' to resolve 'salt://top.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[DEBUG   ] No dest file found 
[INFO    ] Fetching file from saltenv 'base', ** done ** 'top.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/top.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/top.sls' using 'jinja' renderer: 0.00839495658875
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:

base:
  'group1':
    - match: nodegroup 
    - directories

  'G@osfinger:CentOS-5 and *data*'
    - match: compound
    - directories 

[ERROR   ] Unable to render top file: could not found expected ':'
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Minion return retry timer set to 8 seconds (randomized)
[INFO    ] Returning information for job: 20170131162731951718
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '21070fldata.bnd.corp', 'tcp://x.x.x.x:4506'
21010storage pushed with salt -C 'G@osfinger:CentOS-5 and *data*' state.apply on master
[root@21010storage minion.d]# salt-minion -l debug 
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: 21010storage
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Minion "21010storage"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[INFO    ] The salt minion is starting up
[INFO    ] Minion is starting as user 'root'
[DEBUG   ] AsyncEventPublisher PUB socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pub.ipc
[DEBUG   ] AsyncEventPublisher PULL socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pull.ipc
[INFO    ] Starting pub socket on ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pub.ipc
[INFO    ] Starting pull socket on ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pull.ipc
[DEBUG   ] Minion '21010storage' trying to tune in
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (6495)
[DEBUG   ] Setting zmq_reconnect_ivl to '6495ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/99-master-address.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/99-master-address.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG   ] LazyLoaded timezone.get_offset
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command ['date', '+%z'] in directory '/root'
[DEBUG   ] output: -0500
[DEBUG   ] LazyLoaded config.merge
[DEBUG   ] LazyLoaded mine.update
[INFO    ] Added mine.update to scheduler
[INFO    ] Updating job settings for scheduled job: __mine_interval
[DEBUG   ] SaltEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2017-02-01T00:27:13.400313', 'complete': True, 'schedule': {'__mine_interval': {'function': 'mine.update', 'jid_include': True, 'minutes': 60, 'maxrunning': 2}}}
[DEBUG   ] Persisting schedule
[DEBUG   ] Handling event '/salt/minion/minion_schedule_add_complete\n\n\x83\xa6_stamp\xba2017-02-01T00:27:13.400313\xa8complete\xc3\xa8schedule\x81\xaf__mine_interval\x84\xa8function\xabmine.update\xabjid_include\xc3\xa7minutes<\xaamaxrunning\x02'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[INFO    ] Minion is ready to receive requests!
[INFO    ] Running scheduled job: __mine_interval
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: This job was scheduled with a max number of 2
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun': 'mine.update', 'jid': '20170131192715143437', 'pid': 21338, 'id': '21010storage', 'schedule': '__mine_interval'}
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] MinionEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pub.ipc
[DEBUG   ] MinionEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_9c8fa35d52_pull.ipc
[DEBUG   ] Sending event - data = {'clear': False, 'cmd': '_mine', 'data': {}, 'id': '21010storage', '_stamp': '2017-02-01T00:27:15.207124'}
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20170131192715143437
[DEBUG   ] Handling event '_minion_mine\n\n\x85\xa5clear\xc2\xa3cmd\xa5_mine\xa4data\x80\xa2id\xac21010storage\xa6_stamp\xba2017-02-01T00:27:15.207124'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20170131192715143437
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', '21010storage', 'tcp://x.x.x.x:4506')
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20170131192715143437
[DEBUG   ] compound_match: 21010storage ? G@osfinger:CentOS-5 and *data*
[DEBUG   ] grains target: osfinger:CentOS-5
[DEBUG   ] Attempting to match 'CentOS-5' in 'osfinger' using delimiter ':'
[DEBUG   ] compound_match 21010storage ? "G@osfinger:CentOS-5 and *data*" => "False and False"


### Steps to Reproduce Issue
1) Tested minions that are accepted, Issues with one odd minion not communicating
2) created top sls with compound matching.   pushed with 
`salt -N group1 state.apply `
or 
`salt -C   'G@osfinger:CentOS-5 and *data*' state.apply 
3) not able to push to minions

### Versions Report

[root@SaltMaster minion]# salt --versions-report Salt Version: Salt: 2016.11.2

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pygit2: Not Installed Python: 2.7.5 (default, Nov 6 2016, 00:28:07) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.3.1611 Core machine: x86_64 release: 3.10.0-514.el7.x86_64 system: Linux version: CentOS Linux 7.3.1611 Core

[root@21010storage minion]# rpm -qa | grep salt salt-2015.8.7-1.el6.noarch salt-minion-2015.8.7-1.el6.noarc [root@21070fldata minion.d]# rpm -qa | grep salt salt-2016.3.3-1.el5 salt-minion-2016.3.3-1.el5

roscoecairney commented 7 years ago

Maybe your top.sls is not valid YAML?

Try adding a colon at the end of 'G@osfinger:CentOS-5 and *data*'

gtmanfred commented 7 years ago

good catch @roscoecairney

@Muzammilshus is the top file a direct copy of your top file, or is the missing : just a typo from the copy over?

Thanks, Daniel

Muzammilshus commented 7 years ago

So i applied your suggested change and i got it to push for the bottom top.sls, 

base:
  'group1':
    - match: nodegroup 
    - directories

  'G@osfinger:CentOS-5 and *data*':
    - match: compound
    - directories

but now when i want to push all my packages i get this error below. also I have killed all salt-master, salt-minion services multiple times, cleared salt master cache, to try to fix it, but keep getting this error

base:
  'group1':
    - match: nodegroup 
    - directories
    - cpjarshfiles
    - itp_prop
    - javaextract

  'G@osfinger:CentOS-5 and *data*':
    - match: compound
    - directories
    - cpjarshfiles
    - itp_prop
    - javaextract 
[root@SaltMaster master]# salt -C 'G@osfinger:CentOS-5 and *data*' test.ping -l debug 
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/f5/0ed095a8da99adbe6e536483bfc27545c62bdce6b299bd395868ed95193c0f/.minions.p
[DEBUG   ] get_iter_returns for jid 20170201111615094768 sent to set(['21070fldata.bnd.corp']) will timeout at 11:16:20.104691
[DEBUG   ] jid 20170201111615094768 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded nested.output
21070fldata.bnd.corp:
    True
[DEBUG   ] jid 20170201111615094768 found all minions set(['21070fldata.bnd.corp'])
[root@SaltMaster master]# salt -C 'G@osfinger:CentOS-5 and *data*' state.apply -l debug 
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'SaltMaster.bnd.corp_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/55/7ac027121e1e480f611c53844e83ba75b30e64d50ce76cb52da2ec0297d287/.minions.p
[DEBUG   ] get_iter_returns for jid 20170201111622182734 sent to set(['21070fldata.bnd.corp']) will timeout at 11:16:27.192665
[DEBUG   ] jid 20170201111622182734 return from 21070fldata.bnd.corp
[DEBUG   ] LazyLoaded highstate.output
21070fldata.bnd.corp:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 22675 and was started at 2017, Feb 01 11:06:59.618911 with jid 20170201110659618911
[DEBUG   ] jid 20170201111622182734 found all minions set(['21070fldata.bnd.corp'])
ERROR: Minions returned with non-zero exit code
roscoecairney commented 7 years ago

Can you show the contents of these files?

/srv/salt/cpjarshfiles/init.sls /srv/salt/itp_prop/init.sls /srv/salt/javaextract/init.sls

gtmanfred commented 7 years ago

Can you run it when salt-run jobs.active doesn't show any running state jobs on those minions?

Thanks, Daniel

Muzammilshus commented 7 years ago

@gtmanfred, I think that was the issue, even though no jobs were running, i cleared the job cache in the minion and it ran but took 17 mins for 5 minions. I was sending over a java tarball around 300mb with jdk files. Is this normal ?

gtmanfred commented 7 years ago

How long does it take to transfer it without salt, using scp?

It should also cache the file on the minion so it shouldn't have to transfer it every time.

There should also be output to show how long each stateid took to run in the highstate output.

Muzammilshus commented 7 years ago

@gtmanfred around 20 mins per minion with rsync, not scp. Ill do a test run tonight and post the results with salt