search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
13.99k stars 5.47k forks source link

salt-call event.send doesn't send events from minion #24198

Closed ahammond closed 8 years ago

ahammond commented 9 years ago

I ran the following two commands, one on the master and the other on the minion.

[root@inf-vm-sam-1 ~]# salt-call event.send test_from_master
local:
    True
[root@pvn-vm-uhs-1 ~]# salt-call event.send event_from_minion
local:
    True

On the master, I saw the following (added some whitespace for readability)

[root@inf-vm-sam-1 ~]# salt-run state.event | grep -v salt/auth
test_from_master    {"_stamp": "2015-05-27T23:44:56.173014", "pretag": null, "cmd": "_minion_event", "tag": "test_from_master", "data": {"__pub_fun": "event.send", "__pub_jid": "20150527164456161142", "__pub_pid": 2946, "__pub_tgt": "salt-call"}, "id": "inf-vm-sam-1"}

salt/job/20150527164456194678/ret/inf-vm-sam-1  {"tgt_type": "glob", "fun_args": ["test_from_master"], "jid": "20150527164456194678", "return": true, "retcode": 0, "tgt": "inf-vm-sam-1", "cmd": "_return", "_stamp": "2015-05-27T23:44:56.197411", "arg": ["test_from_master"], "fun": "event.send", "id": "inf-vm-sam-1"}

salt/job/20150527164520185068/ret/pvn-vm-uhs-1  {"tgt_type": "glob", "fun_args": ["event_from_minion"], "jid": "20150527164520185068", "return": true, "retcode": 0, "tgt": "pvn-vm-uhs-1", "cmd": "_return", "_stamp": "2015-05-27T23:45:20.185847", "arg": ["event_from_minion"], "fun": "event.send", "id": "pvn-vm-uhs-1"}
^C
Exiting gracefully on Ctrl-c

I expected to see a test_from_minion event in there, but only saw test_from_master, however both salt/job events for the two salt-call commands was present.

[root@inf-vm-sam-1 ~]# salt-run --versions-report
           Salt: 2015.5.0
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.5
           Mako: Not Installed
ahammond commented 9 years ago

@thatch45 Am I missing something here? This is almost identical to the example in the docs and it doesn't work.

jfindlay commented 9 years ago

@ahammond, thanks for the report, this looks like a bug to me.

ahammond commented 9 years ago

With debug logging added. I would love a way to set debug for everything except pillars... 

[root@cie-vm-bbm-1 ~]# salt-call -l debug event.send event_from_minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: cie-vm-bbm-1
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'cie-vm-bbm-1', 'tcp://192.168.33.2:4506')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded event.send
[DEBUG   ] MinionEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_04ca7ce023_pub.ipc
[DEBUG   ] MinionEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_04ca7ce023_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2015-06-08T17:55:27.595183'}
[DEBUG   ] Sending event - data = {'pretag': None, '_stamp': '2015-06-08T17:55:28.596555', 'tag': 'event_from_minion', 'data': {'__pub_fun': 'event.send', '__pub_jid': '20150608105527593517', '__pub_pid': 14945, '__pub_tgt': 'salt-call'}, 'events': None}
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'cie-vm-bbm-1', 'tcp://192.168.33.2:4506')
[DEBUG   ] LazyLoaded nested.output
local:
    True

Master log

[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from cie-vm-bbm-1
[INFO    ] Authentication accepted from cie-vm-bbm-1
[DEBUG   ] Sending event - data = {'id': 'cie-vm-bbm-1', '_stamp': '2015-06-08T17:55:24.597783', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuU68w+//9TnC6q8sva8U\nszfwKhsL7M8dsPSFlg3cU6puCMxV/vXRWyncXpYa0rE5zTMGfN/IOZ0ybaBrU8C1\nCp4QwbfPeRPXU0Rctek5++/4ARMjBEI3bElgteut+yipdfYeA8v4qIQDwPcRnZog\nLdO7Z0RU0y/oU4wc4FO0IgkFiG9bXWaBpiAUFFF+G9OFN4cle2xal5OjLVy/1vZ3\nH++Jw/g/aMh150AZjQmXRE6vVHicSb9xJeuUNgAUYo01TBz2M3i1cjVzF4yYgXVu\neZmCaDLFMgL3bkU5TVU54O391guCxbDaXwmUxf4ZmaFNKB6INO3gNdIV6d9VgQ6J\nbQIDAQAB\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[DEBUG   ] Gathering reactors for tag salt/auth
[INFO    ] AES payload received with command _pillar
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded cmd.run
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/top.sls:
base:
  '*':
    - authorized_keys
    - branch_based
    - collectd
    - elk.forwarder
    - groups
    - host_roles
    - hosts
    - hosts_by_legacy
    - networks
    - nginx
    - ntp
    - storage_servers
    - ssh_keys
    - users
    - zabbix

  '*-*-u*': #this should match any HSS servers
    - elite_ssl

  'cie-*':
    - buildbot
    - storage_servers
    - yum_repo
    - zabbix

  'dev-*-u*':
    - elite.volunteers

  'depot*':
    - data_transfer_users
    - depot
    - geoip
    - dns_server

  'dns-*':
    - dns_server

  'dumbdns*':
    - af_dumbdns
    - af_probe
    - circus
    - tls_certs
    - wsapi

  'inf-*-els-*':
      - elk.es
      - elk.kibana

  'inf-*-log-*':
      - elk.logstash
      - elk.es

  'inf-*-ops-*':
    - data_transfer_users
    - depot
    - dns_server
    - geoip

  'inf-*':
    - zabbix

  'inf-*-sam-*':
    - colo_bootservers
    - salt_map
    - stashpulluser

  'inf-*-sto-*':
    - colo_bootservers
    - depot
    - data_transfer_users
    - dns_server
    - zabbix_kpi.alternative
    - openvpn
    - openvpn.client

  'prd-*':
    - storage_servers
    - yum_repo
    - zabbix
    - zabbix_kpi.alternative

  'prd-*-u*':
    - elite.live

  'pvn-*':
    - storage_servers
    - yum_repo
    - zabbix
    - zabbix_kpi.alternative

  'pvn-*-u*':
    - elite.volunteers

  'qae-*':
    - storage_servers
    - yum_repo
    - zabbix

  'qae-*-u*':
    - elite.volunteers

  '*jenkins*':
    - jenkins

  '*rdb*':
    - circus
    - riak
    - tls_certs

  '*wsapi*':
    - circus
    - riak
    - tls_certs
    - wsapi

[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['authorized_keys', 'branch_based', 'collectd', 'elk.forwarder', 'groups', 'host_roles', 'hosts', 'hosts_by_legacy', 'networks', 'nginx', 'ntp', 'storage_servers', 'ssh_keys', 'users', 'zabbix']), ('*-*-u*', ['elite_ssl']), ('cie-*', ['buildbot', 'storage_servers', 'yum_repo', 'zabbix']), ('dev-*-u*', ['elite.volunteers']), ('depot*', ['data_transfer_users', 'depot', 'geoip', 'dns_server']), ('dns-*', ['dns_server']), ('dumbdns*', ['af_dumbdns', 'af_probe', 'circus', 'tls_certs', 'wsapi']), ('inf-*-els-*', ['elk.es', 'elk.kibana']), ('inf-*-log-*', ['elk.logstash', 'elk.es']), ('inf-*-ops-*', ['data_transfer_users', 'depot', 'dns_server', 'geoip']), ('inf-*', ['zabbix']), ('inf-*-sam-*', ['colo_bootservers', 'salt_map', 'stashpulluser']), ('inf-*-sto-*', ['colo_bootservers', 'depot', 'data_transfer_users', 'dns_server', 'zabbix_kpi.alternative', 'openvpn', 'openvpn.client']), ('prd-*', ['storage_servers', 'yum_repo', 'zabbix', 'zabbix_kpi.alternative']), ('prd-*-u*', ['elite.live']), ('pvn-*', ['storage_servers', 'yum_repo', 'zabbix', 'zabbix_kpi.alternative']), ('pvn-*-u*', ['elite.volunteers']), ('qae-*', ['storage_servers', 'yum_repo', 'zabbix']), ('qae-*-u*', ['elite.volunteers']), ('*jenkins*', ['jenkins']), ('*rdb*', ['circus', 'riak', 'tls_certs']), ('*wsapi*', ['circus', 'riak', 'tls_certs', 'wsapi'])]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/authorized_keys/init.sls:
authorized_keys:
  infdata:
    infdata:
      - command="/bin/rysnc-only.sh"
  uhssdata:
    uhssdata:
      - command="/bin/rysnc-only.sh"

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('authorized_keys', OrderedDict([('infdata', OrderedDict([('infdata', ['command="/bin/rysnc-only.sh"'])])), ('uhssdata', OrderedDict([('uhssdata', ['command="/bin/rysnc-only.sh"'])]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/branch_based/init.sls:
branch_based:

  placeholder

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('branch_based', 'placeholder')])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/collectd/init.sls:
collectd:
  servers:
    collectd01.anchorfree.net:
      port: 25826
      username: cd_user
      password: WI500Wc2jI8T67x82T43G20C
    collectd03.anchorfree.net:
      port: 25826
      username: cd_user
      password: WI500Wc2jI8T67x82T43G20C
[DEBUG   ] Results of YAML rendering: 
OrderedDict([('collectd', OrderedDict([('servers', OrderedDict([('collectd01.anchorfree.net', OrderedDict([('port', 25826), ('username', 'cd_user'), ('password', 'WI500Wc2jI8T67x82T43G20C')])), ('collectd03.anchorfree.net', OrderedDict([('port', 25826), ('username', 'cd_user'), ('password', 'WI500Wc2jI8T67x82T43G20C')]))]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/elk/forwarder.sls:
elk.forwarder:
  lookup:

    forwarder:
      servers:
        - "74.115.0.166:5000"
      paths:
        syslog:
          - "/var/log/*.log"
          - "/var/log/messages"
        stdin:
          - "-"
      timeout: 10
      user: root
      group: root
      nice: 0

  cert: |
        ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... ) 
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/groups/init.sls:
groups:
  adm:
    - gid: 6010
  sysadmin:
    - gid: 6020
  rvm:
    - gid: 6030
[DEBUG   ] Results of YAML rendering: 
OrderedDict([('groups', OrderedDict([('adm', [OrderedDict([('gid', 6010)])]), ('sysadmin', [OrderedDict([('gid', 6020)])]), ('rvm', [OrderedDict([('gid', 6030)])])]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/host_roles/init.sls:
host_roles:
  pvn-vm-uhs-1:
    - ar_volunteer
    - infra_volunteer
    - srv_volunteer

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('host_roles', OrderedDict([('pvn-vm-uhs-1', ['ar_volunteer', 'infra_volunteer', 'srv_volunteer'])]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/hosts/init.sls:
hosts:
  ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... )
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/hosts_by_legacy/init.sls:
hosts_by_legacy:
  ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... )
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/networks/init.sls:
 ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... )
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/nginx/init.sls:
nginx:
  lookup:

    worker_processes: 1

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('nginx', OrderedDict([('lookup', OrderedDict([('worker_processes', 1)]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/ntp/init.sls:
ntp:
  comment: "Why do I need a comment?"
  localnetworks: []
  ntpservers:
    - 74.115.4.111
  lookup:
    client: ntp
    server: ntpd
    service: ntpd
    ntpdate: ntpdate
    ntp_conf: /etc/ntp.conf
    ntpd_conf: /etc/ntpd.conf

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('ntp', OrderedDict([('comment', 'Why do I need a comment?'), ('localnetworks', []), ('ntpservers', ['74.115.4.111']), ('lookup', OrderedDict([('client', 'ntp'), ('server', 'ntpd'), ('service', 'ntpd'), ('ntpdate', 'ntpdate'), ('ntp_conf', '/etc/ntp.conf'), ('ntpd_conf', '/etc/ntpd.conf')]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/storage_servers/init.sls:

storage_servers:
  {}

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('storage_servers', OrderedDict())])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/ssh_keys/init.sls:
ssh_keys:
   ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... )
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/users/init.sls:
users:
     ... snip ...
[DEBUG   ] Results of YAML rendering: 
OrderedDict( ... snip ... )
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/zabbix/init.sls:
zabbix:
  lookup:
    ServerActive: zabbix.noc.anchorfree.net,127.0.0.1
    Server: zabbix.noc.anchorfree.net,127.0.0.1
    AllowRoot: 1
    DebugLevel: 4
    LogFileSize: 5
    ListenPort: 5515

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('zabbix', OrderedDict([('lookup', OrderedDict([('ServerActive', 'zabbix.noc.anchorfree.net,127.0.0.1'), ('Server', 'zabbix.noc.anchorfree.net,127.0.0.1'), ('AllowRoot', 1), ('DebugLevel', 4), ('LogFileSize', 5), ('ListenPort', 5515)]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/buildbot/init.sls:
buildbot:
  lookup:
    debug_password: KcerLzNFZWB7mNXcpqtbTzgotm4mmH
    master_address: 192.168.33.100
    shared_secret: XfgmBBgWwirWJbeDCDyiUJ4DvQh8yc
    slave_names:

      - cie-vm-bsl-1
      - cie-vm-bsl-1.virtual.anchorfree.net

      - cie-vm-bsl-2
      - cie-vm-bsl-2.virtual.anchorfree.net

      - cie-vm-bsl-3
      - cie-vm-bsl-3.virtual.anchorfree.net

      - cie-vm-bsl-4
      - cie-vm-bsl-4.virtual.anchorfree.net

      - cie-vm-bsl-5
      - cie-vm-bsl-5.virtual.anchorfree.net

      - cie-vm-bsl-6
      - cie-vm-bsl-6.virtual.anchorfree.net

      - cie-vm-bsl-7
      - cie-vm-bsl-7.virtual.anchorfree.net

      - cie-vm-bsl-8
      - cie-vm-bsl-8.virtual.anchorfree.net

      - cie-vm-bsl-9
      - cie-vm-bsl-9.virtual.anchorfree.net

    stash_uri: http://stash.anchorfree.net/
    stash_user: ahammond
    stash_password: H@mm0nD
    userpass:
      -
        - pyflakes
        - pyflakes

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('buildbot', OrderedDict([('lookup', OrderedDict([('debug_password', 'KcerLzNFZWB7mNXcpqtbTzgotm4mmH'), ('master_address', '192.168.33.100'), ('shared_secret', 'XfgmBBgWwirWJbeDCDyiUJ4DvQh8yc'), ('slave_names', ['cie-vm-bsl-1', 'cie-vm-bsl-1.virtual.anchorfree.net', 'cie-vm-bsl-2', 'cie-vm-bsl-2.virtual.anchorfree.net', 'cie-vm-bsl-3', 'cie-vm-bsl-3.virtual.anchorfree.net', 'cie-vm-bsl-4', 'cie-vm-bsl-4.virtual.anchorfree.net', 'cie-vm-bsl-5', 'cie-vm-bsl-5.virtual.anchorfree.net', 'cie-vm-bsl-6', 'cie-vm-bsl-6.virtual.anchorfree.net', 'cie-vm-bsl-7', 'cie-vm-bsl-7.virtual.anchorfree.net', 'cie-vm-bsl-8', 'cie-vm-bsl-8.virtual.anchorfree.net', 'cie-vm-bsl-9', 'cie-vm-bsl-9.virtual.anchorfree.net']), ('stash_uri', 'http://stash.anchorfree.net/'), ('stash_user', 'ahammond'), ('stash_password', 'H@mm0nD'), ('userpass', [['pyflakes', 'pyflakes']])]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/storage_servers/init.sls:

storage_servers:
  {}

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('storage_servers', OrderedDict())])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/yum_repo/init.sls:
yum_repo:
  cie:
    pvn-ch-uhs-1: /rpms/build/test/pilot
  custom:
    pvn-ch-uhs-1: /rpms/build/test/pilot
  dev:
    pvn-ch-uhs-1: /rpms/build/test/pilot
  hotfix:
    pvn-ch-uhs-2: /rpms/uhss/prd/hotfix
  prd:
    pvn-sj-uhs-2: /rpms/uhss/prd/released
  pvn:
    pvn-sj-uhs-1: /rpms/uhss/pvn/released
    pvn-ny-uhs-266: /rpms/uhss/pvn/released
  qae:
    qae-sj-qai-47: /rpms/uhss/qae/released
    qae-sj-qai-347: /rpms/uhss/elitebackend/released
  revenuedev:
    pvn-sj-uhs-210: /rpms/uhss/revenuedev/released
  revenuestage:
    pvn-sj-uhs-2: /rpms/uhss/revenuestage/released
    pvn-ch-uhs-244: /rpms/uhss/revenuestage/released
    pvn-fr-uhs-194: /rpms/uhss/revenuestage/released
    pvn-zu-uhs-220: /rpms/uhss/revenuestage/released
    pvn-ch-uhs-246: /rpms/uhss/revenuestage/released
    pvn-ny-uhs-265: /rpms/uhss/revenuestage/released

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('yum_repo', OrderedDict([('cie', OrderedDict([('pvn-ch-uhs-1', '/rpms/build/test/pilot')])), ('custom', OrderedDict([('pvn-ch-uhs-1', '/rpms/build/test/pilot')])), ('dev', OrderedDict([('pvn-ch-uhs-1', '/rpms/build/test/pilot')])), ('hotfix', OrderedDict([('pvn-ch-uhs-2', '/rpms/uhss/prd/hotfix')])), ('prd', OrderedDict([('pvn-sj-uhs-2', '/rpms/uhss/prd/released')])), ('pvn', OrderedDict([('pvn-sj-uhs-1', '/rpms/uhss/pvn/released'), ('pvn-ny-uhs-266', '/rpms/uhss/pvn/released')])), ('qae', OrderedDict([('qae-sj-qai-47', '/rpms/uhss/qae/released'), ('qae-sj-qai-347', '/rpms/uhss/elitebackend/released')])), ('revenuedev', OrderedDict([('pvn-sj-uhs-210', '/rpms/uhss/revenuedev/released')])), ('revenuestage', OrderedDict([('pvn-sj-uhs-2', '/rpms/uhss/revenuestage/released'), ('pvn-ch-uhs-244', '/rpms/uhss/revenuestage/released'), ('pvn-fr-uhs-194', '/rpms/uhss/revenuestage/released'), ('pvn-zu-uhs-220', '/rpms/uhss/revenuestage/released'), ('pvn-ch-uhs-246', '/rpms/uhss/revenuestage/released'), ('pvn-ny-uhs-265', '/rpms/uhss/revenuestage/released')]))]))])
[DEBUG   ] Jinja search path: ['/srv/pillar']
[DEBUG   ] Rendered data from file: /srv/pillar/zabbix/init.sls:
zabbix:
  lookup:
    ServerActive: zabbix.noc.anchorfree.net,127.0.0.1
    Server: zabbix.noc.anchorfree.net,127.0.0.1
    AllowRoot: 1
    DebugLevel: 4
    LogFileSize: 5
    ListenPort: 5515

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('zabbix', OrderedDict([('lookup', OrderedDict([('ServerActive', 'zabbix.noc.anchorfree.net,127.0.0.1'), ('Server', 'zabbix.noc.anchorfree.net,127.0.0.1'), ('AllowRoot', 1), ('DebugLevel', 4), ('LogFileSize', 5), ('ListenPort', 5515)]))]))])
[DEBUG   ] LazyLoaded pillar_files.ext_pillar
[INFO    ] AES payload received with command _return
[INFO    ] Got return from cie-vm-bbm-1 for job 20150608105533810151
[DEBUG   ] Sending event - data = {'tgt_type': 'glob', 'fun_args': ['event_from_minion'], 'jid': '20150608105533810151', 'return': True, 'retcode': 0, 'tgt': 'cie-vm-bbm-1', 'cmd': '_return', '_stamp': '2015-06-08T17:55:33.810762', 'arg': ['event_from_minion'], 'fun': 'event.send', 'id': 'cie-vm-bbm-1'}
[DEBUG   ] Gathering reactors for tag salt/job/20150608105533810151/ret/cie-vm-bbm-1
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/file_client.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/file_client.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: inf-vm-sam-1
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/auto_accept.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/auto_accept.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/elite_deployed_branch.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/elite_deployed_branch.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/extension_modules.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/extension_modules.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar_files.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar_files.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/state_aggregate.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/state_aggregate.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/worker_threads.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/worker_threads.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] Could not LazyLoad config.merge
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] This salt-master instance has accepted 2 minion keys.
garethgreenaway commented 9 years ago

In your minion configuration, what does your master line look like?

ahammond commented 9 years ago

Just an IP

# cat /etc/salt/minion.d/master.conf
master: 192.168.33.2
garethgreenaway commented 9 years ago

Probably not the same thing I was seeing with multi master but could be a similar cause.

garethgreenaway commented 9 years ago

In the minion log after firing the event, do you see any lines that begin with Handle event?

ahammond commented 9 years ago

Nope, no lines with handle in them.

cachedout commented 9 years ago

@ahammond I just merged a fix from @garethgreenaway in #24607 into the head of the 2015.5 branch that might be related. Any chance you could test it?

ahammond commented 9 years ago

Does this fix require updating the minion, the master or both?

garethgreenaway commented 9 years ago

Just minion.

ahammond commented 8 years ago

I'm running 2015.5.2 now and the problem appears to be resolved.