search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.23k stars 5.49k forks source link

pkg & diskusage beacons not sending alerts #26320

Closed schlagify closed 9 years ago

schlagify commented 9 years ago

I have a minion configured with the following /etc/salt/minion.d/beacon.conf:

# Initialize the wtmp beacon when the minion starts up

beacons:
  pkg:
    - pkgs:
        - bind9-host
        - sysdig
    - refresh:  True
  diskusage:
    - /var:  85%

My minion has new versions of bind9-host & sysdig available for upgrading and /var is used past 90%, but I do not see any events being sent by the beacons when i run salt-run state.event pretty=True on my master. I've restarted salt-minion a bunch of times, and see no activity for these beacons in /var/log/salt/minion. I'm not finding activity in the debug log on my salt-master either, but maybe I'm missing something since I'm not familiar w/the log data yet for this. I can provide an excerpt of our debug log if useful.

salt-master --versions-report
           Salt: 2015.5.3
         Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: 1.4.3
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.6.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: Not Installed
        Tornado: 4.2

salt-minion --versions-report
                  Salt: 2015.5.3
                Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
                Jinja2: 2.6
              M2Crypto: 0.21.1
        msgpack-python: 0.1.10
          msgpack-pure: Not Installed
              pycrypto: 2.6
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 13.1.0
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.7.0
               Tornado: Not Installed
 Debian source package: 2015.5.3+ds-1~bpo70+2

OS: Debian Wheezy

jfindlay commented 9 years ago

@schlagify, thanks for the report. Your debug log output might be useful.

schlagify commented 9 years ago

ok, only problem is i dont have much of anything on the master side. i have log_level_logfile: debug set and a minion configured with this:

# cat /etc/salt/minion.d/beacon.conf 

beacons:
  wtmp: {}
  inotify:
    /etc/nginx:
      recurse:  True
      mask:
        - close_write
        - delete
        - dont_follow
    /etc/supervisor/conf.d:
      recurse:  True
      mask:
        - close_write
        - delete
        - dont_follow
    /etc/uwsgi/apps-enabled:
      recurse:  True
      mask:
        - close_write
        - delete
        - dont_follow
  pkg:
    - pkgs:
      - sysdig
      - datadog-agent
    - refresh:  True
  diskusage:
    - /var:     6%

when i bounced the minion, i saw the following events come through (which don't include any beacon events)

salt-run state.event pretty=True
salt/auth       {
    "_stamp": "2015-08-17T18:04:11.643763", 
    "act": "accept", 
    "id": "minion1", 
    "pub": "-----BEGIN PUBLIC KEY-----\n[munged]\n-----END PUBLIC KEY-----\n",
    "result": true
}
salt/auth       {
    "_stamp": "2015-08-17T18:04:11.911669",
    "act": "accept", 
    "id": "minion1", 
    "pub": "-----BEGIN PUBLIC KEY-----\n[munged]\n-----END PUBLIC KEY-----\n",
    "result": true
}
minion_start    {
    "_stamp": "2015-08-17T18:04:12.401142", 
    "cmd": "_minion_event", 
    "data": "Minion minion1 started at Mon Aug 17 14:04:12 2015", 
    "id": "minion1", 
    "pretag": null, 
    "tag": "minion_start"
}
salt/minion/minion1/start       {
    "_stamp": "2015-08-17T18:04:12.404669", 
    "cmd": "_minion_event", 
    "data": "Minion minion1 started at Mon Aug 17 14:04:12 2015", 
    "id": "minion1", 
    "pretag": null, 
    "tag": "salt/minion/minion1/start"
}

and the debug log on my master shows this (again, no beacon info b/c nothing was sent by the beacons in the first place):

2015-08-17 14:11:22,311 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master
2015-08-17 14:11:22,358 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/api.conf'
2015-08-17 14:11:22,358 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/api.conf
2015-08-17 14:11:22,360 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/master.conf'
2015-08-17 14:11:22,360 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/master.conf
2015-08-17 14:11:22,364 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/reactor.conf'
2015-08-17 14:11:22,364 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/reactor.conf
2015-08-17 14:11:22,366 [salt.config                              ][DEBUG   ][20278] Using cached minion ID from /etc/salt/minion_id: salt-master01
2015-08-17 14:11:22,367 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master
2015-08-17 14:11:22,410 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/api.conf'
2015-08-17 14:11:22,410 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/api.conf
2015-08-17 14:11:22,411 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/master.conf'
2015-08-17 14:11:22,412 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/master.conf
2015-08-17 14:11:22,416 [salt.config                              ][DEBUG   ][20278] Including configuration from '/etc/salt/master.d/reactor.conf'
2015-08-17 14:11:22,416 [salt.config                              ][DEBUG   ][20278] Reading configuration from /etc/salt/master.d/reactor.conf
2015-08-17 14:11:22,566 [salt.utils.lazy                          ][DEBUG   ][20278] LazyLoaded local_cache.clean_old_jobs
2015-08-17 14:11:22,591 [salt.utils.lazy                          ][DEBUG   ][20278] Could not LazyLoad config.merge
2015-08-17 14:11:22,591 [salt.fileserver                          ][DEBUG   ][20278] Updating roots fileserver cache
2015-08-17 14:11:22,601 [salt.utils.verify                        ][DEBUG   ][20278] This salt-master instance has accepted 84 minion keys.
2015-08-17 14:11:23,967 [salt.master                              ][INFO    ][20368] Clear payload received with command _auth
2015-08-17 14:11:23,967 [salt.master                              ][INFO    ][20368] Authentication request from minion1
2015-08-17 14:11:24,150 [salt.master                              ][INFO    ][20368] Authentication accepted from minion1
2015-08-17 14:11:24,198 [salt.utils.event                         ][DEBUG   ][20368] Sending event - data = {'id': 'minion1', '_stamp': '2015-08-17T18:11:24.198262', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\n[munged]\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
2015-08-17 14:11:24,203 [salt.utils.reactor                       ][DEBUG   ][20325] Gathering reactors for tag salt/auth
2015-08-17 14:11:24,393 [salt.master                              ][INFO    ][20385] Clear payload received with command _auth
2015-08-17 14:11:24,394 [salt.master                              ][INFO    ][20385] Authentication request from minion1
2015-08-17 14:11:24,541 [salt.master                              ][INFO    ][20385] Authentication accepted from minion1
2015-08-17 14:11:24,590 [salt.utils.event                         ][DEBUG   ][20385] Sending event - data = {'id': 'minion1', '_stamp': '2015-08-17T18:11:24.590069', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\n[munged]\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
2015-08-17 14:11:24,595 [salt.utils.reactor                       ][DEBUG   ][20325] Gathering reactors for tag salt/auth
2015-08-17 14:11:24,752 [salt.utils.lazy                          ][DEBUG   ][20379] LazyLoaded jinja.render
2015-08-17 14:11:24,752 [salt.utils.lazy                          ][DEBUG   ][20379] LazyLoaded yaml.render
2015-08-17 14:11:24,754 [salt.utils.lazy                          ][DEBUG   ][20379] LazyLoaded cmd.run
2015-08-17 14:11:24,754 [salt.utils.jinja                         ][DEBUG   ][20379] Jinja search path: ['/var/salt/pillar']
2015-08-17 14:11:24,756 [salt.template                            ][DEBUG   ][20379] Rendered data from file: /var/salt/pillar/top.sls:
base:
  '^.*':
    - match:  pcre
    - defaults
[munged]

2015-08-17 14:11:24,761 [salt.utils.lazy                          ][DEBUG   ][20379] LazyLoaded config.get
2015-08-17 14:11:24,762 [salt.loaded.int.render.yaml              ][DEBUG   ][20379] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('^.*', [OrderedDict([('match', 'pcre')]), 'defaults']), ('[munged]', [OrderedDict([('match', 'pcre')]), 'environments.all.web.billboard']), ('[munged]', [OrderedDict([('match', 'pcre')]), 'environments.test.redis.snapshots']), ('[munged]..', [OrderedDict([('match', 'pcre')]), 'environments.prod.redis.snapshots']), ('[munged]', [OrderedDict([('match', 'pcre')]), 'environments.all.packetbeat.thapi']), ('[munged]', [OrderedDict([('match', 'pcre')]), 'environments.all.packetbeat.snapshots']), ('[munged]', [OrderedDict([('match', 'pcre')]), 'environments.prod.rabbitmq.billboard'])]))])
2015-08-17 14:11:24,762 [salt.utils.jinja                         ][DEBUG   ][20379] Jinja search path: ['/var/salt/pillar']
2015-08-17 14:11:24,764 [salt.template                            ][DEBUG   ][20379] Rendered data from file: /var/salt/pillar/defaults.sls:
salt_warning: |
  ###################### NOTE #########################
  # This file is managed by salt, do not edit locally #
  # All changes will be overwritten by salt!          #
  ###################### NOTE #########################

likewise_requiremembershipof:      '"[munged]"'
likewise_homedirprefix:            [munged]
ad_auth:
  accounts:
    [munged]:  [munged]

raven:
  servers:
    - https://getsentry.com
  public_key: [munged]
  secret_key: [munged]
  project:    salt
  tags:
    - os
    - master
    - saltversion
    - cpuarch

2015-08-17 14:11:24,767 [salt.loaded.int.render.yaml              ][DEBUG   ][20379] Results of YAML rendering: 
OrderedDict([('salt_warning', '###################### NOTE #########################\n# This file is managed by salt, do not edit locally #\n# All changes will be overwritten by salt!          #\n###################### NOTE #########################\n'), ('likewise_requiremembershipof', '[munged]'), ('likewise_homedirprefix', '[munged]'), ('ad_auth', OrderedDict([('accounts', OrderedDict([('[munged]', '[munged]')]))])), ('raven', OrderedDict([('servers', ['https://getsentry.com']), ('public_key', '[munged]'), ('secret_key', '[munged]'), ('project', 'salt'), ('tags', ['os', 'master', 'saltversion', 'cpuarch'])]))])
2015-08-17 14:11:25,550 [salt.utils.event                         ][DEBUG   ][20388] Sending event - data = {'_stamp': '2015-08-17T18:11:25.550113', 'pretag': None, 'cmd': '_minion_event', 'tag': 'minion_start', 'data': 'Minion minion1 started at Mon Aug 17 14:11:25 2015', 'id': 'minion1'}
2015-08-17 14:11:25,552 [salt.utils.reactor                       ][DEBUG   ][20325] Gathering reactors for tag minion_start
2015-08-17 14:11:25,555 [salt.utils.event                         ][DEBUG   ][20403] Sending event - data = {'_stamp': '2015-08-17T18:11:25.554618', 'pretag': None, 'cmd': '_minion_event', 'tag': 'salt/minion/minion1/start', 'data': 'Minion minion1 started at Mon Aug 17 14:11:25 2015', 'id': 'minion1'}
2015-08-17 14:11:25,556 [salt.utils.reactor                       ][DEBUG   ][20325] Gathering reactors for tag salt/minion/minion1/start

Finally, /var/log/salt/minion on the minion shows the following when i shutdown and start back up salt-minion:

2015-08-17 14:29:33,304 [salt.cli.daemons                         ][INFO    ][20391] The salt minion is shut down
2015-08-17 14:29:35,016 [salt.cli.daemons ][INFO    ][20643] Setting up the Salt Minion "minion1"
2015-08-17 14:29:35,103 [salt.cli.daemons                         ][INFO    ][20643] The salt minion is shut down
2015-08-17 14:29:35,106 [salt.cli.daemons                         ][INFO    ][20649] The salt minion is shut down
2015-08-17 14:29:36,137 [salt.loaded.int.module.cmdmod            ][INFO    ][20651] Executing command 'date +%z' in directory '/root'
2015-08-17 14:29:36,148 [salt.minion                              ][INFO    ][20651] Added mine.update to scheduler
2015-08-17 14:29:36,148 [salt.utils.schedule                      ][INFO    ][20651] Updating job settings for scheduled job: __mine_interval
2015-08-17 14:29:36,150 [salt.cli.daemons                         ][INFO    ][20651] The salt minion is starting up
2015-08-17 14:29:36,150 [salt.minion                              ][INFO    ][20651] Minion is starting as user 'root'
2015-08-17 14:29:36,151 [salt.minion                              ][INFO    ][20651] Starting pub socket on ipc:///var/run/salt/minion/minion_event_3f8afd737b_pub.ipc
2015-08-17 14:29:36,151 [salt.minion                              ][INFO    ][20651] Starting pull socket on ipc:///var/run/salt/minion/minion_event_3f8afd737b_pull.ipc
2015-08-17 14:29:36,163 [salt.minion                              ][INFO    ][20651] Minion is ready to receive requests!
2015-08-17 14:29:36,164 [salt.utils.schedule                      ][INFO    ][20651] Running scheduled job: __mine_interval

To prove that the minion should be having beacon events, see the following. There is a new version of sysdig pkg available to be installed, and /var is at 7% utilization (6% is the configured diskusage beacon setting), both of which should trigger beacon events:

# apt-show-versions  | grep sysdig
sysdig/stable-amd64 upgradeable from 0.1.99 to 0.1.102
# salt-call pkg.upgrade_available sysdig
local:
    True
# salt-call pkg.latest_version sysdig
local:
    0.1.102
# salt-call pkg.version sysdig
local:
    0.1.99

# df -h /var
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-varlv  9.2G  580M  8.2G   7% /var
schlagify commented 9 years ago

Looks like the pkg beacon is new in 2015.8.0 so it's not even available in my version (source: https://groups.google.com/forum/#!topic/salt-users/uULaSLYZ69Y) .. guess that one's solved.

jjones028 commented 9 years ago

The diskusage beacon requires the python-psutil package, that seemed to do the trick for me.

jfindlay commented 9 years ago

Probably all that is needed here then is to check for the presence of psutil in the beacon's __virtual__ function.

rallytime commented 9 years ago

It appears that the pkg beacon didn't quite make it into 2015.8 as originally described, and is in fact currently only on the develop branch. I have added two fixes to this issue. The first is to gate the import of psutil and add documentation noting the dependency in the diskusage beacon. The second fix is to add a versionadded directive to the pkg beacon indicating it will be available in Salt Boron.

@schlagify With these changes, do you consider this issue fixed?

schlagify commented 9 years ago

@rallytime Yes, thank you!

rallytime commented 9 years ago

PRs have now been merged. Thanks!