search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.23k stars 5.49k forks source link

2015.8.3 pillar beacons bugged? #29451

Closed githubcdr closed 8 years ago

githubcdr commented 9 years ago

Hi,

After upgrading to 2015.8.3 I get the following error spamming journald; My beacons are configured in pillar and worker in previous versions.

[CRITICAL] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1699, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 359, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'
[root@flaim ~]# salt skynet.eigenhuis.lan pillar.item beacons --out=yaml
skynet.eigenhuis.lan:
  beacons:
    btmp: {}
    diskusage:
    - /: 80%
    - interval: 43200
    load:
    - 1m:
      - 2.0
      - 3.0
    - 5m:
      - 2.0
      - 3.5
    - 15m:
      - 2.1
      - 3.0
    network_info:
      eth0:
      - type: greater
      - errin: 50
      - errout: 50
      - dropin: 50
      - dropout: 50
    wtmp: {}
[root@flaim ~]# salt-minion --versions-report
Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: 1.0.1
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: 0.6.3
        Tornado: 4.2.1
            ZMQ: 4.1.2
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: 1.3.7
        libnacl: 1.4.3
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.6
   mysql-python: 1.2.5
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: 0.23.2
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: 0.2.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-71-generic
         system: Ubuntu 14.04 trusty

I tried changing the config and clearing the cache but the error stays.

githubcdr commented 9 years ago

might be related to https://github.com/saltstack/salt/issues/27949, but this is from the documentation

jfindlay commented 9 years ago

@githubcdr, thanks for the report.

The-Loeki commented 8 years ago

Confirmed; 

beacons:
  load:
    - interval: 5
    - 1m:
      - 0.0
      - 25.0
    - 5m:
      - 0.0
      - 100.0
    - 15m:
      - 0.0
      - 100.0

borks; it will be passed on to _determine_beacon_config as:

{'load': [{'interval': 5}, {'1m': [0.0, 25.0]}, {'5m': [0.0, 100.0]}, {'15m': [0.0, 100.0]}]}

Changing it to 

beacons:
  load:
    interval: 5
    1m:
      - 0.0
      - 25.0
    5m:
      - 0.0
      - 100.0
    15m:
      - 0.0
      - 100.0

Works around the problem for now, passing on:

{'load': {'5m': [0.0, 100.0], '1m': [0.0, 25.0], 'interval': 5, '15m': [0.0, 100.0]}}

Which works as expected

The-Loeki commented 8 years ago

I have to say I'm not entirely sure what the exact problem is here; the first if in _determine_beacon_config tests whether beacons is a list or dict, but both conditions would bork on this construct, clearly broken.

OTOH wasn't that first syntax suppposed to deliver an ordereddict rather than a list of dicts???

fredrikaverpil commented 8 years ago

But basically, this makes it impossible to use beacons when defined in pillar (which is extremely useful). Am I right?

Has anyone found a fix or a workaround?

The-Loeki commented 8 years ago

@fredrikaverpil negative, just one of the syntaxes is broken. Use the 'alternative' syntax outlined earlier (which is the 'workaround' we use in production).

fredrikaverpil commented 8 years ago

I can't get this to generate anything in the event bus, which is from the docs:

beacons:
  diskusage:
    - /: 5%

...and changing it into the following still doesn't make anything show in the event bus:

beacons:
  diskusage:
    /: 5%

I'm viewing the event bus in order to catch any such event: salt-run state.event pretty=True and the minion has well over 5% disk usage.

asyncmind0 commented 8 years ago

the diskusage beacon expects a list of dictionaries instead of a dict, this is inconsistent with the other beacon configs, although seems to be fixed in the v2016

githubcdr commented 8 years ago

This state generates a unexpected configuration file.

diskusage:
  beacon.present:
    - /: 25%
    - interval: 20

The following file is created when using the beacons.save module, notice the list changed to a dict here.

beacons:
  diskusage:
    /: 25%
    interval: 20

End result is a non working setup, with a critical error as showed below.

May 17 11:07:38 salt-minion[24793]: [CRITICAL] The beacon errored:
May 17 11:07:38 salt-minion[24793]:     mount = diskusage.keys()[0]
May 17 11:07:38 salt-minion[24793]: AttributeError: 'str' object has no attribute 'keys'

This makes the diskusage beacon as described in the docs practically unusable :(

fredrikaverpil commented 8 years ago

I get errors in either case on 2015.8.8.2 with diskusage.beacon.

# /srv/pillar/top.sls

base:
  '*':
    - schedule.sls
    - beacons.sls

(schedule.sls runs just fine)

Alternative 1

# /srv/pillar/beacons.sls

beacons:
  diskusage:
    - /: 5%

Minion log:

2016-05-24 16:07:31,593 [salt.utils.lazy  ][DEBUG   ][10453] LazyLoaded diskusage.beacon
2016-05-24 16:07:31,593 [salt.minion      ][CRITICAL][10453] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

Alternative 2

# /srv/pillar/beacons.sls

beacons:
  diskusage:
    /: 5%

Minion log:

2016-05-24 15:53:38,752 [salt.minion      ][CRITICAL][1079] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 66, in beacon
    mount = diskusage.keys()[0]
AttributeError: 'str' object has no attribute 'keys'

Any idea of a workaround?

cachedout commented 8 years ago

@fredrikaverpil

Looks like the diskusage beacon was completely broken. I have submitted a patch to fix it here: https://github.com/saltstack/salt/pull/33474

Apologies for that.

fredrikaverpil commented 8 years ago

With the new patch, and using the syntax from the diskusage.py contents example, I now get this (as in the OP's traceback):

beacons:
  diskusage:
    - /: 5%
2016-05-24 16:55:01,775 [salt.minion      ][CRITICAL][14648] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'
cachedout commented 8 years ago

@fredrikaverpil Yes, that is the same stacktrace as the original report. This appears to be related to the way the configuration is coming in from pillar.

fredrikaverpil commented 8 years ago

@cachedout Ok!

Also, not sure if related or not, but when you perform a salt 'someminion' beacons.list, you'll only get a proper response if the beacon was defined in the minion config. If defined in the pillar, the response is missing the beacon info. Perhaps that is expected, or should I open up a new issue for that?

cachedout commented 8 years ago

@fredrikaverpil Sounds related to this one. Let's fix this one first and if that issue persists after that, I'll ask you to open a separate issue. Thanks!

fredrikaverpil commented 8 years ago

@cachedout Sounds great, thank you! It's possible that that was intended behavior, as the beacons do turn up when you run a salt 'someminion' pillar.items when the beacons are defined in the pillar.

cachedout commented 8 years ago

OK @githubcdr and @fredrikaverpil give this a try: https://github.com/saltstack/salt/pull/33476

fredrikaverpil commented 8 years ago

@cachedout so I've downloaded the new __init__.py and the new diskusage.py into my /srv/salt/_beacons folder. I performed a salt 'myminion' saltutil.sync_all and I can see the new files being synced:

myminion
    ----------
    beacons:
        - beacons.__init__
        - becons.diskusage

I can also see them being synced in the minion log:

2016-05-25 07:56:36,528 [salt.fileclient  ][INFO    ][10340] Caching directory u'_beacons/' for environment 'base'
2016-05-25 07:56:36,555 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', looking at rel_path u'_beacons/__init__.py' to resolve u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,556 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', ** considering ** path u'/var/cache/salt/minion/files/base/_beacons/__init__.py' to resolve u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,557 [salt.fileclient  ][DEBUG   ][10340] Fetching file from saltenv 'base', ** attempting ** u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,557 [salt.fileclient  ][DEBUG   ][10340] No dest file found
2016-05-25 07:56:36,579 [salt.fileclient  ][INFO    ][10340] Fetching file from saltenv 'base', ** done ** u'_beacons/__init__.py'
2016-05-25 07:56:36,605 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', looking at rel_path u'_beacons/diskusage.py' to resolve u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,605 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', ** considering ** path u'/var/cache/salt/minion/files/base/_beacons/diskusage.py' to resolve u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,606 [salt.fileclient  ][DEBUG   ][10340] Fetching file from saltenv 'base', ** attempting ** u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,606 [salt.fileclient  ][DEBUG   ][10340] No dest file found
2016-05-25 07:56:36,634 [salt.fileclient  ][INFO    ][10340] Fetching file from saltenv 'base', ** done ** u'_beacons/diskusage.py'
2016-05-25 07:56:36,634 [salt.loaded.int.module.saltutil][DEBUG   ][10340] Local cache dir: '/var/cache/salt/minion/files/base/_beacons'
2016-05-25 07:56:36,635 [salt.loaded.int.module.saltutil][INFO    ][10340] Copying '/var/cache/salt/minion/files/base/_beacons/__init__.py' to '/var/cache/salt/minion/extmods/beacons/__init__.py'
2016-05-25 07:56:36,635 [salt.loaded.int.module.saltutil][INFO    ][10340] Copying '/var/cache/salt/minion/files/base/_beacons/diskusage.py' to '/var/cache/salt/minion/extmods/beacons/diskusage.py'

I then restarted the salt service on myminion. But I still get this in my minion log:

2016-05-25 08:04:00,646 [salt.utils.lazy  ][DEBUG   ][11398] LazyLoaded diskusage.beacon
2016-05-25 08:04:00,647 [salt.minion      ][CRITICAL][11398] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

So I checked the diff on /usr/lib/python2.7/site-packages/salt/beacons/__init__.py on my minion against the newly patched /srv/salt/_beacons/__init__.py on the master and the former is the old one, and hasn't been synced properly, it seems. And just like __init__.py, the diskusage.py file in /usr/lib/python2.7/site-packages/salt/beacons/ on the minion is also the old, broken, one.

What am I missing or doing wrong since these new patched files in _beacons doesn't make it onto the minion?

cachedout commented 8 years ago

The __init__.py file won't be synced into the correct location. You need to copy the updated version into /usr/lib/python2.7/site-packages/salt/beacons/ by hand or by using something like Salt's file.managed.

fredrikaverpil commented 8 years ago

@cachedout I copied the files in place by hand, per your instructions, and after restarting the salt-minion service I now get this on the minion:

2016-05-26 10:50:09,252 [salt.minion      ][CRITICAL][7636] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/var/cache/salt/minion/extmods/beacons/diskusage.py", line 69, in beacon
    _current_usage = psutil.disk_usage(mount)
  File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 1787, in disk_usage
    return _psplatform.disk_usage(path)
  File "/usr/lib64/python2.7/site-packages/psutil/_psposix.py", line 122, in disk_usage
    st = os.statvfs(path)
TypeError: must be string, not dict
githubcdr commented 8 years ago

Hi @cachedout,

Some errors here too after replacing "/usr/lib/python2.7/site-packages/salt/beacons/init.py"

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1757, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 365, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 66, in beacon
    mount = diskusage.keys()[0]
zsh 2703 # salt skynet beacons.list
skynet
    beacons:
      diskusage:
        /: 25%

using this state

diskusage:
  beacon.present:
    - /: 25%
    - interval: 20
Salt Version:
           Salt: 2015.8.10

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.24.0
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.11 (default, Mar 31 2016, 06:18:34)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4
           cffi: 1.6.0
       cherrypy: 5.4.0
       dateutil: 2.5.3
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: 0.24.0
        libnacl: 1.4.5
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4

System Versions:
           dist:
        machine: x86_64
        release: 4.5.4-1-ARCH
githubcdr commented 8 years ago

@cachedout is there anything I can do to test this further? I got the feeling that a big part of the beacon functionality is broken on a low level, preventing any use.

It would be really nice if beacons received some love, because this is what makes salt dynamic compared to other systems.

githubcdr commented 8 years ago

Hi,

Sorry for re-opening this issue, but it seems the beacon emit function causes a critical error, the config part is loading correctly now.

I see this in /var/log/salt/minion

2016-06-09 21:51:29,089 [salt.minion      ][CRITICAL][19796] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 2022, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 380, in process_beacons
    return self.beacons.process(b_conf)  # pylint: disable=no-member
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 72, in beacon
    mount = diskusage.keys()[0]
AttributeError: 'str' object has no attribute 'keys'

I removed all .pyc leftovers and double checked the logs.