Salt master not properly generating the map

alfredopalhares commented 9 years ago

I have the following map.jinja on a formula

{%
  set package_table = {
    "Debian": {
      "source": "salt://sensu/files/sensu.list",
      "filename": "/etc/apt/sources.list.d/sensu.list",
      "pkgs": [
        "ruby",
        "ruby-dev",
        "git",
        "build-essential"
      ]
    },
    "Redhat": {
      "source": "salt://sensu/files/sensu.repo",
      "filename": "/etc/yum.repos.d/sensu.repo",
      "pkgs": [
        "rubygems",
        "git"
      ]
    },
  }
%}

{%
  set sensu = {
    "client": {
      "name": salt["grains.get"]("fqdn"),
      "address": salt["network.ip_addrs"]("eth0")[0],
      "subscriptions": ["all"]
    },
    "rabbitmq": {
      "host": "1.2.3.4.5",
      "password": "password",
      "port": "1234",
      "user": "user",
      "vhost": "/vhsot"
    }
  }
%}

{% do sensu.update(salt["pillar.get"]("sensu")) %}

{% if grains["os_family"] in package_table %}
  {% set map = package_table[grains["os_family"]] %}
{% endif %}

# vi: set ft=sls :

The sensu["name"] and sensu["address"] do not get set when using salt from a salt-master. While testing with salt-call and kitchen-salt they get they proper value just fine.

Both on the salt masterless, the minions and the master are on Helium above go the detailed versions-report Test box: Always built from scratch

salt-call --versions-report
                  Salt: 2014.7.2
                Python: 2.7.6 (default, Mar 22 2014, 22:59:38)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2014.7.2+ds-1trusty2

The minions:

salt-minion --versions-report
                  Salt: 2014.7.2
                Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2014.7.2+ds-1trusty2

The master: --versions-report is breaking

# salt --versions-report
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 241, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 45, in run
    self.parse_args()
  File "/usr/lib/python2.7/dist-packages/salt/utils/parsers.py", line 126, in parse_args
    self.print_versions_report()
  File "/usr/lib/python2.7/dist-packages/salt/utils/parsers.py", line 193, in print_versions_report
    print('\n'.join(version.versions_report()), file=file)
  File "/usr/lib/python2.7/dist-packages/salt/version.py", line 542, in versions_report
    libs = list(versions_information(include_salt_cloud=include_salt_cloud))
  File "/usr/lib/python2.7/dist-packages/salt/version.py", line 527, in versions_information
    imp = __import__(imp)
  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 75, in <module>
    nacl = _get_nacl()
  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 73, in _get_nacl
    raise OSError(msg)
OSError: Could not locate nacl lib, searched for libsodium.so, libsodium.so.13, libsodium.so.10, libsodium.so.5, libsodium.so.4,  and tweetnacl.so

basepi commented 9 years ago

That versions-report stacktrace is weird. How did you install the master? Seems like we should get that resolved, because you may have something weird going on with your install.

The sensu["name"] and sensu["address"] do not get set when using salt from a salt-master. While testing with salt-call and kitchen-salt they get they proper value just fine.

Do the rest of the values get set correctly? As in, are those the only things missing, but the rest of sensu is there?

when using salt from a salt-master

Can you clarify? What are you doing differently? The map should still execute on the minion side as part of a state run or whatever.

alfredopalhares commented 9 years ago

@baseapi I installed salt-master with the salt-bootstrap script then just apt-get install salt-master

Do the rest of the values get set correctly? As in, are those the only things missing, but the rest of sensu is there?

Yes, these are the only 2 values that do not work for some reason.

Can you clarify? What are you doing differently? The map should still execute on the minion side as part of a state run or whatever.

salt "mynodename" state.sls sensu.client

alfredopalhares commented 9 years ago

After more experimentation. I think there is some pillar data problem that is overriding the features some.

I am using yamlex to merge some pillar data after a few tries I am seeing what the problem seems to reside. Here are some example of the files:

Client state:

#!yamlex
sensu:
  client:
    subscriptions: !aggregate
      - all
  rabbitmq:
    host: "123.456.789.123"
    password: "password"
    port: 5672
    user: "user"
    vhost: "/vhost"

This pillar data would be present on all minions.

The are groups like so:

#!yamlex
sensu:
  client:
    subscriptions: !aggregate
      - mygroup

On the pillar/top.sls file a node can have as many groups at it desires.

  "mynode*":
    - sensu/groups/mygroup1
    - sensu/groups/mygroup2

Different behaviour occurs on the following situations:

All files are used, pillar-source-merging-strategy set to smart Only the last of the subscriptions value of the array stay on the node. both sensu["name"] and sensu["address] do not get properly set. The files all heave the #!yamlex shebang on the top
All files are used, pillar-source-merging-strategy set to aggregate The subscriptions get the proper array values, name and addressdo not get set, also the rabbitmq values used and the default on the sensu-formula, this latest event does not happen when the merge strategy is set to smart.
Only sensu client file is enabled, pillar-source-merging-strategy set to aggregate This time the name and address values get properly set, the subscriptions array get only the values from that file (as its supposed to be). But the rabbitmq value does not get properly set.

basepi commented 9 years ago

Hrm, I don't know yamlex well enough to know whether there's a bug here or not off the top of my head. But thanks for your good detective work, at least we'll know where to look when we find time to dive into this one.

alfredopalhares commented 9 years ago

ping

basepi commented 9 years ago

I have to admit this is not very high priority for us at the moment, @masterkorp. Not very many users are trying to do complex merging with yamlex like this, and we have a lot of bugs which are affecting many users. Just wanted to manage expectations. Sorry for the inconvenience this is causing.

alfredopalhares commented 9 years ago

@basepi Thanks for the explanation.

I don't mind getting my hands on this. If I can get some guidance on the problem.

basepi commented 9 years ago

There's a number of places where the merging happens, like this one: https://github.com/saltstack/salt/blob/develop/salt/pillar/__init__.py#L445-L449

It's possible we need a new merging strategy or something. The fix may be in the file I linked above or in https://github.com/saltstack/salt/blob/develop/salt/utils/dictupdate.py

dkiser commented 9 years ago

@basepi @masterkorp

I just tested out the following additions to dictupdate.py in order to merge lists in the update function. Seemed to work for me for merging lists of sensu subscriptions between different pillar files.

I am not using the yamlex renderer, but defaulting to smart renderer and this test worked out fine. Thoughts? Worth a PR?

def update(dest, upd):
    for key, val in six.iteritems(upd):
        try:
            if isinstance(val, OrderedDict):
                klass = OrderedDict
            elif isinstance(val, list):
                klass = list
            else:
                klass = dict
            dest_subkey = dest.get(key, klass())
        except AttributeError:
            dest_subkey = None
        if isinstance(dest_subkey, collections.Mapping) \
                and isinstance(val, collections.Mapping):
            ret = update(dest_subkey, val)
            dest[key] = ret
        elif isinstance(dest_subkey, list) \
                and isinstance(val, list):
            dest[key] = dest.get(key, []) + val
        elif key:
            dest[key] = upd[key]
    return dest

alfredopalhares commented 9 years ago

:+1: Will test this soon!

dkiser commented 9 years ago

@masterkorp - let me know if it works, and i'll put in a PR. I ran into the same issue trying to merge subscription lists for sensu amongst different pillars as I wanted my pillars to be "modular" when including different subscriptions for different host roles.

dkiser commented 8 years ago

@masterkorp - I finally got around to making a PR for this.

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

saltstack / salt

Salt master not properly generating the map #22241