search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.16k stars 5.48k forks source link

grains.filter_by grain value doesn't support list #39709

Closed oudemen closed 4 years ago

oudemen commented 7 years ago

Description of Issue/Question

grains.filter_by grain value doesn't support a list when use list ,return AttributeError: 'list' object has no attribute 'split'

Setup

map.jinja:

{% set fluentd = salt['grains.filter_by']({
    'Ubuntu-12.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'Ubuntu-14.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'Ubuntu-16.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'CentOS-6': {'pkg': 'td-agent_2.3.4-0.el6.amd64.deb'},
    'CentOS-7': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'Debian': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'RedHat': {'pkg': 'td-agent_2.3.4-0.elrh.amd64.deb'},
},grain=['osfinger','os_family'], default='RedHat') %}

init.sls:

{% from "install/fluentd/map.jinja" import fluentd with context %}
/tmp/salt.debug:
  file.managed:
    - contents: |
    {%- for k,v in fluentd.items() %}
        {{ k }} => {{ v }}
    {%- endfor %}

Steps to Reproduce Issue

salt -S 192.168.12.192 state.sls install.fluentd 

192.168.12.192:
    Data failed to compile:
----------
    Rendering SLS 'base:install.fluentd' failed: Jinja error: 'list' object has no attribute 'split'
/var/cache/salt/minion/files/base/install/fluentd/map.jinja(1):
---
{% set fluentd = salt['grains.filter_by']({    <======================
    'Ubuntu-12.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'Ubuntu-14.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'Ubuntu-16.04': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
    'CentOS-6': {'pkg': 'td-agent_2.3.4-0.el6.amd64.deb'},
    'CentOS-7': {'pkg': 'td-agent_2.3.4-0.precise.amd64.deb'},
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 669, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 1, in top-level template code
  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 713, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 769, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/install/fluentd/map.jinja", line 1, in top-level template code
    {% set fluentd = salt['grains.filter_by']({
  File "/usr/lib/python2.6/site-packages/salt/modules/grains.py", line 555, in filter_by
    val = salt.utils.traverse_dict_and_list(__grains__, grain, [])
  File "/usr/lib/python2.6/site-packages/salt/utils/__init__.py", line 1547, in traverse_dict_and_list
    for each in key.split(delimiter):
AttributeError: 'list' object has no attribute 'split'

Versions Report

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.19.0-25-generic
         system: Linux
        version: Ubuntu 14.04 trusty
gtmanfred commented 7 years ago

What exactly are you trying to do here?

Have it check for osfinger, and if it is not there, then try os_family?

gtmanfred commented 7 years ago

Can you try making it a comma seperated list?

THis could just be a documentation issue

try grain='osfinger,os_family'

ThePsyjo commented 7 years ago

Same problem at my side, doing grain='osfinger,os_family' does not work either, it will fall back to default then.

The grain value could be a list. The function will return the lookup_dict value for a first found item in the list matching one of the lookup_dict keys.

I cannot find evidence that this is possible because this value is undergone split if it is str or not. A traversal is applied to this string, but this will give you an empty list if you do 'osfinger:os_family' because there is no dict behind osfinger that holds the key os_family. And yes, the only delimiter you can apply here is ':'.

timwsuqld commented 6 years ago

I also have this issue, it doesn't seem to work according to the docs

ldav1s commented 5 years ago

Here's a command line version that may be the same or related issue. My salt looks like:

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.5.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-138-generic
         system: Linux
        version: Ubuntu 16.04 xenial

I'm trying to filter the disks grains says I have:

$ sudo salt-call --local grains.get disks
local:
    - sdb
    - loop0
    - loop1
    - loop2
    - loop3
    - loop4
    - loop5
    - loop6
    - loop7

OK, lets filter on loop disks:

$ sudo salt-call --local grains.filter_by '{lo*: x}' disks
local:
    x

I'm expecting an "x" for each loop disk, I guess. Picking and choosing is no help either:

$ sudo salt-call --local grains.filter_by '{"loop1": y}' disks
local:
    y

$ sudo salt-call --local grains.filter_by '{"loop0": x, "loop1": y}' disks
local:
    x
gtmanfred commented 5 years ago

@saltstack/team-triage can yall take a look?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

timwsuqld commented 4 years ago

Still an issue

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

timwsuqld commented 4 years ago

@SaltStackSupport Can we please have the stalebot not be so aggressive. 30 days is way too aggressive

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

timwsuqld commented 4 years ago

Not stale

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

sagetherage commented 4 years ago

@timwsuqld is this an issue on newer versions besides 2018.3, as well?

timwsuqld commented 4 years ago

Still relevant. I can replicate it as follows:

test_grains.jinja

{% set test_grains = salt['grains.filter_by']({
    'Ubuntu-16.04': {
        'apacheuser': 'www-data',
    },
    'Ubuntu-18.04': {
        'apacheuser': 'www-data',
    },
    'RedHat': {
        'apacheuser': 'apache',
    },
}, grain=['os_finger', 'osfamily']) %}

test_grains.sls

{% from "test_grains.jinja" import test_grains with context %}
echo {{ test_grains.apacheuser }}:
  cmd.run
# salt-call --local --file-root=./states/ state.sls test_grains test=True
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 443, in render_jinja_tmpl
    trace=tracestr)
SaltRenderError: Jinja error: 'list' object has no attribute 'split'
/var/cache/salt/minion/files/base/test_grains.jinja(1):
---
{% set test_grains = salt['grains.filter_by']({    <======================
    'Ubuntu-16.04': {
        'apacheuser': 'www-data',
    },
    'Ubuntu-18.04': {
        'apacheuser': 'www-data',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1152, in __init__
    body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/test_grains.jinja", line 1, in top-level template code
    {% set test_grains = salt['grains.filter_by']({
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 600, in filter_by
    base=base)
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 504, in filter_by
    val = traverse_dict_and_list(traverse, lookup, [])
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 578, in traverse_dict_and_list
    for each in key.split(delimiter):
AttributeError: 'list' object has no attribute 'split'

[CRITICAL] Rendering SLS 'base:test_grains' failed: Jinja error: 'list' object has no attribute 'split'
/var/cache/salt/minion/files/base/test_grains.jinja(1):
---
{% set test_grains = salt['grains.filter_by']({    <======================
    'Ubuntu-16.04': {
        'apacheuser': 'www-data',
    },
    'Ubuntu-18.04': {
        'apacheuser': 'www-data',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1152, in __init__
    body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/test_grains.jinja", line 1, in top-level template code
    {% set test_grains = salt['grains.filter_by']({
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 600, in filter_by
    base=base)
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 504, in filter_by
    val = traverse_dict_and_list(traverse, lookup, [])
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 578, in traverse_dict_and_list
    for each in key.split(delimiter):
AttributeError: 'list' object has no attribute 'split'

local:
    Data failed to compile:
----------
    Rendering SLS 'base:test_grains' failed: Jinja error: 'list' object has no attribute 'split'
/var/cache/salt/minion/files/base/test_grains.jinja(1):
---
{% set test_grains = salt['grains.filter_by']({    <======================
    'Ubuntu-16.04': {
        'apacheuser': 'www-data',
    },
    'Ubuntu-18.04': {
        'apacheuser': 'www-data',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1152, in __init__
    body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/test_grains.jinja", line 1, in top-level template code
    {% set test_grains = salt['grains.filter_by']({
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 600, in filter_by
    base=base)
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 504, in filter_by
    val = traverse_dict_and_list(traverse, lookup, [])
  File "/usr/lib/python2.7/dist-packages/salt/utils/data.py", line 578, in traverse_dict_and_list
    for each in key.split(delimiter):
AttributeError: 'list' object has no attribute 'split'

The part in the docs that suggests this should be possible: https://docs.saltstack.com/en/master/ref/modules/all/salt.modules.grains.html#salt.modules.grains.filter_by

grain --

The name of a grain to match with the current system's grains. For example, the value of the "os_family" grain for the current system could be used to pull values from the lookup_dict dictionary.

Changed in version 2016.11.0: The grain value could be a list. The function will return the lookup_dict value for a first found item in the list matching one of the lookup_dict keys.

Trying grain='os_family,osfinger' also does not work.

timwsuqld commented 4 years ago

For completeness, the current version I'm running all that on.

$ salt-call --versions
Salt Version:
           Salt: 3000.1

Dependency Versions:
           cffi: 1.12.3
       cherrypy: Not Installed
       dateutil: 2.4.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Nov  7 2019, 10:07:09)
   python-gnupg: 0.4.1
         PyYAML: 3.11
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-96-generic
         system: Linux
        version: Ubuntu 18.04 bionic
sagetherage commented 4 years ago

thank you @timwsuqld we have disabled stale bot so that won't be an issue here any longer - I am kicking this back to triage and then will follow up tomorrow.

sagetherage commented 4 years ago

I guess I kicked back too many issues to "re-triage" since this and a handful still are not. I will follow up with the person on triage and review this again, tomorrow, Friday 2020-04-17

sagetherage commented 4 years ago

thank you @Akm0d !!