search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
13.98k stars 5.47k forks source link

[BUG] can't use grains in pillar anymore #66669

Closed silenius closed 4 days ago

silenius commented 4 days ago

Description Accessing grains in pillar result in a rendering error on the master:

Setup

jcigar@salt pillar]$ salt pg15-slave.prod.lan grains.get ip4_interfaces:vlan51:0
pg15-slave.prod.lan:
    192.168.10.63

[jcigar@salt pillar]$ pwd
/usr/local/etc/salt/pillar

[jcigar@salt pillar]$ cat pgbouncer/pg15_prod_slave.sls 
#!yaml|gpg

include:
  - pgbouncer.pg15_prod

pgbouncer:
  config:
    present:
      pgbouncer:
        listen_addr: {{ salt.grains.get('ip4_interfaces:vlan51:0', 'localhost') }}
      databases:
        '*': host=pg15-slave.prod.lan auth_user=pgbouncer

jcigar@salt pillar]$ salt pg15-slave.prod.lan saltutil.refresh_pillar           
pg15-slave.prod.lan:
    True

However, I'm getting a renderer error:

2024-06-26 11:33:32,019 [salt.master                                                         :284 ][INFO    ][18655] User jcigar Published command saltutil.refresh_pillar with jid 20240626093332018496
2024-06-26 11:33:32,329 [salt.utils.job                                                      :284 ][INFO    ][18656] Got return from pg15-slave.prod.lan for job 20240626093332018496
2024-06-26 11:33:34,491 [salt.pillar                                                         :284 ][CRITICAL][18654] Rendering SLS 'pgbouncer.pg15_prod_slave' failed, render error:
while parsing a flow mapping                                                                                                                                                          
  in "<unicode string>", line 10, column 23:               
            listen_addr: {{ salt.grains.get('ip4_interface ... 
                          ^                                                                                                                                                           
expected ',' or '}', but got '<scalar>'             
  in "<unicode string>", line 10, column 79:        
     ... nterfaces:vlan51:0', 'localhost') }}                                          
                                         ^                                       
Traceback (most recent call last):                                               
  File "/usr/local/lib/python3.11/site-packages/salt/renderers/yaml.py", line 62, in render 
    data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/utils/yamlloader.py", line 158, in load
    return yaml.load(stream, Loader=Loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                  
  File "/usr/local/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()                            
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()           
           ^^^^^^^^^^^^^^^^^^^^^^            
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()                                                                                                                                                                                                                                                                                                                                       
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 129, in compose_mapping_node
    item_key = self.compose_node(node, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node 
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
                         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 549, in parse_flow_mapping_key
    raise ParserError("while parsing a flow mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a flow mapping
  in "<unicode string>", line 10, column 23:
            listen_addr: {{ salt.grains.get('ip4_interface ... 
                          ^
expected ',' or '}', but got '<scalar>'
  in "<unicode string>", line 10, column 79:
     ... nterfaces:vlan51:0', 'localhost') }}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/salt/pillar/__init__.py", line 947, in render_pstate
    state = compile_template(
            ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/template.py", line 100, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/loader/lazy.py", line 159, in __call__ 
    ret = self.loader.run(run_func, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/loader/lazy.py", line 1260, in _run_as 
    return _func_or_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/salt/renderers/yaml.py", line 68, in render 
    raise SaltRenderError(exc)
salt.exceptions.SaltRenderError: while parsing a flow mapping
  in "<unicode string>", line 10, column 23:
            listen_addr: {{ salt.grains.get('ip4_interface ... 
                          ^
expected ',' or '}', but got '<scalar>'
  in "<unicode string>", line 10, column 79:
     ... nterfaces:vlan51:0', 'localhost') }}
                                         ^
2024-06-26 11:33:34,493 [salt.pillar                                                         :284 ][CRITICAL][18654] Pillar render error: Rendering SLS 'pgbouncer.pg15_prod_slave' failed. Please see master log for details.

This is on FreeBSD 13.3, master and minion are running in jails

Versions Report

Master:

[jcigar@salt pillar]$ salt --versions-report
Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.11.9 (main, May 24 2024, 09:56:37) [Clang 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g600970

Dependency Versions:
          cffi: 1.16.0
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.8
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.2
     pycparser: 2.22
      pycrypto: Not Installed
  pycryptodome: 3.20.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.5

System Versions:
          dist: freebsd 13.3 
        locale: utf-8
       machine: amd64
       release: 13.3-RELEASE-p1
        system: FreeBSD
       version: FreeBSD 13.3

Minion:

root@pg15-slave:/usr/local/etc # salt --versions-report
Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.11.9 (main, May 24 2024, 09:56:37) [Clang 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g600970

Dependency Versions:
          cffi: 1.16.0
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.8
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.2
     pycparser: 2.22
      pycrypto: Not Installed
  pycryptodome: 3.20.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.5

System Versions:
          dist: freebsd 13.3 
        locale: utf-8
       machine: amd64
       release: 13.3-RELEASE-p1
        system: FreeBSD
       version: FreeBSD 13.3
silenius commented 4 days ago

The issue is with #!yaml|gpg, it works if I remove it... but as I have GPG encoded data in my pillar I absolutely need it

lkubb commented 4 days ago

You're not parsing the Jinja at all, just the YAML. Try it with #!jinja|yaml|gpg.

silenius commented 4 days ago

It works with #!jinja|yaml|gpg .. !

Thank you very much