Undefined jinja variable

[gl1ch]

Hi, after upgrading to 0.17 I started getting an error "Undefined jinja variable; line 353 in template" when I try and set the following variable using grains. This worked before, did something change?

{%- set hostname = grains['id'] %}

snmp-package:
  pkg:
    - name: {{ pillar['packages']['snmp'] }}
    - installed

snmp-service:
  service:
    - name: {{ pillar['packages']['snmp-service'] }}
    - running
    - watch:
      - pkg: snmp-package
      - file: /etc/snmp/snmpd.conf
  file.managed:
    - name: /etc/snmp/snmpd.conf
    - user: root
    - group: root
    - mode: 600
    - source: salt://linux-server/snmp/etc/snmpd.jinja
    - template: jinja
    - require:
      - pkg: snmp-package

{% if 'postfix' in pillar['servers'][hostname]['roles'] %}
/etc/snmp/snmp_post_queue.sh:
  file.managed:
    - user: root
    - group: root
    - mode: 655
    - source: salt://linux-server/snmp/etc/snmp_post_queue.sh
    - require:
      - pkg: snmp-package
{% endif %}

am-vm-salt01:
    Data failed to compile:
----------
    Rendering SLS linux-server.snmp failed, render error: Undefined jinja variable; line 353 in template
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1929, in render_state
    rendered_sls=mods
  File "/usr/lib/python2.7/dist-packages/salt/template.py", line 69, in compile_template
    ret = render(input_data, env, sls, **render_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/renderers/jinja.py", line 42, in render
    tmp_data.get('data', 'Unknown render error in jinja renderer')
SaltRenderError: Undefined jinja variable; line 353 in template

[basepi]

It's possible there is a grains issue (see #7624). What OS are you using?

[gl1ch]

Currently using debian 7.1.0 on master and minion. Also it appears that "salt 'server*' grains.get id" is pulling the correct information so the module seems to be working.

[basepi]

So is line 353 the grains['id'] line?

[gl1ch]

No, the file doesnt have that many lines. To test I took my snmpd.jinja template down to the following two lines and it fails with the error above:

{%- set hostname = grains['id'] %}
agentAddress udp:127.0.0.1:161,udp:{{ pillar['servers'][hostname]['network_adapters']['eth0']['ip'] }}:161

but if I change my snmpd.jinja template to the following two lines it works:

{%- set hostname = grains['id'] %}
{{ hostname }}

[gl1ch]

Also here is the error message from a debug output on the minion when only using the two lines above:

[CRITICAL] Rendering SLS linux-server.snmp failed, render error: Undefined jinja variable; line 353 in template Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1929, in render_state rendered_sls=mods File "/usr/lib/python2.7/dist-packages/salt/template.py", line 69, in compile_template ret = render(input_data, env, sls, **render_kwargs) File "/usr/lib/python2.7/dist-packages/salt/renderers/jinja.py", line 42, in render tmp_data.get('data', 'Unknown render error in jinja renderer')

[basepi]

Was this happening on a previous version of salt or did it just start on 0.17?

[basepi]

And if it worked on previous versions, from what version did you upgrade?

[gl1ch]

This was not happening until upgrading to 0.17 as far as I know. I was using 0.16.4 before and I am still able to run these states on minions running 0.16.x (will double check the versions tomorrow).

[oba11]

Same thing happened to me too after upgrading to 0.17. it doesn't happen with 0.16.4 if I downgrade it.

[parruc]

I think my problem could be related to this one:

After the update to 0.17 I get "Undefined jinja variable" on all the "file.managed" entries that try to use variables passed through the "defaults" or "context" fields. It worked on 0.16.4.

[kpostrup]

I had the same problem when using pillars, e.g. {{pillar['mysql']['listen-addr']}}. Found out it was casued by the minion ID changing, from FQDN to short hostnames. Then the pillars where no longer accessible by that minion.

[parruc]

In my case the minion can access the pillar data from the sls but stops working in "file.managed" statements that use defaults or context (not using grains there)

[basepi]

Ah, now that @kpostrup posted, I think that might be the problem for most of you. There was a regression between 0.16.4 and 0.17.0 that made it so that /etc/hostname would be used to guess minion ID instead of socket.getfqdn(). So if your minion ID changed, then the targeting in your pillar may be broken, which would result in minions not having their pillar. Can anyone verify if this is their problem or not?

Note the minion IDs will revert back to socket.getfqdn() for 0.17.1, and we also will cache minion IDs so they don't change unexpectedly again.

[gl1ch]

I dont think it is the hostname changing. To validate it I created a pillar for "hostname" and another for "hostname.domain" and the error message I still get the undefined variable.

[borgstrom]

FWIW, I just ran into this issue and it turns out my problem was that I was importing a macro from a .sls file that used {{ pillar }} and I was not importing it with context. I moved the macro into another file for organizational purposes, but I'm sure adding with context to the import would've worked too.

[basepi]

@gl1ch can you try the with context and @borgstrom mentioned? Curious if that's your problem.

[gl1ch]

Sure, @basepi but I think I am confused as to how I would use the 'with context' in my state. Currently I am declaring the variable in my state file and using that same variable in that same state file. In the past I have used contexts to pass variables to other files but this is all in the same file. Am I missing something?

[basepi]

Hehe, I also wasn't sure how to apply the with context that @borgstrom mentioned -- I haven't used the include syntax very extensively. And I can't find it in my cursory docs search. Can you elaborate, @borgstrom?

[borgstrom]

with context wont apply in @gl1ch's case.

I was just chiming in because I got the exact same traceback (render error: Undefined jinja variable; line 353 in template) and found this ticket. This way if anyone ends up on this bug with searching and their problem is an import issue it might help them solve it.

Here's further reading on how with context works with jinja includes: http://jinja.pocoo.org/docs/templates/#import-visibility

Essentially, if you use pillar or any other builtin variable in a file that you include in another file you need to use with context or those builtin variables won't be available.

[basepi]

Oh, gotcha. Thanks for the explanation.

@gl1ch Have you had a chance to test on the newly-released 0.17.1?

[gl1ch]

@basepi Yup, still happening though.

[ranl]

I have a similar issue maybe it's related (this happen when I upgraded from 0.17.0 to 0.17.1 )

ran.sls

/root/jinja:
  file.managed:
    - source: salt://ran.jinja
    - template: jinja
    - defaults:
      users: {{ salt['config.get']('users', {}) }}

pillar.sls

users:
  solr-master:
    user: solr
    role: solr-user
    realm: Solr

ran.jinja

{% for user in users %}
{{ users[user]['user'] }}
{% endfor %}

when I run the ran.sls state I'm getting

minion:
----------
    State: - file
    Name:      /root/jinja
    Function:  managed
        Result:    False
        Comment:   Undefined jinja variable; line 2 in template

---
{% for user in users %}
{{ users[user]['user'] }}    <======================
{% endfor %}
---
        Changes:

after a little debugging I've noticed that the users jinja variable is not being translated properly it should be like the pillar.sls but it's actually

users:
  solr-master: "{ 'realm': 'Solr', 'role': 'solr-user', 'user': 'solr' }"

it made the solr-master value into a sting instead of a dict for now I've downgraded to 0.17.0 and its working ...

[basepi]

@ranl Does it show that same way in salt '*' pillar.items?

@gl1ch Good to know. Thanks for the update.

[amahon]

Experiencing a very similar problem to both @gl1ch and @ranl. Can include source if it would be helpful, but it is pretty similar to what's posted above.

FWIW, running 0.17.1-2 on a pretty clean Arch system.

[ranl]

@basepi , yes pillar.items is working correctly this looks like a problem in the jinja render

[jslatts]

+1 here. Seeing that exact issue with this code:

{% for name,config in pillar['password_protected_routes'].iteritems() %}
location ~/{{ config['route'] }} {
  auth_basic "Restricted";
  auth_basic_user_file /etc/nginx/htpasswd/{{ name }};
}
{% endfor %}

Error message says that name cannot be found in ninja template variables.

[basepi]

Ninja template variables? Those sound awesome. ;)

So it's probably somewhere in the renderer system, not pillar itself. Good to know.

[basepi]

Hrm, not seeing any recent changes in the renderers directory that would cause this. I'll keep looking.

[jslatts]

@basepi I'm typing one-handed here :) But these template variables are rather ninja-ish in that they seem to be hiding in plain sight!

[valentinbud]

+1 on this matter.

I have also posted a message on the mailing list about it 1. There is something fishy with salt and python-jinja2. I am running version 0.17.1 on Debian Wheezy boxes. My dev box has unstable repository activated and python-jinja2 installed from unstable at version 2.7.1-1. On this box the states are working. On the box where python-jinja2 is installed from stable and at version 2.6-1 the same state is failing.

Any ideas?

[whiteinge]

Update: the following test is incorrect.

Something odd is definitely going on here. The data is getting cast to a string at some point. Here's a shell session with @ranl 's Pillar data from a few comments above:

>>> JINJA('''{{ pillar['users'] }}''')
u"OrderedDict([('solr-master', OrderedDict([('user', 'solr'), ('role', 'solr-user'), ('realm', 'Solr')]))])"
>>> JINJA('''{{ pillar['users'].keys() }}''')
u"['solr-master']"

[basepi]

@ranl Since your issue appeared in 0.17.1 I think it's probably a different issue. Looks like it may be the same issue as #8068, in fact.

[basepi]

I lied. I don't think #8068 is the same as your problem @ranl. However, I do think your problem is probably separate in some way from the original problem for this issue, as yours shows up only 0.17.1.

[gl1ch]

Sorry to but in but I was just wondering if anyone could reproduce what I was seeing. I also noticed that my previous example wasnt complete as it didnt have the associated pillar data so I thought I would add it:

JINJA Template
{%- set hostname = grains['id'] %}
agentAddress udp:127.0.0.1:161,udp:{{ pillar['servers'][hostname]['network_adapters']['eth0']['ip'] }}:161

PILLAR (servers.sls):
servers:
  ubuntu01:
    domain: domain.com
    gateway: 192.168.1.1
    gwdev: eth0
    dns:
      - 192.168.1.10
      - 192.168.1.11
    network_adapters:
      eth0: {
      en: 'True',
      ip: 192.168.1.5,
      sn: 255.255.255.0,
      nw: 192.168.1.0,
      bc: 192.168.1.255 }

[whiteinge]

@gl1ch Thanks that was helpful. I can reproduce the error.

[basepi]

@whiteinge @ranl Let's move the discussion of the pillar string-casting to #8079. I think it's different from the original issue on this thread.

[ranl]

@basepi ok fine by me, I'll keep an eye on #8079

[ranl]

Guys, please look at #8245 this might help some of you

[diegows]

Hi, As workaround, I'm using allow_undefined: true in the minion config. No the best option for sure, but at least helps me to finish the development of my states :)

[cowmix]

Using the tag 0.17.2 I'm getting the same error.

[nterupt]

I am seeing this issue as well. I am trying to coordinate SSH keys between some servers to set up an SSH tunnel.
This is the area generating the error:

/sbin/restorecon -v /home/autossh/.ssh/known_hosts:
  cmd.wait:
    - watch:
      - ssh_auth: {{ pillar['remote_target_public_autossh_key_rsa'] }} <======================

I have run: salt 'server_name' pillar.items and this entry correctly appears.

I have also run: salt 'server_name' pillar.get remote_target_public_autossh_key_rsa and get the correct entry back.

I am really starting to believe I am going nuts here. Most of the issues I see on here seem to be related to complex pillar hierarchies but this seems pretty simple to me.

Edit: This issue is occuring with 0.17.2 on both the master and minion

[basepi]

As #8245 stated, there seems to be a bug in Jinja which is causing it to point at the wrong point in the template. That may not be the area that's actually generating the error. Can you try removing all your other states temporarily and see if that one still fails, with only the one jinja variable?

[nterupt]

Thanks very much for your advice. Your intuition was correct and the error log was incorrect as mentioned in https://github.com/saltstack/salt/issues/8245. When I reduced the SLS file down to one pillar, the root cause ended up being a malformed pillar string that caused a compile error.

[basepi]

That's really good to know. I'm thinking most of the currently-reported jinja problems are related to this misinformation from jinja. Those who are participating on this thread should remove various jinja references in their sls files until the errors go away to verify that you know where the error is originating.

[paulcollinsiii]

@basepi So I have a smaller test case that I think produces this issue.

{% set some_file = '/opt/.test' %}

{% if salt['file.missing'](some_file) %}
another file:
  file.managed:
    - name: {{ some_file }}_extra
{% endif %}

a test file:
  file.managed:
    - name: {{ some_file }}

When that if statement is in there I get a render error with the error pointing at the if line. Remove the if line and the state works (creating both those test files). I'm not sure this is exactly like @gl1ch 's problem, but it appears that when you try to use a context variable that's been defined in the template as an argument to a function call that touches something from salt, it explodes.

This is on Debian 7.2 with salt pkg version 0.17.2-2~bpo70+1~dst.1

[basepi]

What's the error you're getting? With that exact example, I'm getting Jinja variable 'dict object' has no attribute 'file.missing'; -- this is expected, since file.missing is part of the file state, not the module. The salt dictionary in the templating engine only contains modules, not states.

Changing your above example to the following works for me:

{% set some_file = '/opt/.test' %}

{% if not salt['file.file_exists'](some_file) %}
another file:
  file.managed:
    - name: {{ some_file }}_extra
{% endif %}

a test file:
  file.managed:
    - name: {{ some_file }}

[paulcollinsiii]

Gah. That would explain it. I had seen that {% if ... %} example off the mailing list so I had assumed it was just correct. I was getting undefined variable but it wasn't telling me which part was undefined. Either that or I failed to read the entire error message. I'll check tonight and report back =)

[paulcollinsiii]

For reference the error I was getting was

hsvr1.petyr.dyndns.org:
    Data failed to compile:
----------
    Rendering SLS saltbox.test failed, render error: Undefined jinja variable; line 3 in template

---
{% set is_installed_file = '/opt/.test' %}

{% if salt['file.missing'](is_installed_file) %}    <======================
another file:
  file.managed:
    - name: {{ is_installed_file}}_extra
{% endif %}

[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1964, in render_state
    rendered_sls=mods
  File "/usr/lib/python2.7/dist-packages/salt/template.py", line 69, in compile_template
    ret = render(input_data, env, sls, **render_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/renderers/jinja.py", line 42, in render
    tmp_data.get('data', 'Unknown render error in jinja renderer')
SaltRenderError: Undefined jinja variable; line 3 in template

---
{% set is_installed_file = '/opt/.test' %}

{% if salt['file.missing'](is_installed_file) %}    <======================
another file:
  file.managed:
    - name: {{ is_installed_file}}_extra
{% endif %}

[...]
---

Is there some setting I missed to get Jinja to tell me that the dict key was the error? Changing my example to your code did work for me as well. Thanks.

[basepi]

Hrm, that is strange. Mine gave me the error I posted above, pointing exactly to the dict issue. It's possible we've improved these error messages on the current develop branch -- that's where I was testing.

[basepi]

Anyway, it really looks like the errors on this thread are related to jinja just pointing at the wrong line (or giving a strange error), so I'm going to close this issue and we can continue those specific issues on #8245.