2016.3.1 grain value in jinja is empty while has a value when requested on the CLI

[UtahDave]

Description of Issue/Question

When I run salt '<minion id>' grains.get 'ip4_interfaces:eth1' on the salt master I get a value back. When I make the same call within a jinja managed file I get an empty list.

Setup

cat /srv/salt/pkgrepo/apache/init.sls

httpd_stuff:
  pkg.installed:
    - pkgs:
      - httpd
      - mod_ssl
  file.managed:
    - name: /etc/httpd/conf/httpd.conf
    - source: salt://pkgrepo/apache/httpd.conf
    - template: jinja
    - user: root
    - group: root
    - mode: 644

cat /srv/salt/pkgrepo/apache/httpd.conf

<-- snip -->

Listen {{ salt['grains.get']('ip4_interfaces:eth1').pop() }}:80

<-- snip -->

Steps to Reproduce Issue

[root@qa-master ~]# salt '*backend03' grains.get 'ip4_interfaces:eth1'
pkgrepo-backend03:
    - 10.xxx.xxx.xxx

[root@qa-master ~]# salt '*backend03' state.sls pkgrepo.apache
pkgrepo-backend03:
----------
          ID: httpd_stuff
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:39:46.112488
    Duration: 1717.498 ms
     Changes:   
----------
          ID: httpd_stuff
    Function: file.managed
        Name: /etc/httpd/conf/httpd.conf
      Result: False
     Comment: Unable to manage file: Jinja error: pop from empty list
              Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 366, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 43, in top-level template code
              IndexError: pop from empty list

              ; line 43

              ---
              [...]
              # Change this to Listen on specific IP addresses as shown below to 
              # prevent Apache from glomming onto all bound IP addresses.
              #

              #Listen 80
              Listen {{ salt['grains.get']('ip4_interfaces:eth1').pop() }}:80    <======================

              #
              # Dynamic Shared Object (DSO) Support
              #
              # To be able to use the functionality of a module which was built as a DSO you
              [...]
              ---
     Started: 15:39:47.833510
    Duration: 306.052 ms
     Changes:   

Versions Report

[root@qa-master ~]# salt '*backend03' test.versions_report
pkgrepo-backend03:
    Salt Version:
               Salt: 2016.3.1

    Dependency Versions:
               cffi: Not Installed
           cherrypy: Not Installed
           dateutil: Not Installed
              gitdb: Not Installed
          gitpython: Not Installed
              ioflo: Not Installed
             Jinja2: 2.7.2
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: 0.21.1
               Mako: Not Installed
       msgpack-pure: Not Installed
     msgpack-python: 0.4.7
       mysql-python: Not Installed
          pycparser: Not Installed
           pycrypto: 2.6.1
             pygit2: Not Installed
             Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
       python-gnupg: Not Installed
             PyYAML: 3.11
              PyZMQ: 14.7.0
               RAET: Not Installed
              smmap: Not Installed
            timelib: Not Installed
            Tornado: 4.2.1
                ZMQ: 4.0.5

    System Versions:
               dist: centos 7.2.1511 Core
            machine: x86_64
            release: 3.10.0-123.8.1.el7.x86_64
             system: Linux
            version: CentOS Linux 7.2.1511 Core

[UtahDave]

@whiteinge asked me to test if the grains.get works in the sls file directly and it does.

I changed to this:

httpd_stuff:
  pkg.installed:
    - pkgs:
      - httpd
      - mod_ssl
  file.managed:
    - name: /etc/httpd/conf/httpd.conf
    - source: salt://pkgrepo/apache/httpd.conf
    - template: jinja
    - user: root
    - group: root
    - mode: 644
    - mystuff: {{ salt['grains.get']('ip4_interfaces:eth1').pop() }}

and mystuff was assigned the ip address as the value correctly

[UtahDave]

OK, for completeness, here's the workaround:

cat /srv/salt/pkgrepo/apache/init.sls

httpd_stuff:
  pkg.installed:
    - pkgs:
      - httpd
      - mod_ssl
  file.managed:
    - name: /etc/httpd/conf/httpd.conf
    - source: salt://pkgrepo/apache/httpd.conf
    - template: jinja
    - user: root
    - group: root
    - mode: 644 
    - context:
        eth1_ip: {{ salt['grains.get']('ip4_interfaces:eth1').pop() }}

cat /srv/salt/pkgrepo/apache/httpd.conf

<-- snip -->

Listen {{ eth1_ip }}:80

<-- snip -->

When I apply the state with the above changes it works correctly. I'm not sure why the grains.get works in the sls file and not within the jinja context in the httpd.conf

[cachedout]

This is not a bug in Salt. It's an error in your template. It's easy to do, though, since this is a subtle side-effect in Jinja2.

What you're seeing here is a single-element list being returned from this grain. When you call pop(), it modifies the list in place before evaluating it, so it thinks the list is empty. What you want to do here instead is either access the element directly or iterate over the list.

[whiteinge]

Whoops, I overlooked that. Good catch, Mike. Modifying the loaded grains object is definitely bad juju. This is better done using the | first() filter in Jinja, or by looping over each value.

That said, that error message is very perplexing. The result of that Jinja operation should be a string (the return value of pop()). The pop() operation should only be executed one time and at that time the list is decidedly not empty. It looks like we're doing two passes over the templated file 😦 :

{{ salt.cmd.run('touch /tmp/gotfoo && echo foo >> /tmp/gotfoo') }}

Since we're looping twice and it does modify the list it is indeed empty on the second pass.