Open eedgar opened 3 years ago
baby-gnu
commented
3 years ago Interesting point.
I must admit that I always extend formulas by providing new sls under the same salt:// path.
With the following master configuration:
fileserver_backend:
- roots
- gitfsI can create /srv/salt/apache/www/foo.sls and salt will lookup that sls with the following URL salt://apache/www/foo.sls.
I'll make some try tomorrow and post my finding.
baby-gnu
commented
3 years ago It took a little bit longer but I finally made little tests with relative imports and it's not trivial:
map.jinja in .sls files is working (at least with my 3001 test bed)map.jinja is not working because the relative pathes are expanded relatively to the .sls fileSo, salt 'testmachine2' state.show_sls foo.test-relative-import works but salt 'testmachine2' state.show_sls foo.subcomponent.test-relative-import does not.
To make my test, I created the following files:
foo/test-relavite-import.sls
{%- from "./map.jinja" import mapdata with context %}
foo/test-relative-import:
test.nop:
- name: {{ mapdata }}foo/subcomponent/test-relative-import.sls
{%- from "../map.jinja" import mapdata with context %}
foo/subcomponent/test-relative-import:
test.nop:
- name: {{ mapdata }}foo/map.jinja
{%- from "./test-relative-sub-import.jinja" import submapdata %}
{%- set mapdata = submapdata %}foo/test-relative-sub-import.jinja
{%- set submapdata = {'foo': 'bar'} %}bar/test-relative-import.sls
{%- from "foo/map.jinja" import mapdata with context %}
bar/test-relative-import:
test.nop:
- name: {{ mapdata }}bar/subcomponent/test-relative-import.sls
{%- from "foo/map.jinja" import mapdata with context %}
bar/subcomponent/test-relative-import:
test.nop:
- name: {{ mapdata }}Here are the outputs:
salt 'testmachine2' state.show_sls foo.test-relative-import
testmachine2.example.net:
----------
foo/test:
----------
__env__:
base
__sls__:
foo.test-relative-import
test:
|_
----------
name:
----------
foo:
bar
- nop
|_
----------
order:
10000salt 'testmachine2*' state.show_sls foo.subcomponent.test-relative-import
testmachine2.example.net:
- Rendering SLS 'base:foo.subcomponent.test-relative-import' failed: Jinja error: ./test-relative-sub-import.jinja
/var/cache/salt/minion/files/base/foo/map.jinja(1):
---
{%- from "./test-relative-sub-import.jinja" import submapdata %} <======================
{%- set mapdata = submapdata %}
---
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1073, in make_module
return TemplateModule(self, self.new_context(vars, shared, locals))
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1152, in __init__
body_stream = list(template.root_render_func(context))
File "/var/cache/salt/minion/files/base/foo/map.jinja", line 1, in top-level template code
{%- from "./test-relative-sub-import.jinja" import submapdata %}
File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 204, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./test-relative-sub-import.jinjasalt 'testmachine2*' state.show_sls bar.test-relative-import
testmachine2.example.net:
----------
bar/test-relative-import:
----------
__env__:
base
__sls__:
bar.test-relative-import
test:
|_
----------
name:
----------
foo:
bar
- nop
|_
----------
order:
10000salt 'testmachine2*' state.show_sls bar.subcomponent.test-relative-import
testmachine2.example.net:
----------
bar/subcomponent/test-relative-import:
----------
__env__:
base
__sls__:
bar.subcomponent.test-relative-import
test:
|_
----------
name:
----------
foo:
bar
- nop
|_
----------
order:
10000This could be solved if we find a consistent way to find the path of the current included/imported file instead of current sls file:
map.jinja (in my test, tlpdir is set to bar when importing it in bar/test-relative-import.sls using with context but stay foo without the context (now I wonder why we require the context)/map.jinja from that pathIs there a way to achive this consistently across salt versions? I'm a bit lost about what is possible or not [1], [2] :-/
@myii: do you have any hint?
Regards.
baby-gnu
commented
3 years ago I made a little test:
openssh-formula and openntpd-formula which have the _mapdata verificationwith context when importing from map.jinjabin/kitchen verify for all supported platformsThe Inspec _mapdata_spec.rb see no difference in the mapdata contents…
The load with context is only useful when the imported .jinja require to access variables set in the .sls.
without the with context, map.jinja can't use the something variable
{%- set something = "this is something" %}
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/map.jinja" import mapdata %}with the with context, map.jinja can use the something variable
{%- set something = "this is something" %}
{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- from tplroot ~ "/map.jinja" import mapdata with context %}I think we could say that map.jinja must be loaded without with context and then, map.jinja lookup files relatively to it's tpldir.
baby-gnu
commented
3 years ago I made more tests and I think this solution may be interesting:
{#- Make sure `map.jinja` is imported without the context #}
{#- Import `with context` override the `tplfile` and `tpldir` variables #}
{%- if not tplfile.endswith("/map.jinja") %}
{{ raise("Import error: map.jinja must be imported without context. tplfile='" ~ tplfile ~ "'") }}
{%- endif %}
{#- `tplroot` is `tpldir`: the directory where is `map.jinja` #}
{%- set tplroot = tpldir %}It's working with v5 map.jinja for openvpn-formula for all kitchen supported platforms but not with relative imports like ../map.jinja because tpldir is ../ in this case.
This means that for the above example, I must remove all the with context and replace ../map.jinja by foo/map.jinja in foo.subcomponent.test-relative-import.
I could even add another test and raise in map.jinja:
{%- if tplfile.startswith("../") %}
{{ raise("Import error: map.jinja must be imported with absolute path. tplfile='" ~ tplfile ~ "'") }}
{%- endif %}This will permit to load map.jinja of whatever formula from any other without troubles.
kritzi-at
commented
3 years ago tpldir does not work with older versions.
for example it does not work Salt Version 3000.6
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 404, in render_jinja_tmpl
buf=tmplstr)
SaltRenderError: Jinja variable 'tpldir' is undefined
/var/cache/salt/minion/files/base/apache/map.jinja(4):
---
# -*- coding: utf-8 -*-
# vim: ft=jinja
{%- set tplroot = tpldir.split('/')[0] %} <======================
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
[...]
---This is a big issue for non zero-day admins
baby-gnu
commented
3 years ago tpldir does not work with older versions.
for example it does not work Salt Version 3000.6
Hello @kritzi-at, that's strange because I tested on all supported platforms with a test bed test-tplvars-formula:
test-tplvars-formula/test-tplvars/test.sls
{%- from "test-tplvars/test-tplvars.jinja" import jinjafile, jinjadir %}
test-tplvars-variable:
file.managed:
- name: /tmp/tplvars.txt
- source: salt://test-tplvars/tplvars.txt.jinja
- template: jinja
- context:
jinjafile: {{ jinjafile }}
jinjadir: {{ jinjadir }}test-tplvars-formula/test-tplvars/tplvars.txt.jinja
tplfile: {{ jinjafile }}
tpldir: {{ jinjadir }}test-tplvars-formula/test-tplvars/test-tplvars.jinja
{%- if not tplfile.endswith('/test-tplvars.jinja') %}
{{ raise("Import error: map.jinja must be imported without context") }}
{%- endif %}
{%- set jinjafile = tplfile %}
{%- set jinjadir = tpldir %}test-tplvars-formula/test/integration/default/controls/tplvars.rb
# frozen_string_literal: true
control "test-tplvars/test" do
title 'check value of tplvars for imported jinja file'
describe file('/tmp/tplvars.txt') do
it { should be_file }
its('content') { should include "tplfile: test-tplvars/test-tplvars.jinja" }
its('content') { should include "tpldir: test-tplvars" }
end
endI use kitchen files from template-formula with minor changes for this pseudo formula:
After running ./bin/kitchen verify, I have only errors for centos-6 and amazonlinux-1 which could no be created.
kritzi-at
commented
3 years ago I have now upgraded to salt-minion 3002.2 Error is still the same:
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 5, in top-level template code
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1073, in make_module
return TemplateModule(self, self.new_context(vars, shared, locals))
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1152, in __init__
body_stream = list(template.root_render_func(context))
File "/var/cache/salt/minion/files/base/apache/map.jinja", line 4, in top-level template code
{%- set tplroot = tpldir.split('/')[0] %}
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 430, in getattr
return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'tpldir' is undefined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 505, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'tpldir' is undefined
/var/cache/salt/minion/files/base/apache/map.jinja(4):
---
# -*- coding: utf-8 -*-
# vim: ft=jinja
{%- set tplroot = tpldir.split('/')[0] %} <======================
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
[...]
---When i use slsutil.renderer the tpldir seems to be defined, but not as expected:
# salt-call slsutil.renderer salt://apache/map.jinja
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 5, in top-level template code
File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./defaults.yaml
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 543, in render_jinja_tmpl
"Jinja error: {}{}".format(exc, out), line, tmplstr, trace=tracestr
salt.exceptions.SaltRenderError: Jinja error: ./defaults.yaml
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 5, in top-level template code
File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./defaults.yaml
; line 5
---
# -*- coding: utf-8 -*-
# vim: ft=jinja
{%- set tplroot = tpldir.split('/')[0] %}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %} <======================
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
{%- import_yaml tplroot ~ "/oscodenamemap.yaml" as oscodename %}
[...]
---Hello @kritzi-at, can you try a salt 'minion' state.apply test-tplvars.test with my tests files?
Note that apache-formula is working on different environment as you can see in the last pipeline run, even on a 2017.7.
kritzi-at
commented
3 years ago Hello @baby-gnu
Test script works fine on Ubuntu with minion 3002.2
# salt-call state.apply test-tplvars.test
local:
----------
ID: test-tplvars-variable
Function: file.managed
Name: /tmp/tplvars.txt
Result: True
Comment: File /tmp/tplvars.txt updated
Started: 16:05:23.627108
Duration: 75.168 ms
Changes:
----------
diff:
New file
mode:
0644
Summary for local
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 75.168 ms
# cat /tmp/tplvars.txt
tplfile: test-tplvars/test-tplvars.jinja
tpldir: test-tplvarsand on CentOS with minion 3000.6
# salt-call state.apply test-tplvars.test
local:
Name: /tmp/tplvars.txt - Function: file.managed - Result: Changed Started: - 16:05:00.786759 Duration: 74.754 ms
Summary for local
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 74.754 ms
# cat /tmp/tplvars.txt
tplfile: test-tplvars/test-tplvars.jinja
tpldir: test-tplvarsThanks @kritzi-at, so now I'm wondering why applying apache-formula is not working for you.
Can you explain a little your setup and how you run the formula?
kritzi-at
commented
3 years ago Hello @baby-gnu, i found the issue. The problem was related to the template_file i set in the pillar. The template file was custom made and included apache/map.jinja without using it.
Removing the line fixed everything.
baby-gnu
commented
3 years ago More tests to see the differences between with context and without context : https://gist.github.com/baby-gnu/c8c6d1b423ac64b88011ffba2b4e884d
So, only tplfile is different and with context the tplroot variable is propagated from the .sls to the imported .jinja file.
Your setup
Formula commit hash / release tag
starting from e2e1be18e093dca2d603b68075b388d0f8b61d3d
Versions reports (master & minion)
Salt Version: Salt: 3002.1
Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.6.2 mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: Not Installed Python: 3.8.5 (default, Jul 28 2020, 12:59:40) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 18.1.1 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2
System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-52-generic system: Linux version: Ubuntu 20.04 focal
Pillar / config used
trying to extend the formula by creating a new www/init.sls and having the contents look like this {% from "apache/map.jinja" import apache with context %} include:
gives this error local: Data failed to compile:
the workaround
{% set curr_tpldir = tpldir %} {% set tpldir = 'apache' %} {% from "apache/map.jinja" import apache with context %} {% set tpldir = curr_tpldir %} include:
Bug details
Describe the bug
replace the tpldir/tplroot logic with hardcoded apache paths
Steps to reproduce the bug
Expected behaviour
Attempts to fix the bug
Additional context