feat(map): update to v5 map.jinja

baby-gnu commented 3 years ago

PR progress checklist (to be filled in by reviewers)

[ ] Changes to documentation are appropriate (or tick if not required)
[ ] Changes to tests are appropriate (or tick if not required)
[ ] Reviews completed

What type of PR is this?

Primary type

[ ] [build] Changes related to the build system
[ ] [chore] Changes to the build process or auxiliary tools and libraries such as documentation generation
[ ] [ci] Changes to the continuous integration configuration
[x] [feat] A new feature
[ ] [fix] A bug fix
[ ] [perf] A code change that improves performance
[ ] [refactor] A code change that neither fixes a bug nor adds a feature
[ ] [revert] A change used to revert a previous commit
[ ] [style] Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)

Secondary type

[x] [docs] Documentation changes
[x] [test] Adding missing or correcting existing tests

Does this PR introduce a `BREAKING CHANGE`?

Yes.

BREAKING CHANGE: map.jinja now exports a generic mapdata variable

BREAKING CHANGE: The per grain parameter values are now under TEMPLATE/parameters/

Related issues and/or pull requests

186
saltstack-formulas/openssh-formula#191
saltstack-formulas/openvpn-formula#134

Describe the changes you're proposing

feat(map): update to v5 map.jinja

The v5 map.jinja is a generic and configurable system to load configuration values, exposed as the mapdata variable, from different places:

YAML files and templates from the fileserver for non-secret data
pillars or SDB are preferred for secret data
grains or config.get

The map.jinja optional sources are configured with compound targeting like syntax [<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY> with the following default ordered sources:

Y:G@osarch: YAML file and Jinja template named after osarch grain
Y:G@os_family: YAML file and Jinja template named after os_family grain
Y:G@os YAML file and Jinja template named after os grain
Y:G@osfinger YAML file and Jinja template named after osfinger grain
C@{{ tplroot ~ ':lookup' }}: dict retrieved with salt["config.get"]
C@{{ tplroot }}: dict retrieved with salt["config.get"]
Y:G@id: YAML file and Jinja template named after osarch grain

This is done by two new libraries:

libmatchers.jinja provides the parse_matchers macro to parse strings looking like compound matchers, for example Y:G@osarch
libmapstack.jinja provides the mapstack macro to load configuration from different sources described by matchers

Post-processing of mapdata variable can be done in a parameters/post-map.jinja.

The v5 map.jinja is documented in docs/map.jinja.rst.

Pillar / config required to test the proposed changes

Debug log showing how the proposed changes work

Log of the processing of map.jinja

``` [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/map.jinja' to resolve 'salt://TEMPLATE/map.jinja' [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/libmapstack.jinja' to resolve 'salt://TEMPLATE/libmapstack.jinja' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/libmapstack.jinja' to resolve 'salt://TEMPLATE/libmapstack.jinja' [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/libmatchers.jinja' to resolve 'salt://TEMPLATE/libmatchers.jinja' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/libmatchers.jinja' to resolve 'salt://TEMPLATE/libmatchers.jinja' [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/libsaltcli.jinja' to resolve 'salt://TEMPLATE/libsaltcli.jinja' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/libsaltcli.jinja' to resolve 'salt://TEMPLATE/libsaltcli.jinja' [DEBUG ] [libsaltcli] the salt command type has been identified to be: local [DEBUG ] map.jinja configuration: process matcher: 'map_jinja.yaml' [DEBUG ] map.jinja configuration: use built-in defaults for matcher: option: null query: map_jinja.yaml query_delimiter: ':' query_method: config.get type: F [DEBUG ] map.jinja configuration: lookup 'map_jinja.yaml' with 'config.get' [DEBUG ] map.jinja configuration: parsed matchers: - option: null query: map_jinja.yaml query_delimiter: ':' query_method: config.get type: F value: [] [DEBUG ] map.jinja configuration: built-in configuration: values: sources: - Y:G@osarch - Y:G@os_family - Y:G@os - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE - Y:G@id [DEBUG ] map.jinja configuration: load configuration values from parameters/map_jinja.yaml [DEBUG ] Could not find file 'salt://parameters/map_jinja.yaml' in saltenv 'base' [DEBUG ] map.jinja configuration: load configuration values from parameters/map_jinja.yaml.jinja [DEBUG ] Could not find file 'salt://parameters/map_jinja.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja configuration: load configuration values from TEMPLATE/parameters/map_jinja.yaml [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/map_jinja.yaml' in saltenv 'base' [DEBUG ] map.jinja configuration: load configuration values from TEMPLATE/parameters/map_jinja.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/map_jinja.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja configuration: final configuration values: values: sources: - Y:G@osarch - Y:G@os_family - Y:G@os - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE - Y:G@id [DEBUG ] map.jinja: load parameters from sources: - Y:G@osarch - Y:G@os_family - Y:G@os - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE - Y:G@id [DEBUG ] map.jinja: process matcher: 'defaults.yaml' [DEBUG ] map.jinja: use built-in defaults for matcher: option: null query: defaults.yaml query_delimiter: ':' query_method: config.get type: F [DEBUG ] map.jinja: lookup 'defaults.yaml' with 'config.get' [DEBUG ] map.jinja: process matcher: 'Y:G@osarch' [DEBUG ] map.jinja: parse matcher: 'Y:G@osarch' [DEBUG ] map.jinja: parse as 2 metadata matcher: option: G query: osarch query_delimiter: ':' type: Y [DEBUG ] map.jinja: lookup 'osarch' with 'grains.get' [DEBUG ] map.jinja: process matcher: 'Y:G@os_family' [DEBUG ] map.jinja: parse matcher: 'Y:G@os_family' [DEBUG ] map.jinja: parse as 2 metadata matcher: option: G query: os_family query_delimiter: ':' type: Y [DEBUG ] map.jinja: lookup 'os_family' with 'grains.get' [DEBUG ] map.jinja: process matcher: 'Y:G@os' [DEBUG ] map.jinja: parse matcher: 'Y:G@os' [DEBUG ] map.jinja: parse as 2 metadata matcher: option: G query: os query_delimiter: ':' type: Y [DEBUG ] map.jinja: lookup 'os' with 'grains.get' [DEBUG ] map.jinja: process matcher: 'Y:G@osfinger' [DEBUG ] map.jinja: parse matcher: 'Y:G@osfinger' [DEBUG ] map.jinja: parse as 2 metadata matcher: option: G query: osfinger query_delimiter: ':' type: Y [DEBUG ] map.jinja: lookup 'osfinger' with 'grains.get' [DEBUG ] map.jinja: process matcher: 'C@TEMPLATE:lookup' [DEBUG ] map.jinja: parse matcher: 'C@TEMPLATE:lookup' [DEBUG ] map.jinja: parse as 1 metadata matcher: option: C query: TEMPLATE:lookup query_delimiter: ':' type: C [DEBUG ] map.jinja: lookup 'TEMPLATE:lookup' with 'config.get' [DEBUG ] map.jinja: process matcher: 'C@TEMPLATE' [DEBUG ] map.jinja: parse matcher: 'C@TEMPLATE' [DEBUG ] map.jinja: parse as 1 metadata matcher: option: C query: TEMPLATE query_delimiter: ':' type: C [DEBUG ] map.jinja: lookup 'TEMPLATE' with 'config.get' [DEBUG ] map.jinja: process matcher: 'Y:G@id' [DEBUG ] map.jinja: parse matcher: 'Y:G@id' [DEBUG ] map.jinja: parse as 2 metadata matcher: option: G query: id query_delimiter: ':' type: Y [DEBUG ] map.jinja: lookup 'id' with 'grains.get' [DEBUG ] map.jinja: parsed matchers: - option: null query: defaults.yaml query_delimiter: ':' query_method: config.get type: F value: [] - option: G query: osarch query_delimiter: ':' query_method: grains.get type: Y value: amd64 - option: G query: os_family query_delimiter: ':' query_method: grains.get type: Y value: Debian - option: G query: os query_delimiter: ':' query_method: grains.get type: Y value: Debian - option: G query: osfinger query_delimiter: ':' query_method: grains.get type: Y value: Debian-10 - option: C query: TEMPLATE:lookup query_delimiter: ':' query_method: config.get type: C value: &id001 master: template-master winner: lookup added_in_lookup: lookup_value - option: C query: TEMPLATE query_delimiter: ':' query_method: config.get type: C value: lookup: *id001 pkg: name: bash service: name: systemd-journald config: /etc/template-formula.conf tofs: files_switch: - any/path/can/be/used/here - id - roles - osfinger - os - os_family source_files: TEMPLATE-config-file-file-managed: - example.tmpl.jinja TEMPLATE-subcomponent-config-file-file-managed: - subcomponent-example.tmpl.jinja winner: pillar added_in_pillar: pillar_value - option: G query: id query_delimiter: ':' query_method: grains.get type: Y value: 65840c220770 [DEBUG ] map.jinja: built-in configuration: values: {} [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/defaults.yaml [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/parameters/defaults.yaml' to resolve 'salt://TEMPLATE/parameters/defaults.yaml' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/parameters/defaults.yaml' to resolve 'salt://TEMPLATE/parameters/defaults.yaml' [DEBUG ] map.jinja: loaded configuration values from TEMPLATE/parameters/defaults.yaml: values: added_in_defaults: defaults_value config: /etc/TEMPLATE pkg: name: TEMPLATE rootgroup: root service: name: TEMPLATE subcomponent: config: /etc/TEMPLATE-subcomponent-formula.conf winner: defaults [DEBUG ] map.jinja: merged configuration values from TEMPLATE/parameters/defaults.yaml, merge: strategy='smart', merge_lists='False': values: added_in_defaults: defaults_value config: /etc/TEMPLATE pkg: name: TEMPLATE rootgroup: root service: name: TEMPLATE subcomponent: config: /etc/TEMPLATE-subcomponent-formula.conf winner: defaults [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/defaults.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/defaults.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/osarch/amd64.yaml [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/parameters/osarch/amd64.yaml' to resolve 'salt://TEMPLATE/parameters/osarch/amd64.yaml' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/parameters/osarch/amd64.yaml' to resolve 'salt://TEMPLATE/parameters/osarch/amd64.yaml' [DEBUG ] map.jinja: loaded configuration values from TEMPLATE/parameters/osarch/amd64.yaml: values: arch: amd64 [DEBUG ] map.jinja: merged configuration values from TEMPLATE/parameters/osarch/amd64.yaml, merge: strategy='smart', merge_lists='False': values: added_in_defaults: defaults_value arch: amd64 config: /etc/TEMPLATE pkg: name: TEMPLATE rootgroup: root service: name: TEMPLATE subcomponent: config: /etc/TEMPLATE-subcomponent-formula.conf winner: defaults [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/osarch/amd64.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/osarch/amd64.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/os_family/Debian.yaml [DEBUG ] In saltenv 'base', looking at rel_path 'TEMPLATE/parameters/os_family/Debian.yaml' to resolve 'salt://TEMPLATE/parameters/os_family/Debian.yaml' [DEBUG ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/TEMPLATE/parameters/os_family/Debian.yaml' to resolve 'salt://TEMPLATE/parameters/os_family/Debian.yaml' [DEBUG ] map.jinja: loaded configuration values from TEMPLATE/parameters/os_family/Debian.yaml: values: config: /etc/TEMPLATE.d/custom.conf pkg: name: TEMPLATE-debian [DEBUG ] map.jinja: merged configuration values from TEMPLATE/parameters/os_family/Debian.yaml, merge: strategy='smart', merge_lists='False': values: added_in_defaults: defaults_value arch: amd64 config: /etc/TEMPLATE.d/custom.conf pkg: name: TEMPLATE-debian rootgroup: root service: name: TEMPLATE subcomponent: config: /etc/TEMPLATE-subcomponent-formula.conf winner: defaults [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/os_family/Debian.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/os_family/Debian.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/os/Debian.yaml [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/os/Debian.yaml' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/os/Debian.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/os/Debian.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/osfinger/Debian-10.yaml [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/osfinger/Debian-10.yaml' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/osfinger/Debian-10.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/osfinger/Debian-10.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: merge 'TEMPLATE:lookup' retrieved with 'config.get', merge: strategy='smart', lists='False': added_in_lookup: lookup_value master: template-master winner: lookup [DEBUG ] map.jinja: merge 'TEMPLATE' retrieved with 'config.get', merge: strategy='smart', lists='False': added_in_pillar: pillar_value config: /etc/template-formula.conf lookup: added_in_lookup: lookup_value master: template-master winner: lookup pkg: name: bash service: name: systemd-journald tofs: files_switch: - any/path/can/be/used/here - id - roles - osfinger - os - os_family source_files: TEMPLATE-config-file-file-managed: - example.tmpl.jinja TEMPLATE-subcomponent-config-file-file-managed: - subcomponent-example.tmpl.jinja winner: pillar [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/id/65840c220770.yaml [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/id/65840c220770.yaml' in saltenv 'base' [DEBUG ] map.jinja: load configuration values from TEMPLATE/parameters/id/65840c220770.yaml.jinja [DEBUG ] Could not find file 'salt://TEMPLATE/parameters/id/65840c220770.yaml.jinja' in saltenv 'base' [DEBUG ] map.jinja: final configuration values: values: added_in_defaults: defaults_value added_in_lookup: lookup_value added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf lookup: added_in_lookup: lookup_value master: template-master winner: lookup master: template-master pkg: name: bash rootgroup: root service: name: systemd-journald subcomponent: config: /etc/TEMPLATE-subcomponent-formula.conf tofs: files_switch: - any/path/can/be/used/here - id - roles - osfinger - os - os_family source_files: TEMPLATE-config-file-file-managed: - example.tmpl.jinja TEMPLATE-subcomponent-config-file-file-managed: - subcomponent-example.tmpl.jinja winner: pillar [DEBUG ] map.jinja: save parameters in variable 'mapdata' [DEBUG ] map.jinja: post-processing of 'mapdata' [DEBUG ] Could not find file 'salt://TEMPLATE/post-map.jinja' in saltenv 'base' ```

Documentation checklist

[x] Updated the README (e.g. Available states).
[ ] Updated pillar.example.

Testing checklist

[x] Included in Kitchen (i.e. under state_top).
[x] Covered by new/existing tests (e.g. InSpec, Serverspec, etc.).
[ ] Updated the relevant test pillar.

Additional context

myii commented 3 years ago

@baby-gnu Some initial feedback, just diff-ing across the three formulas. This PR and openvpn-formula are using the same library files and map.jinja. The openssh-formula is now out of sync, though:

--- .../openssh-formula/openssh/libmapstack.jinja
+++ .../template-formula/TEMPLATE/libmapstack.jinja
@@ -95,8 +95,10 @@
 {%-   set stack = defaults | default({"values": {} }, boolean=True) %}

 {#-   Build configuration file names based on matchers #}
+{%-   set config_get_strategy = salt["config.get"](tplroot ~ ":strategy", None) %}
 {%-   set matchers = parse_matchers(
         matchers,
+        config_get_strategy=config_get_strategy,
         log_prefix=log_prefix
       )
       | load_yaml %}
@@ -194,6 +196,19 @@
 {%-           set yaml_names = [yaml_names] %}
 {%-         endif %}

+{#-         Try to load a `.yaml.jinja` file for each `.yaml` file #}
+{%-         set all_yaml_names = [] %}
+{%-         for name in yaml_names %}
+{%-           set extension = name.rpartition(".")[2] %}
+{%-           if extension not in ["yaml", "jinja"] %}
+{%-             do all_yaml_names.extend([name ~ ".yaml", name ~ ".yaml.jinja"]) %}
+{%-           elif extension == "yaml" %}
+{%-             do all_yaml_names.extend([name, name ~ ".jinja"]) %}
+{%-           else %}
+{%-             do all_yaml_names.append(name) %}
+{%-           endif %}
+{%-         endfor %}
+
 {#-         `yaml_dirname` can be an empty string with literal path like `myconf.yaml` #}
 {%-         set yaml_dir = [
               param_dir,
@@ -202,15 +217,10 @@
             | select
             | join("/") %}

-{%-         for yaml_name in yaml_names %}
-{#-           Make sure to have a `.yaml` extension #}
-{#-           Use `.rpartition` to strip last `.yaml` in `dir.yaml/file.yaml` #}
+{%-         for yaml_name in all_yaml_names %}
 {%-           set yaml_filename = [
                 yaml_dir.rstrip("/"),
-                yaml_name.rpartition(".yaml")
-                | reject("equalto", ".yaml")
-                | join
-                ~ ".yaml"
+                yaml_name
               ]
               | select
               | join("/") %}
@@ -228,7 +238,7 @@
 {%-             do salt["log.debug"](
                   log_prefix
                   ~ "loaded configuration values from "
-                  ~ yaml_name
+                  ~ yaml_filename
                   ~ ":\n"
                   ~ yaml_values
                   | yaml(False)
@@ -271,7 +281,7 @@
 {%-             do salt["log.debug"](
                   log_prefix
                   ~ "merged configuration values from "
-                  ~ yaml_name
+                  ~ yaml_filename
                   ~ ", merge: strategy='"
                   ~ strategy
                   ~ "', merge_lists='"

--- .../openssh-formula/openssh/map.jinja
+++ .../template-formula/TEMPLATE/map.jinja
@@ -38,6 +38,7 @@

 {#- Load formula parameters values #}
 {%- set _formula_matchers = ["defaults.yaml"] + map_sources %}
+
 {%- set _formula_settings = mapstack(
       matchers=_formula_matchers,
       dirs=[formula_param_dir],
@@ -59,3 +60,7 @@

 {%- do salt["log.debug"]("map.jinja: save parameters in variable 'mapdata'") %}
 {%- set mapdata = _formula_settings["values"] %}
+
+{#- Per formula post-processing of `mapdata` if it exists #}
+{%- do salt["log.debug"]("map.jinja: post-processing of 'mapdata'") %}
+{%- include tplroot ~ "/post-map.jinja" ignore missing %}

baby-gnu commented 3 years ago

@myii yes, the openssh-formula should be updated, the reference is now openvpn-formula.

I push a fixed commit because I forgot to delete the old defaults.yaml and os*map.yaml.

baby-gnu commented 3 years ago

Lovely! Ready for merge in my opinion but let's give @saltstack-formulas/wg a bit of time to review.

Thanks @myii. This time will give me some time for the next move ;-)

saltstack-formulas-travis commented 3 years ago

:tada: This PR is included in version 5.0.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

saltstack-formulas / template-formula