fix(libmapstack): allow mapping by booleans and numbers

stasjok commented 2 years ago

PR progress checklist (to be filled in by reviewers)

[ ] Changes to documentation are appropriate (or tick if not required)
[ ] Changes to tests are appropriate (or tick if not required)
[ ] Reviews completed

What type of PR is this?

Primary type

[ ] [build] Changes related to the build system
[ ] [chore] Changes to the build process or auxiliary tools and libraries such as documentation generation
[ ] [ci] Changes to the continuous integration configuration
[ ] [feat] A new feature
[x] [fix] A bug fix
[ ] [perf] A code change that improves performance
[ ] [refactor] A code change that neither fixes a bug nor adds a feature
[ ] [revert] A change used to revert a previous commit
[ ] [style] Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)

Secondary type

[ ] [docs] Documentation changes
[ ] [test] Adding missing or correcting existing tests

Does this PR introduce a `BREAKING CHANGE`?

No.

Related issues and/or pull requests

Describe the changes you're proposing

If result of the lookup is a number, then map.jinja fails with TypeError: object of type 'int' has no len() here https://github.com/saltstack-formulas/template-formula/blob/080fdcd9f136f19d2a7dd15b17bf4de5c1c4a3c8/TEMPLATE/libmapstack.jinja#L184 If result of the lookup is a list of numbers, then map.jinja fails with Jinja variable 'int object' has no attribute 'rpartition' here https://github.com/saltstack-formulas/template-formula/blob/080fdcd9f136f19d2a7dd15b17bf4de5c1c4a3c8/TEMPLATE/libmapstack.jinja#L202 In this PR I tried to allow it to be numbers, booleans, list of numbers.

Usecases are:

Uyuni or SUSE Manager exposes its groups as pillar value group_ids which is a list of integers: https://www.uyuni-project.org/uyuni-docs/en/uyuni/specialized-guides/salt/salt-custom-states.html#_apply_a_custom_state_at_highstate
I can imagine someone might want to use mapping by Y:G@efi (returns boolean) or similar

Pillar / config required to test the proposed changes

parameters/map_jinja.yaml:

values:
  sources:
    # default values
    - "Y:G@osarch"
    - "Y:G@os_family"
    - "Y:G@os"
    - "Y:G@osfinger"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"

    - "Y:G@cpusockets"
    - "Y:I@group_ids"
    - "Y:G@efi"

    # default values
    - "Y:G@id"

and add some values to files

parameters/efi/False.yaml
parameters/efi/True.yaml
parameters/cpusockets/1.yaml
parameters/cpusockets/2.yaml
parameters/group_ids/14.yaml
parameters/group_ids/2.yaml
parameters/group_ids/26.yaml

state.apply btrfsmaintenance._mapdata should not fail, values should be present.

Debug log showing how the proposed changes work

Documentation checklist

[ ] Updated the README (e.g. Available states).
[ ] Updated pillar.example.

Testing checklist

[ ] Included in Kitchen (i.e. under state_top).
[ ] Covered by new/existing tests (e.g. InSpec, Serverspec, etc.).
[ ] Updated the relevant test pillar.

Additional context

baby-gnu commented 2 years ago

Hello.

I'm validating your changes against all the platforms with the following patch to trigger the errors:

test-pr243.patch

``` diff diff --git a/TEMPLATE/parameters/map_jinja.yaml b/TEMPLATE/parameters/map_jinja.yaml new file mode 100644 index 0000000..7abb412 --- /dev/null +++ b/TEMPLATE/parameters/map_jinja.yaml @@ -0,0 +1,19 @@ +# -*- coding: utf-8 -*- +# vim: ft=yaml +# +# For testing purpose of `map.jinja` +--- +values: + sources: + - Y:G@osarch + - Y:G@os_family + - Y:G@os + - Y:G@osfinger + - C@TEMPLATE:lookup + - C@TEMPLATE + + # Test https://github.com/saltstack-formulas/template-formula/pull/243 + - Y:I@TEMPLATE:group_ids + - Y:G@efi + + - Y:G@id diff --git a/test/salt/pillar/default.sls b/test/salt/pillar/default.sls index 995d8f7..130aa7a 100644 --- a/test/salt/pillar/default.sls +++ b/test/salt/pillar/default.sls @@ -17,6 +17,10 @@ TEMPLATE: name: systemd-journald config: /etc/template-formula.conf + group_ids: + - 12 + - 13 + tofs: # The files_switch key serves as a selector for alternative # directories under the formula files directory. See TOFS pattern diff --git a/test/integration/default/files/_mapdata/almalinux-8.yaml b/test/integration/default/files/_mapdata/almalinux-8.yaml index 964d67f..d1129c1 100644 --- a/test/integration/default/files/_mapdata/almalinux-8.yaml +++ b/test/integration/default/files/_mapdata/almalinux-8.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/amazonlinux-1.yaml b/test/integration/default/files/_mapdata/amazonlinux-1.yaml index 5e9a51d..2dde824 100644 --- a/test/integration/default/files/_mapdata/amazonlinux-1.yaml +++ b/test/integration/default/files/_mapdata/amazonlinux-1.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/amazonlinux-2.yaml b/test/integration/default/files/_mapdata/amazonlinux-2.yaml index 72cc079..3b447ad 100644 --- a/test/integration/default/files/_mapdata/amazonlinux-2.yaml +++ b/test/integration/default/files/_mapdata/amazonlinux-2.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/arch-base-latest.yaml b/test/integration/default/files/_mapdata/arch-base-latest.yaml index f229d77..fb5410d 100644 --- a/test/integration/default/files/_mapdata/arch-base-latest.yaml +++ b/test/integration/default/files/_mapdata/arch-base-latest.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/centos-6.yaml b/test/integration/default/files/_mapdata/centos-6.yaml index e483d0d..3c8a415 100644 --- a/test/integration/default/files/_mapdata/centos-6.yaml +++ b/test/integration/default/files/_mapdata/centos-6.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/centos-7.yaml b/test/integration/default/files/_mapdata/centos-7.yaml index cbb0b4f..1234e88 100644 --- a/test/integration/default/files/_mapdata/centos-7.yaml +++ b/test/integration/default/files/_mapdata/centos-7.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/centos-8.yaml b/test/integration/default/files/_mapdata/centos-8.yaml index b964150..1c02084 100644 --- a/test/integration/default/files/_mapdata/centos-8.yaml +++ b/test/integration/default/files/_mapdata/centos-8.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/debian-10.yaml b/test/integration/default/files/_mapdata/debian-10.yaml index cde0770..2b0ec05 100644 --- a/test/integration/default/files/_mapdata/debian-10.yaml +++ b/test/integration/default/files/_mapdata/debian-10.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/debian-11.yaml b/test/integration/default/files/_mapdata/debian-11.yaml index d30e2e9..b0336e8 100644 --- a/test/integration/default/files/_mapdata/debian-11.yaml +++ b/test/integration/default/files/_mapdata/debian-11.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/debian-9.yaml b/test/integration/default/files/_mapdata/debian-9.yaml index bf5aae9..40a82fb 100644 --- a/test/integration/default/files/_mapdata/debian-9.yaml +++ b/test/integration/default/files/_mapdata/debian-9.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/fedora-31.yaml b/test/integration/default/files/_mapdata/fedora-31.yaml index a36a2fa..d0de3a6 100644 --- a/test/integration/default/files/_mapdata/fedora-31.yaml +++ b/test/integration/default/files/_mapdata/fedora-31.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/fedora-32.yaml b/test/integration/default/files/_mapdata/fedora-32.yaml index 4aac6c1..3175398 100644 --- a/test/integration/default/files/_mapdata/fedora-32.yaml +++ b/test/integration/default/files/_mapdata/fedora-32.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/fedora-33.yaml b/test/integration/default/files/_mapdata/fedora-33.yaml index 9508dd3..84c96a4 100644 --- a/test/integration/default/files/_mapdata/fedora-33.yaml +++ b/test/integration/default/files/_mapdata/fedora-33.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/fedora-34.yaml b/test/integration/default/files/_mapdata/fedora-34.yaml index c5ef79b..0835513 100644 --- a/test/integration/default/files/_mapdata/fedora-34.yaml +++ b/test/integration/default/files/_mapdata/fedora-34.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/gentoo-2-sysd.yaml b/test/integration/default/files/_mapdata/gentoo-2-sysd.yaml index 05e6ed5..53cc9f0 100644 --- a/test/integration/default/files/_mapdata/gentoo-2-sysd.yaml +++ b/test/integration/default/files/_mapdata/gentoo-2-sysd.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/gentoo-2-sysv.yaml b/test/integration/default/files/_mapdata/gentoo-2-sysv.yaml index 84b4fb9..36d8f53 100644 --- a/test/integration/default/files/_mapdata/gentoo-2-sysv.yaml +++ b/test/integration/default/files/_mapdata/gentoo-2-sysv.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/opensuse-15.yaml b/test/integration/default/files/_mapdata/opensuse-15.yaml index 7a52c39..a6af9ef 100644 --- a/test/integration/default/files/_mapdata/opensuse-15.yaml +++ b/test/integration/default/files/_mapdata/opensuse-15.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/opensuse-tumbleweed.yaml b/test/integration/default/files/_mapdata/opensuse-tumbleweed.yaml index dffde67..08350cf 100644 --- a/test/integration/default/files/_mapdata/opensuse-tumbleweed.yaml +++ b/test/integration/default/files/_mapdata/opensuse-tumbleweed.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/oraclelinux-7.yaml b/test/integration/default/files/_mapdata/oraclelinux-7.yaml index 4eae50d..a5c8026 100644 --- a/test/integration/default/files/_mapdata/oraclelinux-7.yaml +++ b/test/integration/default/files/_mapdata/oraclelinux-7.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/oraclelinux-8.yaml b/test/integration/default/files/_mapdata/oraclelinux-8.yaml index 40f09bb..9368e02 100644 --- a/test/integration/default/files/_mapdata/oraclelinux-8.yaml +++ b/test/integration/default/files/_mapdata/oraclelinux-8.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/rockylinux-8.yaml b/test/integration/default/files/_mapdata/rockylinux-8.yaml index 1ca14ea..33a594c 100644 --- a/test/integration/default/files/_mapdata/rockylinux-8.yaml +++ b/test/integration/default/files/_mapdata/rockylinux-8.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/ubuntu-16.yaml b/test/integration/default/files/_mapdata/ubuntu-16.yaml index 83c9f8b..8009014 100644 --- a/test/integration/default/files/_mapdata/ubuntu-16.yaml +++ b/test/integration/default/files/_mapdata/ubuntu-16.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/ubuntu-18.yaml b/test/integration/default/files/_mapdata/ubuntu-18.yaml index 86a231b..7c2d642 100644 --- a/test/integration/default/files/_mapdata/ubuntu-18.yaml +++ b/test/integration/default/files/_mapdata/ubuntu-18.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: diff --git a/test/integration/default/files/_mapdata/ubuntu-20.yaml b/test/integration/default/files/_mapdata/ubuntu-20.yaml index cc56582..cf12037 100644 --- a/test/integration/default/files/_mapdata/ubuntu-20.yaml +++ b/test/integration/default/files/_mapdata/ubuntu-20.yaml @@ -7,6 +7,9 @@ values: added_in_pillar: pillar_value arch: amd64 config: /etc/template-formula.conf + group_ids: + - 12 + - 13 lookup: added_in_lookup: lookup_value master: template-master @@ -19,6 +22,8 @@ values: - Y:G@osfinger - C@TEMPLATE:lookup - C@TEMPLATE + - Y:I@TEMPLATE:group_ids + - Y:G@efi - Y:G@id master: template-master pkg: ```

There is no need to create the parameters/groups_ids/* and parameters/efi/* files as the errors occur before reading their content.

@myii, should we ask to include my patch in the PR to test or could we reserve such changes to a dedicated, to be created someday, test-mapstack-formula repository?

Including it will add some more stuff to be managed by users when converting the formula for their use.

myii commented 2 years ago

@myii, should we ask to include my patch in the PR to test or could we reserve such changes to a dedicated, to be created someday, test-mapstack-formula repository?

Including it will add some more stuff to be managed by users when converting the formula for their use.

@baby-gnu Does that mean we would also need to update the bin/convert-formula.sh script? If so, it's probably best if we add this in a separate PR once this is merged.

saltstack-formulas-travis commented 2 years ago

:tada: This PR is included in version 5.0.7 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

myii commented 2 years ago

Pushed out across relevant formulas using https://github.com/myii/ssf-formula/pull/418 (with author set as Stanislav Asunkin).

saltstack-formulas / template-formula