[BUG] Grain loading precedence documentation is inconsistent

[awestendorf]

Description

We're using Salt 3002.2 and trying to use a custom grain to declare meta_role for a machine, based on a static role grain, but are unable to do so because of the order in which grains are processed. At the time of evaluation, /etc/salt/grains has not yet been loaded and so it is not passed in the grains argument to the function.

In reading the documentation for grains, this behavior is called out here:

For custom grains, if the function takes an argument grains, then the previously rendered grains will be passed in. Because the rest of the grains could be rendered in any order, the only grains that can be relied upon to be passed in are core grains.

However, just above that, the documentation says:

The order of evaluation is as follows: Core grains. Custom grains in /etc/salt/grains. Custom grains in /etc/salt/minion. Custom grain modules in _grains directory, synced to minions.

Similarly, grains from /etc/salt/minion override both core grains and custom grain modules, and grains in _grains will override any grains of the same name.

If I modify the example below to return role, I have confirmed that it does not override the value of role in /etc/salt/grains, because the file is loaded after grain modules in the _grains directory have been evaluated. I have tried putting a static file in that directory but it is not synced, seemingly confirming that only custom grain modules are expected in the _grains directory, and so the custom modules should have access to, and be able to override, grains defined in /etc/salt/grains, per the documentation.

Setup

Grain role on a machine, a list stored in /etc/salt/grains. In this example we could say the value could be one or more of [public_service, backend_service, postgres, redis].

# /etc/salt/grains
role:
- public_service

The meta_role grain code would look something like this:

# _grains/meta_role.py

meta_role_map = dict(
  application=['public_service', 'backend_service'],
  database=['postgres', 'redis']
)

def main(grains=None):
  grains = grains or {}
  meta_roles = set()
  roles = grains.get('role', [])

  for role in roles:
    for meta_role, mapped_roles in meta_role_map.items():
      if role in mapped_roles:
        meta_roles.add(meta_role)

  return dict(
    meta_role=list(meta_roles)
  )

Steps to Reproduce the behavior

After declaring the above static and dynamic grains, and syncing them to a minion, run the following commands:

sudo salt-call grains.get role
sudo salt-call grains.get meta_role

Expected behavior

I expect that grains.get role returns the list [public_service], and grains.get meta_role returns the list [application]. However, meta_role returns an empty list because /etc/salt/grains has not been loaded at the time of the _grains/meta_role.py evaluation.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3002.2 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.7 gitpython: 3.1.8 Jinja2: 2.10.3 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.0 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.4.7 pygit2: Not Installed Python: 3.5.2 (default, Nov 23 2017, 16:37:01) python-gnupg: 0.3.8 PyYAML: 5.3.1 PyZMQ: 17.1.2 smmap: 4.0.0 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: ubuntu 16.04 Xenial Xerus locale: UTF-8 machine: x86_64 release: 4.4.0-87-generic system: Linux version: Ubuntu 16.04 Xenial Xerus ```

Additional context

It appears that this bug was described previously in 2018, and recently closed due to inactivity https://github.com/saltstack/salt/issues/50491

[sagetherage]

@awestendorf Salt v3002.2 is vulnerable, we recommend upgrading at your earliest to at least v3002.6 and is this a Bug or is this as the title describes "inconsistent" docs or both, simply clarify, thank you!

[awestendorf]

I consider it a bug because the docs describe the expected grain load order, but the code does not enforce that order. The docs are inconsistent on this because they state that one cannot be certain of what's loaded into the grains argument at the time of execution, which is true for the order of the programmatic grains, but all previous grain load steps should be complete when programmatic grains are evaluated.