[BUG] Linux `pass` fails to render data in highstate `NameError: name '__opts__' is not defined`

[jbartak]

Description

$ sudo salt-call state.highstate --state-output=changes 
[CRITICAL] Rendering SLS 'secrets.all' failed, render error:
name '__opts__' is not defined
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.py", line 938, in render_pstate
    state = compile_template(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/renderers/pass.py", line 38, in render
    return _pass.render(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 192, in render
    return _decrypt_object(pass_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 181, in _decrypt_object
    obj[pass_key] = _decrypt_object(pass_path)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 181, in _decrypt_object
    obj[pass_key] = _decrypt_object(pass_path)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 178, in _decrypt_object
    return _fetch_secret(obj)
  File "/var/cache/salt/minion/extmods/renderers/pass.py", line 16, in _patched_fetch_secret
    pass_data = _fetch_secret(pass_path)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 112, in _fetch_secret
    pass_prefix = __opts__["pass_variable_prefix"]
NameError: name '__opts__' is not defined
[CRITICAL] Pillar render error: Rendering SLS 'secrets.all' failed. Please see master log for details.
local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering SLS 'secrets.all' failed. Please see master log for details.

• No error in syslog, nor in /var/log/salt/master

Setup N/A

Please be as specific as possible and give set-up details.

• [ ] on-prem machine
• [x] VM (Virtualbox, Vagrant, Centos 7)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [ ] classic packaging
• [x] onedir packaging
• [ ] used bootstrap to install

Steps to Reproduce the behavior Install Salt 3006 Onedir, define some structure in pass a try to call highstate.

Expected behavior Have this working

Screenshots N/A

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml $ sudo salt --versions-report Salt Version: Salt: 3006.3 Python Version: Python: 3.10.13 (main, Sep 6 2023, 02:11:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.13.10 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: centos 7.9.2009 Core locale: utf-8 machine: x86_64 release: 3.10.0-1160.95.1.el7.x86_64 system: Linux version: CentOS Linux 7.9.2009 Core ```

Additional context N/A

[jbartak]

Find out that pass.py renderer from Salt 3005 works just fine on Salt 3006.

Looks like the problem started somewhere in https://github.com/saltstack/salt/commit/bcf5dd82ed4d96d617175b0af00c6cf25625f819

[Ch3LL]

@darix any chance you can take a look at this?

[dmach]

I wrote parts of the patch and also unittests for the pass rendeder. To be honest, I'm clueless, because other renderers (such as yaml) use __opts__ without any issues.

[darix]

I have no clue either why saltstack has not popuplated __opts__ in this case. we just use it to access the settings.

[jbartak]

Any chance this will be fixed? We have blocked migration to 3006 due to this and EOL of 3005 is soon :(.

[lkubb]

@jbartak The traceback shows that you were using a custom pass renderer (at /var/cache/salt/minion/extmods/renderers/pass.py), which then delegates to the internal one. Do you import and call it directly from your custom one? In that case, it's explainable why __opts__ would not be populated (this only works via the loader).

[mbebjak]

@lkubb yes we included internal pass renderer from our custom one. Thanks for pointing this out.

I've removed our custom renderer altogether because I have another issue with internal pass renderer.

[CRITICAL] Rendering SLS 'secrets.all' failed, render error:
a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.py", line 938, in render_pstate
    state = compile_template(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 192, in render
    return _decrypt_object(pass_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 181, in _decrypt_object
    obj[pass_key] = _decrypt_object(pass_path)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 181, in _decrypt_object
    obj[pass_key] = _decrypt_object(pass_path)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 178, in _decrypt_object
    return _fetch_secret(obj)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/pass.py", line 170, in _fetch_secret
    return pass_data.rstrip("\r\n")
TypeError: a bytes-like object is required, not 'str'

It seems that it is expected, that pass_data is a string, but it is bytes object.

I was able to "fix" this by modifying line 170 in in internal pass renderer to:

return pass_data.rstrip(b"\r\n")

But I'm not sure how this is intended to work. Should _fetch_secret return bytes object or string?

btw: Only reason why we have our custom pass render is to convert bytes object returned by internal pass rendered to string.

[lkubb]

proc.communicate returns bytes by default, so that's expected (this should have been broken for a long time). Your link goes to an old version of the renderer though, this will be fixed in 3007.0: https://github.com/saltstack/salt/pull/64300 (It seems the backport failed on that, would have assumed it's in 3006 also)

[mbebjak]

@lkubb Thanks. That fix is exactly what we need.

Found the failed backport: https://github.com/saltstack/salt/pull/64300#issuecomment-1577461324

I was not able to get backport tool running so I created backport PR manually: https://github.com/saltstack/salt/pull/66033