[BUG] salt-ssh jinja import from doesn't work due to error jinja2.exceptions.TemplateNotFound

[dosercz]

Description Jinja import from doesn't work with salt-ssh due to error jinja2.exceptions.TemplateNotFound. It works fine while running same setup via salt-minion, so problem is probably in salt-ssh.

Debug output:

[DEBUG   ] In saltenv 'base', looking at rel_path 'salt/repo.sls' to resolve 'salt://salt/repo.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/tmp/.root_29fa9c_salt/running_data/var/cache/salt/minion/files/base/salt/repo.sls' to resolve 'salt://salt/repo.sls'
[DEBUG   ] compile template: /var/tmp/.root_29fa9c_salt/running_data/var/cache/salt/minion/files/base/salt/repo.sls
[DEBUG   ] Jinja search path: ['/var/tmp/.root_29fa9c_salt/running_data/var/cache/salt/minion/files/base']
[DEBUG   ] In saltenv 'base', looking at rel_path 'salt/map.jinja' to resolve 'salt://salt/map.jinja'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/master/files/base/salt/map.jinja' to resolve 'salt://salt/map.jinja'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://salt/map.jinja'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'salt/map.jinja'
[DEBUG   ] Jinja Error
[DEBUG   ] Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: salt/map.jinja
[DEBUG   ] Out:
[DEBUG   ] Line: 1
[DEBUG   ] TmplStr: {% from "salt/map.jinja" import salt with context %}

But file /var/cache/salt/master/files/base/salt/map.jinja is present in salt master FS.

Setup Install salt-master + salt-ssh and setup roster file. The problem occurs at Debian 9 VM with salt v3004. Tested with salt v 2019.2.8 without any problem. No minion installed at target machine, target machine is clean Debian installation.

Steps to Reproduce the behavior Run salt-ssh \* state.sls test with these states: test/init.sls:

include:
  - .repo

...

test/repo.sls:

{% from "test/map.jinja" import salt with context %}

...

test/map.jinja:

{% set salt = salt['grains.filter_by']({
  'stretch': {
    'repo': 'deb [arch=amd64] https://repo.saltproject.io/py3/debian/9/amd64/3004 stretch main',
    'key_url': 'https://repo.saltproject.io/py3/debian/9/amd64/3004/salt-archive-keyring.gpg'
  },
  'buster': {
    'repo': 'deb [arch=amd64] https://repo.saltproject.io/py3/debian/10/amd64/3004 buster main',
    'key_url': 'https://repo.saltproject.io/py3/debian/10/amd64/3004/salt-archive-keyring.gpg'
  },
}, 'lsb_distrib_codename') %}

Expected behavior No error and success as running salt via salt-minion.

Versions Report

salt --versions-report

``` Salt Version: Salt: 3004 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.5.3 docker-py: Not Installed gitdb: 2.0.0 gitpython: 2.1.1 Jinja2: 2.9.4 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: Not Installed Python: 3.5.3 (default, Apr 5 2021, 09:00:41) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 17.1.2 smmap: 2.0.1 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.1 System Versions: dist: debian 9 stretch locale: UTF-8 machine: x86_64 release: 4.9.0-16-amd64 system: Linux version: Debian GNU/Linux 9 stretch ```

[OrangeDog]

This looks like a duplicate of https://github.com/saltstack/salt/issues/21370. You can work around it using extra_filerefs.

[dosercz]

I have tried workaround with extra_filerefs, but it doesn't help in my case. So it's probably different problem. It's strange, that we were using this for years without any problem, but now it's broken in version 3004. Last tested previous version is 2019.2.8, where I can confirm no problems with that.

[baby-gnu]

Hello.

I made some tests and found it broke at 3003.

Regards.

[frenkye]

I just upgraded our master from 3002.7 to 3004 and got same error calling state over salt-ssh like, all formulas worked ok when called from master->minion or on minion via salt-call with no problem.

SSH command:

salt-ssh -v "*" state.sls prometheus

and i was getting error from all sls files:

    - Rendering SLS 'base:prometheus.node' failed: Jinja error: prometheus/map.jinja
    ...
      {% from "prometheus/map.jinja" import binExporter with context %}    <======================

I did try adding --extra-filerefs, but it did not helped. Since i have under this prometheus defined all kind of services, i wanted to lower the error output i was getting and called

salt-ssh -v "*" state.sls prometheus.nginx

To my suprise did get exececuted without problem and after that full formula works as well. The only thing maybe some corrupted cache after upgrade?

[nergdron]

I'm running into this issue now with 3004 as well. I've never had to use extra_filerefs previously, but now it seems busted. I've noticed that, without adding extra_filerefs, if I call a specific state that references the include, it works. but if I call the highstate (such as with salt-ssh hostname state.apply) then it breaks as mentioned here.

is there a workaround that actually works for this case? and is the root cause potentially the cache dir issue that seems to be causing #60003?

[TeddyAndrieux]

To me, it's the same as this one https://github.com/saltstack/salt/issues/60620 (that is wrongly closed as duplicates)

To me, this regression get introduced in 3003

[Rudd-O]

I'm hit by the same bug. It's incredible to me that salt-ssh continues to have these kinds of bugs in 2022. I'm at the breaking point where I'm strongly considering the option raised by another commenter in a separate bug: perhaps it's just better to go back to Ansible.

[nergdron]

@Rudd-O yeah, we just finished a migration from ansible to salt-ssh, just in time for all these 3003 bugs. it's been a real headache. def feels like salt-ssh isn't getting the regression testing / QA / dev love as the rest of the stack, which is really unfortunate.

[baby-gnu]

I did not try the proposed patch but I'll try to have a look at it.

I wonder if salt-ssh workgroup is working or not…

[mattboston]

I'm having the same issue on 3004.1 on CentOS 7. Added the patch #60620 and #60003 and neither of them solved the issue. Here's the configs we're using:

/etc/salt/roster:

  host: 10.10.10.123

/etc/salt/master.d/ssh_defaults.conf

roster_defaults:
  user: salt-user
  priv: /etc/salt/pki/salt-ser
  thin_dir: /etc/salt
  sudo: True
#  tty: True

ssh_minion_opts:
  gpg_keydir: /etc/salt/gpgkeys/

[babs]

The latest patch updated for 3004 me tionned in #60003 worked for me

[baby-gnu]

Hello, #61895 fixes my issues.

[johnnybubonic]

Still getting this with a 3005.1.4 master when I attempt a relative jinja import (from './vars.jinja' import...).

[stratusjerry]

Still getting this with a 3005.1.4 master when I attempt a relative jinja import (from './vars.jinja' import...).

I believe the fix is #63184 which is not in 3005.1.4 but scheduled for 3006

[nergdron]

so when is 3006 getting released?

[TeddyAndrieux]

Any chance this fix get backported to 3005 one day, because right now we are stuck with 3002.9 (and cannot upgrade for the moment to 3006 since moving to onedir is quite a huge change on our side)

[Rudd-O]

onedir

?

[jpmckinney]

@Rudd-O https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade-to-onedir.html#upgrade-to-onedir

[SkypLabs]

The bug seems gone for me in version 3007.0. Anyone can confirm to close this issue?

[aarnaud]

Didn't notice 3007 was release, generally I wait .1 or .2 before upgrading. I had too much issue in the past with upgrading. I will try to test on my labs

[baby-gnu]

Hello @SkypLabs

The bug seems gone for me in version 3007.0. Anyone can confirm to close this issue?

My tests as described in 31531 still do not work:

salt-ssh --versions-report

``` Salt Version: Salt: 3007.0 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: 18.8.0 dateutil: 2.8.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: 1.7.2 looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: 1.14.1 python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.15.1 smmap: Not Installed timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: debian n/a trixie locale: utf-8 machine: x86_64 release: 6.7.9-amd64 system: Linux version: Debian GNU/Linux n/a trixie ```

salt-ssh minion1 state.show_sls

``` [ERROR ] Rendering exception occurred Traceback (most recent call last): File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl output = template.render(**decoded_context) File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "