saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] Performance issue linked to jinja rendrer #64699

Closed doc75 closed 1 year ago

doc75 commented 1 year ago

Description Salt recently to end with timeout, without any change on my side (except certainly a salt version upgrade). I guess that it might be the latest version upgrade, but I cannot be sure as I do not know how to get back to the previous version (if you indicate how I can do this with an apt installation on Ubuntu, I'll be happy to test) After debugging a bit, it appears that the jinja renderer is now taking more than 60 second for each jinja file....

Setup I installed salt from your python3 repo using apt. No specific configuration done execept the name of the master and also enable_fqdns_grains: False on the master in the file /etc/salt/minion.d/fqdns.conf file.

Please be as specific as possible and give set-up details.

Steps to Reproduce the behavior Option #1 (On the master): salt machine-name state.highstate --state-output=changes Option #2 (On a minion): salt-call state.apply

Extract of output of salt-call state.apply

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/update.sls' using 'yaml' renderer: 0.0011582374572753906
[DEBUG   ] In saltenv 'base', looking at rel_path 'common/firewall.sls' to resolve 'salt://common/firewall.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/common/firewall.sls' to resolve 'salt://common/firewall.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/common/firewall.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG   ] In saltenv 'base', looking at rel_path 'common/firewall_macro.sls' to resolve 'salt://common/firewall_macro.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/common/firewall_macro.sls' to resolve 'salt://common/firewall_macro.sls'
[DEBUG   ] In saltenv 'base', looking at rel_path 'common/settings.sls' to resolve 'salt://common/settings.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/common/settings.sls' to resolve 'salt://common/settings.sls'
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/common/firewall.sls' using 'jinja' renderer: 60.518840074539185
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/common/firewall.sls:

On the second PROFILE line, you can see that the jinja renderer took > 60 seconds. This is the case for all .sls file (not specific to one or a subset of the .sls files).

Expected behavior Get back to previous performances (between 25 and 60 seconds for a minion to apply its state).

Screenshots Not applicable

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3005.1 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: Not Installed Python: 3.8.10 (default, May 26 2023, 14:05:08) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 20.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-152-generic system: Linux version: Ubuntu 20.04 focal ``` The only difference on the minion is the `release` in `System version`: `release: 5.13.0-52-generic`

Additional context Add any other context about the problem here.

doc75 commented 1 year ago

After reinstalling master and minion, I do not have anymore the problem. Might be due to some caching or similar topic.