Open baby-gnu opened 8 months ago
dwoz
commented
8 months ago @baby-gnu
This is likely a duplicate of #66124. Are you able to test the changes to salt/pillar/__init__.py from my PR? https://github.com/saltstack/salt/pull/66128/files#diff-8b80a647365268f00563c1c4f3e787812a7ad3e07744c7fd71ffc21c7c75b4af
baby-gnu
commented
8 months ago @baby-gnu This is likely a duplicate of #66124. Are you able to test the changes to
salt/pillar/__init__.pyfrom my PR? https://github.com/saltstack/salt/pull/66128/files#diff-8b80a647365268f00563c1c4f3e787812a7ad3e07744c7fd71ffc21c7c75b4af
I applied the patch manually and it did not solve my problem with cp.list_states.
diff -u /opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.py.orig /opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.pydwoz
commented
8 months ago @lkubb any thoughts on this one?
baby-gnu
commented
8 months ago @lkubb any thoughts on this one?
If I understand the log diff correctly, some fileserver caches are not rebuild :thinking:
lkubb
commented
8 months ago I failed to reproduce this with the test suite, then I upgraded my master to 3006.7 [1]
Then:
salt-run fileserver.clear_file_list_cache/salt-run saltutil.sync_all), I suddenly could reproduce it. cp wrapper file client's opts.cp wrapper file client's opts.gitfs_remotes and what I previously wrote is incorrect.Edit2: @baby-gnu Does salt-call cp.list_states or salt-run fileserver.file_list work for you? When it breaks for me via salt-ssh, it also breaks in regular Salt. I'm not too familiar with gitfs, so I'll stop debugging here. Calling salt-run fileserver.update does not fix it once it's broken.
From what I can tell, the broken file client's opts were not set to the master's opts with overridden cachedir as usual, but to the minion-master hybrid ones that are used for wrapper modules.
Since I cannot reproduce it anymore and had to rush, take the above with a grain of salt. If I'm correct though, I would assume it's related to the latest loader/file client changes.
[1] This severely broke my environment: the pillar was always empty, refreshing revealed the master was returning an empty string as the pillar. #66128 fixed it. Could not reproduce this later as well.
baby-gnu
commented
8 months ago Hello
Edit2: @baby-gnu Does
salt-call cp.list_statesorsalt-run fileserver.file_listwork for you? When it breaks for me viasalt-ssh, it also breaks in regular Salt. I'm not too familiar with gitfs, so I'll stop debugging here. Callingsalt-run fileserver.updatedoes not fix it once it's broken.
Interesting test, on a pristine setup with cache directory cleaned:
salt-ssh minion1 cp.list_states does not worksalt-call --local cp.list_states workssalt-ssh minion1 cp.list_states worksgitfs_remotessalt-ssh minion1 cp.list_states does not list new states (but has the previous ones)salt-call --local cp.list_states does not list new states (but has the previous ones)salt-run fileserver.update returns Truesalt-call --local cp.list_states workssalt-ssh minion1 cp.list_states worksAnother much simpler test:
salt-ssh minion1 cp.list_states does not worksalt-run fileserver.update returns Truesalt-ssh minion1 cp.list_states worksSo:
salt-call unblock the salt-ssh situation the first time but nothing happens when adding new gitfs_remotessalt-run fileserver.update unblock the situation all the timesRegards.
lkubb
commented
8 months ago Thanks for the feedback! Not sure why I couldn't unblock it with a fileserver update, but awesome that it works for you.
It seems the issue is likely not specific to salt-ssh then and related to automatic fileserver refreshes. Still weird that salt-call --local unblocks it the first time when salt-ssh does not since both use the same fileclient.
baby-gnu
commented
8 months ago Where I need 2 salt-call --local cp.list_states to see list of states or salt-run fileserver.update
SkypLabs
commented
8 months ago It seems I'm hitting the same salt-ssh bugged behaviour as the one described by @baby-gnu in his issue description with version 3007.0. I had a look at the Git repositories in my local cache and it turned out they were all in the main branch, which is the default branch set in my ~/.gitconfig but not the one used by my remote formulas nor the default branch I set as value of gitfs_base in master. When I manually switched to the master branch and ran salt-ssh again, it solved the issue.
I hope this might help you find and troubleshoot the root cause of the problem.
SkypLabs
commented
8 months ago I had a look at the Git repositories in my local cache and it turned out they were all in the main branch, which is the default branch set in my ~/.gitconfig but not the one used by my remote formulas nor the default branch I set as value of gitfs_base in master. When I manually switched to the master branch and ran salt-ssh again, it solved the issue.
Actually, it doesn't matter what the current branch is. Only a git fetch does the trick.
# Git repository of a remote formula in cache.
cd /tmp/.salt-root/var/cache/salt/master/gitfs/b0ECZpNJPOnKvlruA3uw6LbNIKomJ5gPGuWSPOAavTk=/_
git fetch
# I can now see my formula in the list.
salt-ssh '*' cp.list_statesAs a quick-and-dirty workaround (at least in my case), to run git fetch for all cached Git repositories:
cd /tmp/.salt-root/var/cache/salt/master/gitfs/
find -maxdepth 2 -type d -name '*_' -exec git -C {} fetch \;It seems I'm hitting the same
salt-sshbugged behaviour as the one described by @baby-gnu in his issue description with version3007.0. I had a look at the Git repositories in my local cache and it turned out they were all in themainbranch, which is the default branch set in my~/.gitconfigbut not the one used by my remote formulas nor the default branch I set as value ofgitfs_baseinmaster. When I manually switched to themasterbranch and ransalt-sshagain, it solved the issue.I hope this might help you find and troubleshoot the root cause of the problem.
Thanks, I tried to set the base parameter for all my gitfs_remotes but it does not work. Doing a manual git fetch for all cached gitfs directories does the job.
Description
States in gitfs are unusable with
salt-sshsince 3006.7Setup
I use a personal
salt-sshconfiguration, the minimal settings is the following:
``` yaml # -*- yaml -*- salt-ssh: config_dir: ~/.salt ```~/.salt/Saltfile
``` yaml # -*- mode: yaml; coding: utf-8 -*- #### #### Global parameters #### # Unfortunately, salt does not support ~ expension # We need to use absolute path for `root_dir` root_dir: /home/me/.salt pki_dir: pki cachedir: cache sock_dir: run pidfile: pids log_file: logs/master.log key_logfile: logs/key.log # Global level logged in file log_level_logfile: debug ## ## file server ## fileserver_backend: - roots - gitfs # Unfortunately, salt does not support ~ expension # We need to use absolute path for `file_roots` and `pillar_roots` file_roots: base: - /home/me/.salt/srv/salt pillar_roots: base: - /home/me/.salt/srv/pillar # List of formula I'm using gitfs_remotes: - https://github.com/saltstack-formulas/openssh-formula.git: - base: v3.0.3 - https://github.com/saltstack-formulas/template-formula.git #### #### Salt-SSH specific configuration #### ssh_minion_opts: log_level: debug ssh_log_file: logs/ssh.log ssh_use_home_key: True ssh_timeout: 5 roster_defaults: # Use ssh-agent authentication priv: agent-forwarding ```~/.salt/master
``` yaml # -*- mode: yaml; coding: utf-8 -*- minion1: host: minion1.example.net user: root ```~/.salt/rosterSteps to Reproduce the behavior
salt-ssh3006.7pygit2using the following command:sudo salt-pip install pygit2salt-ssh minion1 cp.list_statesExpected behavior
All states from formulas git repositories are listed like
``` [WARNING ] Cache version mismatch clearing: '/home/me/.salt/cache/gitfs' minion1: - TEMPLATE - TEMPLATE._mapdata - TEMPLATE.clean - TEMPLATE.config - TEMPLATE.config.clean - TEMPLATE.config.file - TEMPLATE.package - TEMPLATE.package.clean - TEMPLATE.package.install - TEMPLATE.service - TEMPLATE.service.clean - TEMPLATE.service.running - TEMPLATE.subcomponent - TEMPLATE.subcomponent.clean - TEMPLATE.subcomponent.config - TEMPLATE.subcomponent.config.clean - TEMPLATE.subcomponent.config.file - _pillar.known_hosts_salt_ssh - openssh - openssh._mapdata - openssh.auth - openssh.auth_map - openssh.banner - openssh.client - openssh.config - openssh.config_ini - openssh.gather_host_keys - openssh.known_hosts - openssh.moduli - test.salt.pillar.default - test.salt.pillar.define_roles - test.salt.pillar.gentoo - test.salt.pillar.top - test.salt.pillar.upstart ```salt-ssh minion1 cp.list_statesScreenshots
Versions Report
salt-ssh --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt: 3006.7 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: 1.7.2 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: 1.14.1 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.15.1 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: debian n/a trixie locale: utf-8 machine: x86_64 release: 6.6.11-amd64 system: Linux version: Debian GNU/Linux n/a trixie ```Additional context
This was working fine with
3006.6, here is a diff of the debug log
``` diff --- 3006.6.log 2024-02-27 15:09:27.084707047 +0100 +++ 3006.7.log 2024-02-27 15:09:30.980715906 +0100 @@ -7,14 +7,8 @@ LazyLoaded yaml.render compile template: /home/me/.salt/roster Jinja search path: ['/home/me/.salt/cache/files/base'] -Time (in seconds) to render '/home/me/.salt/roster' using 'jinja' renderer: 0.011598825454711914 -Rendered data from file: /home/me/.salt/roster: -# -*- mode: yaml; coding: utf-8 -*- - -minion1: - host: minion1.example.net - user: root -Time (in seconds) to render '/home/me/.salt/roster' using 'yaml' renderer: 0.0032024383544921875 +Time (in seconds) to render '/home/me/.salt/roster' using 'jinja' renderer: 0.0016009807586669922 +Time (in seconds) to render '/home/me/.salt/roster' using 'yaml' renderer: 0.0003859996795654297 The functions from module 'roster_matcher' are being loaded by dir() on the loaded module LazyLoaded roster_matcher.targets The functions from module 'roots' are being loaded by dir() on the loaded module @@ -41,19 +35,7 @@ http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/template-formula.git' set to true Config updates for gitfs remote 'https://github.com/saltstack-formulas/template-formula.git' written to /home/me/.salt/cache/gitfs/hjac4+R0YxjkGX87hEu_77Q_tdzH1DDVE_YdWBaHrk0=/_/.git/config Wrote new gitfs remote map to /home/me/.salt/cache/gitfs/remote_map.txt -Created gitfs object for process 179572 -Updating roots fileserver cache -Updating gitfs fileserver cache -Re-using gitfs object for process 179572 -Wrote new gitfs remote map to /home/me/.salt/cache/gitfs/remote_map.txt -Set update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula.git' -Fetching gitfs remote 'https://github.com/saltstack-formulas/openssh-formula.git' -gitfs received 2501 objects for remote 'https://github.com/saltstack-formulas/openssh-formula.git' -Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula.git' -Set update lock for gitfs remote 'https://github.com/saltstack-formulas/template-formula.git' -Fetching gitfs remote 'https://github.com/saltstack-formulas/template-formula.git' -gitfs received 3528 objects for remote 'https://github.com/saltstack-formulas/template-formula.git' -Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/template-formula.git' +Created gitfs object for process 180703 Using contextvarsdiff -u 3006.6.log 3006.7.log