Open OrangeDog opened 3 years ago
Setting up the key manually and just trying salt-ssh 'myminion' test.ping
does the same thing.
I removed the /pr/*
mapping as it's unused and mentioned in the log. Behaviour is unchanged, but logs are different
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/logging.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/misc.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/misc.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/output.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/peer.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/peer.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/state.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/state.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: csrf-admin-vm.eng.cam.ac.uk
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] Loaded logstash_mod as virtual logstash
[INFO ] Processing `log_handlers.logstash`
[DEBUG ] Adding the 'logstash' provided logging handler: '<DatagramLogstashHandler (TRACE)>'
[DEBUG] Configuration file path: /etc/salt/master
[WARNING] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG] LazyLoaded flat.targets
[DEBUG] LazyLoaded jinja.render
[DEBUG] LazyLoaded yaml.render
[DEBUG] compile template: /etc/salt/roster
[DEBUG] Jinja search path: ['/var/cache/salt/master/files/base']
[DEBUG] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None, istream=None)
[DEBUG] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None, istream=None)
[DEBUG] pygit2 gitfs_provider enabled
[DEBUG] Created gitfs object with uninitialized remotes
[DEBUG] LazyLoaded gitfs.envs
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': https://github.com/saltstack-formulas/apt-formula (desired: https://github.com/saltstack-formulas/apt-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': https://github.com/saltstack-formulas/elasticsearch-formula (desired: https://github.com/saltstack-formulas/elasticsearch-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': https://github.com/saltstack-formulas/kibana-formula (desired: https://github.com/saltstack-formulas/kibana-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': https://github.com/saltstack-formulas/letsencrypt-formula (desired: https://github.com/saltstack-formulas/letsencrypt-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': https://github.com/saltstack-formulas/logstash-formula (desired: https://github.com/saltstack-formulas/logstash-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': https://github.com/saltstack-formulas/openssh-formula (desired: https://github.com/saltstack-formulas/openssh-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': https://github.com/saltstack-formulas/ufw-formula (desired: https://github.com/saltstack-formulas/ufw-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': https://github.com/saltstack-formulas/zabbix-formula (desired: https://github.com/saltstack-formulas/zabbix-formula)
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': true (desired: true)
[DEBUG] Current fetch URL for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': git@bitbucket.org:myorg/saltstack.git (desired: git@bitbucket.org:myorg/saltstack.git)
[DEBUG] Current refspecs for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'])
[DEBUG] Current http.sslVerify for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': true (desired: true)
[DEBUG] Created gitfs object for process 155335
[DEBUG] Updating gitfs fileserver cache
[DEBUG] Re-using gitfs object for process 155335
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/apt-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/apt-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/apt-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/apt-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/kibana-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/kibana-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/logstash-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/logstash-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/openssh-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/openssh-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/ufw-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/ufw-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula'
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula'
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula'
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula'
[DEBUG] Set update lock for gitfs remote 'git@bitbucket.org:myorg/saltstack.git'
[DEBUG] Fetching gitfs remote 'git@bitbucket.org:myorg/saltstack.git'
[DEBUG] gitfs remote 'git@bitbucket.org:myorg/saltstack.git' is up-to-date
[DEBUG] Removed update lock for gitfs remote 'git@bitbucket.org:myorg/saltstack.git'
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
...
The last message repeats indefinitely ~1050 times a second.
After upgrading to 3002.6 it seems to be working now. It is incredibly slow though.
# time salt-ssh 'myminion' test.ping
myminion:
True
real 2m16.370s
user 1m57.812s
sys 0m6.531s
# time ssh -i /etc/salt/pki/master/ssh/salt-ssh.rsa me@myminion.example.com echo "True"
True
real 0m0.869s
user 0m0.043s
sys 0m0.024s
Leaving it to run with the trace logging on, it eventually and repeatedly gets to that [DEBUG] In saltenv
section, for every ref in every repo. Some line pattern counts:
[INFO] Fetching file from saltenv
- 506
[DEBUG] In saltenv
- 742
[DEBUG] Re-using gitfs object for process
- 2255
[TRACE] Returning env cache data
- 123215
It seems to be ignoring the all_saltenvs
option and trying to use everything at once. Combined with whatever's hitting the cache so much it makes it take an incredibly long time.
this maybe gitfs issue and not necessarily salt-ssh, but it definitely could be both - it needs to be addressed
Regular salt
and salt-call
commands aren't doing this, so it's definitely something salt-ssh
as well.
Description Setting up
salt-ssh
for the first time, the command appears to hang forever (5+ minutes). With trace logging enabled it seems to be in some sort of infinite loop.The master is configured with multiple formulae as gitfs repos.
Setup
Steps to Reproduce the behavior
Expected behavior Command should proceed within a few seconds at most.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3002.5 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.10.1 libgit2: 0.28.3 M2Crypto: 0.31.0 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: 1.0.3 Python: 3.8.5 (default, Jan 27 2021, 15:41:15) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 18.1.1 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-67-generic system: Linux version: Ubuntu 20.04 focal ```