search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.06k stars 5.47k forks source link

[BUG] salt-ssh processing all gitfs refs regardless of saltenv settings #59839

Open OrangeDog opened 3 years ago

OrangeDog commented 3 years ago

Description Setting up salt-ssh for the first time, the command appears to hang forever (5+ minutes). With trace logging enabled it seems to be in some sort of infinite loop.

The master is configured with multiple formulae as gitfs repos.

Setup

fileserver_backend:
  - gitfs
gitfs_privkey: /etc/salt/master.d/salt-git-rsa
gitfs_pubkey: /etc/salt/master.d/salt-git-rsa.pub
gitfs_refspecs:
  - '+refs/heads/*:refs/remotes/origin/*'
  - '+refs/tags/*:refs/tags/*'
  - '+refs/pull/*/head:refs/remotes/origin/pr/*'
  - '+refs/pull/*/merge:refs/remotes/origin/merge/*'
gitfs_remotes:
  - https://github.com/saltstack-formulas/apt-formula:
    - all_saltenvs: v0.10.2
  - https://github.com/saltstack-formulas/elasticsearch-formula:
    - all_saltenvs: f70e9da
  - https://github.com/saltstack-formulas/kibana-formula:
    - all_saltenvs: merge/22
  - https://github.com/saltstack-formulas/letsencrypt-formula:
    - all_saltenvs: v2.0.0
  - https://github.com/saltstack-formulas/logstash-formula:
    - all_saltenvs: bab7586
  - https://github.com/saltstack-formulas/openssh-formula:
    - all_saltenvs: v2.0.6
  - https://github.com/saltstack-formulas/ufw-formula:
    - all_saltenvs: v0.5.7
  - https://github.com/saltstack-formulas/zabbix-formula:
    - all_saltenvs: v1.0.3
  - git@bitbucket.org:myorg/saltstack.git:
    - root: salt

state_top_saltenv: base
top_file_merging_strategy: same
myminion:
  host: myminion.example.com
  user: me

Steps to Reproduce the behavior

salt-ssh --askpass --key-deploy 'myminion' test.ping -ltrace
...
[TRACE] Lockfile /var/cache/salt/master/file_lists/gitfs/.pr_|-173.w created
[TRACE] Start rebuilding gitfs file_list cache
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/apt-formula duration=0.01137399673461914 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/elasticsearch-formula duration=0.0034940242767333984 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/kibana-formula duration=0.003408670425415039 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/letsencrypt-formula duration=0.014557361602783203 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/logstash-formula duration=0.0016510486602783203 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/openssh-formula duration=0.02264404296875 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/ufw-formula duration=0.01248621940612793 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/zabbix-formula duration=0.4704718589782715 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=git@bitbucket.org:sustainableroadfreight/saltstack.git duration=0.00020551681518554688 seconds
[TRACE] Lockfile /var/cache/salt/master/file_lists/gitfs/.pr_|-173.w removed
[TRACE] Finished rebuilding gitfs file_list cache
[DEBUG] Re-using gitfs object for process 1614953
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[DEBUG] Re-using gitfs object for process 1614953
[DEBUG] Re-using gitfs object for process 1614953
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[DEBUG] In saltenv 'pr/173', looking at rel_path '_modules/ufw.py' to resolve 'salt://_modules/ufw.py'
[DEBUG] In saltenv 'pr/173', ** considering ** path '/var/cache/salt/master/files/pr/173/_modules/ufw.py' to resolve 'salt://_modules/ufw.py'
[DEBUG] Re-using gitfs object for process 1614953
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[DEBUG] Re-using gitfs object for process 1614953
[DEBUG] Re-using gitfs object for process 1614953
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[DEBUG] In saltenv 'pr/173', looking at rel_path '_states/ufw.py' to resolve 'salt://_states/ufw.py'
[DEBUG] In saltenv 'pr/173', ** considering ** path '/var/cache/salt/master/files/pr/173/_states/ufw.py' to resolve 'salt://_states/ufw.py'
[DEBUG] Re-using gitfs object for process 1614953
[TRACE] Lockfile /var/cache/salt/master/file_lists/gitfs/.pr_|-27.w created
[TRACE] Start rebuilding gitfs file_list cache
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/apt-formula duration=0.009964942932128906 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/elasticsearch-formula duration=0.0029489994049072266 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/kibana-formula duration=0.0025043487548828125 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/letsencrypt-formula duration=0.009569644927978516 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/logstash-formula duration=0.0014698505401611328 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/openssh-formula duration=0.017264127731323242 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[PROFILE] gitfs file_name cache rebuild repo=https://github.com/saltstack-formulas/ufw-formula duration=0.010399580001831055 seconds
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
Exiting gracefully on Ctrl-c

Expected behavior Command should proceed within a few seconds at most.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3002.5 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.10.1 libgit2: 0.28.3 M2Crypto: 0.31.0 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: 1.0.3 Python: 3.8.5 (default, Jan 27 2021, 15:41:15) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 18.1.1 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-67-generic system: Linux version: Ubuntu 20.04 focal ```
OrangeDog commented 3 years ago

Setting up the key manually and just trying salt-ssh 'myminion' test.ping does the same thing.

OrangeDog commented 3 years ago

I removed the /pr/* mapping as it's unused and mentioned in the log. Behaviour is unchanged, but logs are different

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/logging.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/misc.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/misc.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/output.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/peer.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/peer.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/state.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/state.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: csrf-admin-vm.eng.cam.ac.uk
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] Loaded logstash_mod as virtual logstash
[INFO    ] Processing `log_handlers.logstash`
[DEBUG   ] Adding the 'logstash' provided logging handler: '<DatagramLogstashHandler (TRACE)>'
[DEBUG] Configuration file path: /etc/salt/master 
[WARNING] Insecure logging configuration detected! Sensitive data may be logged. 
[DEBUG] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc 
[DEBUG] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc 
[DEBUG] LazyLoaded flat.targets 
[DEBUG] LazyLoaded jinja.render 
[DEBUG] LazyLoaded yaml.render 
[DEBUG] compile template: /etc/salt/roster 
[DEBUG] Jinja search path: ['/var/cache/salt/master/files/base'] 
[DEBUG] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None, istream=None) 
[DEBUG] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None, istream=None) 
[DEBUG] pygit2 gitfs_provider enabled 
[DEBUG] Created gitfs object with uninitialized remotes 
[DEBUG] LazyLoaded gitfs.envs 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': https://github.com/saltstack-formulas/apt-formula (desired: https://github.com/saltstack-formulas/apt-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/apt-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': https://github.com/saltstack-formulas/elasticsearch-formula (desired: https://github.com/saltstack-formulas/elasticsearch-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': https://github.com/saltstack-formulas/kibana-formula (desired: https://github.com/saltstack-formulas/kibana-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': https://github.com/saltstack-formulas/letsencrypt-formula (desired: https://github.com/saltstack-formulas/letsencrypt-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': https://github.com/saltstack-formulas/logstash-formula (desired: https://github.com/saltstack-formulas/logstash-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': https://github.com/saltstack-formulas/openssh-formula (desired: https://github.com/saltstack-formulas/openssh-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': https://github.com/saltstack-formulas/ufw-formula (desired: https://github.com/saltstack-formulas/ufw-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': https://github.com/saltstack-formulas/zabbix-formula (desired: https://github.com/saltstack-formulas/zabbix-formula) 
[DEBUG] Current refspecs for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula': true (desired: true) 
[DEBUG] Current fetch URL for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': git@bitbucket.org:myorg/saltstack.git (desired: git@bitbucket.org:myorg/saltstack.git) 
[DEBUG] Current refspecs for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/pull/*/merge:refs/remotes/origin/merge/*', '+refs/tags/*:refs/tags/*']) 
[DEBUG] Current http.sslVerify for gitfs remote 'git@bitbucket.org:myorg/saltstack.git': true (desired: true) 
[DEBUG] Created gitfs object for process 155335 
[DEBUG] Updating gitfs fileserver cache 
[DEBUG] Re-using gitfs object for process 155335 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/apt-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/apt-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/apt-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/apt-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/elasticsearch-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/kibana-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/kibana-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/kibana-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/letsencrypt-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/logstash-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/logstash-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/logstash-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/openssh-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/openssh-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/openssh-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/ufw-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/ufw-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/ufw-formula' 
[DEBUG] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula' 
[DEBUG] Fetching gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula' 
[DEBUG] gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/zabbix-formula' 
[DEBUG] Set update lock for gitfs remote 'git@bitbucket.org:myorg/saltstack.git' 
[DEBUG] Fetching gitfs remote 'git@bitbucket.org:myorg/saltstack.git' 
[DEBUG] gitfs remote 'git@bitbucket.org:myorg/saltstack.git' is up-to-date 
[DEBUG] Removed update lock for gitfs remote 'git@bitbucket.org:myorg/saltstack.git' 
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p 
[TRACE] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
...

The last message repeats indefinitely ~1050 times a second.

OrangeDog commented 3 years ago

After upgrading to 3002.6 it seems to be working now. It is incredibly slow though.

# time salt-ssh 'myminion' test.ping
myminion:
    True

real    2m16.370s
user    1m57.812s
sys     0m6.531s
# time ssh -i /etc/salt/pki/master/ssh/salt-ssh.rsa me@myminion.example.com echo "True"
True

real    0m0.869s
user    0m0.043s
sys     0m0.024s
OrangeDog commented 3 years ago

Leaving it to run with the trace logging on, it eventually and repeatedly gets to that [DEBUG] In saltenv section, for every ref in every repo. Some line pattern counts:

[INFO] Fetching file from saltenv - 506 [DEBUG] In saltenv - 742 [DEBUG] Re-using gitfs object for process - 2255 [TRACE] Returning env cache data - 123215

It seems to be ignoring the all_saltenvs option and trying to use everything at once. Combined with whatever's hitting the cache so much it makes it take an incredibly long time.

sagetherage commented 3 years ago

this maybe gitfs issue and not necessarily salt-ssh, but it definitely could be both - it needs to be addressed

OrangeDog commented 3 years ago

Regular salt and salt-call commands aren't doing this, so it's definitely something salt-ssh as well.