search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.14k stars 5.47k forks source link

The orchestrate runner refreshes Gitfs repositories on each orchestration #35333

Open multani opened 8 years ago

multani commented 8 years ago

Description of Issue/Question

Running the state.orchestrate runner trigger each time a refresh of the Gitfs file_server backends configured on the Salt master:

I'm considering cloning off the remote Git repositories locally on the Salt master as bare repositories, schedule a periodic out of the loop synchronization between the actual remotes and the local repositories and finally configuring the Salt master to use these local repositories instead.

I wonder if there's something wrong or if that should be the way to go.

Setup

A sample of my Salt master configuration file:

ext_pillar:
- file_tree:
    follow_dir_links: true
    keep_newline: false
    root_dir: /srv/ext_pillar

file_roots:
  base:
  - /srv/salt/states

fileserver_backend:
- roots
- git

gitfs_remotes:
- git@bitbucket.org:XXX.git:
  - root: states
  - mountpoint: salt://XXX

- git@bitbucket.org:ZZZ.git:
  - base: production

# ... x 9

pillar_roots:
  base:
  - /srv/salt/pillar

reactor:
- salt/key:
  - /srv/salt/reactor/bootstrap.sls
- salt/cloud/*/created:
  - /srv/salt/reactor/bootstrap.sls

runner_dirs:
- /srv/salt/states/_runners

show_timeout: true
state_output: changes
timeout: 20

Then, launching an Orchestration does the following:

$ time sudo salt-run --log-file-level=all state.orchestrate even_a_state_which_doesnt_exist

2016-08-10 11:13:39,045 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded state.orchestrate
2016-08-10 11:13:39,046 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master
2016-08-10 11:13:39,090 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/minion.d/_schedule.conf'
2016-08-10 11:13:39,090 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/minion.d/_schedule.conf
2016-08-10 11:13:39,092 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/minion.d/minion.conf'
2016-08-10 11:13:39,092 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/minion.d/minion.conf
2016-08-10 11:13:39,093 [salt.config      ][DEBUG   ][20079] Using cached minion ID from /etc/salt/minion_id: salt-master
2016-08-10 11:13:39,094 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master
2016-08-10 11:13:39,138 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/acls.conf'
2016-08-10 11:13:39,138 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/acls.conf
2016-08-10 11:13:39,139 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/peer-communications.conf'
2016-08-10 11:13:39,140 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/peer-communications.conf
2016-08-10 11:13:39,141 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/salt-master.conf'
2016-08-10 11:13:39,141 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/salt-master.conf
2016-08-10 11:13:39,156 [salt.loader      ][TRACE   ][20079] Loading core.hwaddr_interfaces grain
2016-08-10 11:13:39,200 [salt.loader      ][TRACE   ][20079] Loading core.hostname grain
2016-08-10 11:13:39,201 [salt.loader      ][TRACE   ][20079] Loading core.get_master grain
2016-08-10 11:13:39,202 [salt.loader      ][TRACE   ][20079] Loading core.pythonversion grain
2016-08-10 11:13:39,202 [salt.loader      ][TRACE   ][20079] Loading core.path grain
2016-08-10 11:13:39,202 [salt.loader      ][TRACE   ][20079] Loading core.get_server_id grain
2016-08-10 11:13:39,203 [salt.loader      ][TRACE   ][20079] Loading core.ip6 grain
2016-08-10 11:13:39,252 [salt.loader      ][TRACE   ][20079] Loading core.ip4 grain
2016-08-10 11:13:39,307 [salt.loader      ][TRACE   ][20079] Loading core.saltversion grain
2016-08-10 11:13:39,308 [salt.loader      ][TRACE   ][20079] Loading core.saltpath grain
2016-08-10 11:13:39,308 [salt.loader      ][TRACE   ][20079] Loading core.pythonexecutable grain
2016-08-10 11:13:39,309 [salt.loader      ][TRACE   ][20079] Loading core.fqdn_ip4 grain
2016-08-10 11:13:39,309 [salt.loader      ][TRACE   ][20079] Loading core.fqdn_ip6 grain
2016-08-10 11:13:39,310 [salt.loader      ][TRACE   ][20079] Loading core.ip6_interfaces grain
2016-08-10 11:13:39,310 [salt.loader      ][TRACE   ][20079] Loading core.ip4_interfaces grain
2016-08-10 11:13:39,311 [salt.loader      ][TRACE   ][20079] Loading core.append_domain grain
2016-08-10 11:13:39,311 [salt.loader      ][TRACE   ][20079] Loading core.os_data grain
2016-08-10 11:13:39,540 [salt.modules.smbios][TRACE   ][20079] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
2016-08-10 11:13:39,561 [salt.modules.smbios][TRACE   ][20079] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
2016-08-10 11:13:39,582 [salt.utils       ][TRACE   ][20079] 'virt-what' could not be found in the following search path: ['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin']
2016-08-10 11:13:39,582 [salt.loaded.int.grains.core][DEBUG   ][20079] Please install 'virt-what' to improve results of the 'virtual' grain.
2016-08-10 11:13:39,640 [salt.loader      ][TRACE   ][20079] Loading core.zmqversion grain
2016-08-10 11:13:39,640 [salt.loader      ][TRACE   ][20079] Loading core.saltversioninfo grain
2016-08-10 11:13:39,641 [salt.loader      ][TRACE   ][20079] Loading core.pythonpath grain
2016-08-10 11:13:39,641 [salt.loader      ][TRACE   ][20079] Loading core.id_ grain
2016-08-10 11:13:39,642 [salt.loader      ][TRACE   ][20079] Loading core.locale_info grain
2016-08-10 11:13:39,642 [salt.loader      ][TRACE   ][20079] Loading core.get_machine_id grain
2016-08-10 11:13:39,643 [salt.loader      ][TRACE   ][20079] Loading core.ip_interfaces grain
2016-08-10 11:13:39,644 [salt.loaded.int.grains.disks][TRACE   ][20079] Device md0 reports itself as an HDD
2016-08-10 11:13:39,645 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sda reports itself as an SSD
2016-08-10 11:13:39,645 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sdb reports itself as an HDD
2016-08-10 11:13:39,646 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sdc reports itself as an HDD
2016-08-10 11:13:39,646 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-0 reports itself as an HDD
2016-08-10 11:13:39,646 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-1 reports itself as an HDD
2016-08-10 11:13:39,647 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-2 reports itself as an HDD
2016-08-10 11:13:39,647 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-3 reports itself as an HDD
2016-08-10 11:13:39,647 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-4 reports itself as an HDD
2016-08-10 11:13:39,647 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-5 reports itself as an HDD
2016-08-10 11:13:39,648 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-6 reports itself as an HDD
2016-08-10 11:13:39,648 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop0 reports itself as an HDD
2016-08-10 11:13:39,648 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop1 reports itself as an HDD
2016-08-10 11:13:39,648 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop2 reports itself as an HDD
2016-08-10 11:13:39,648 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop3 reports itself as an HDD
2016-08-10 11:13:39,649 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop4 reports itself as an HDD
2016-08-10 11:13:39,649 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop5 reports itself as an HDD
2016-08-10 11:13:39,649 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop6 reports itself as an HDD
2016-08-10 11:13:39,649 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop7 reports itself as an HDD
2016-08-10 11:13:39,650 [salt.loaded.int.grains.mdadm][TRACE   ][20079] mdadm devices detected: md0
2016-08-10 11:13:39,660 [salt.utils.event ][DEBUG   ][20079] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
2016-08-10 11:13:39,660 [salt.utils.event ][DEBUG   ][20079] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
2016-08-10 11:13:39,661 [salt.utils.event ][DEBUG   ][20079] Sending event - data = {'fun': 'runner.state.orchestrate', 'jid': '20160810111339046280', 'user': 'sudo_multani', '_stamp': '2016-08-10T09:13:39.660417'}
2016-08-10 11:13:39,687 [salt.loader      ][TRACE   ][20079] Error loading runners.nacl: libnacl import error, perhaps missing python libnacl package
2016-08-10 11:13:39,697 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master
2016-08-10 11:13:39,742 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/minion.d/_schedule.conf'
2016-08-10 11:13:39,742 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/minion.d/_schedule.conf
2016-08-10 11:13:39,743 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/minion.d/minion.conf'
2016-08-10 11:13:39,744 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/minion.d/minion.conf
2016-08-10 11:13:39,744 [salt.config      ][DEBUG   ][20079] Using cached minion ID from /etc/salt/minion_id: salt-master
2016-08-10 11:13:39,745 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master
2016-08-10 11:13:39,789 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/acls.conf'
2016-08-10 11:13:39,789 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/acls.conf
2016-08-10 11:13:39,790 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/peer-communications.conf'
2016-08-10 11:13:39,790 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/peer-communications.conf
2016-08-10 11:13:39,792 [salt.config      ][DEBUG   ][20079] Including configuration from '/etc/salt/master.d/salt-master.conf'
2016-08-10 11:13:39,792 [salt.config      ][DEBUG   ][20079] Reading configuration from /etc/salt/master.d/salt-master.conf
2016-08-10 11:13:39,802 [salt.loader      ][TRACE   ][20079] Loading core.hwaddr_interfaces grain
2016-08-10 11:13:39,855 [salt.loader      ][TRACE   ][20079] Loading core.hostname grain
2016-08-10 11:13:39,856 [salt.loader      ][TRACE   ][20079] Loading core.get_master grain
2016-08-10 11:13:39,856 [salt.loader      ][TRACE   ][20079] Loading core.pythonversion grain
2016-08-10 11:13:39,856 [salt.loader      ][TRACE   ][20079] Loading core.path grain
2016-08-10 11:13:39,857 [salt.loader      ][TRACE   ][20079] Loading core.get_server_id grain
2016-08-10 11:13:39,857 [salt.loader      ][TRACE   ][20079] Loading core.ip6 grain
2016-08-10 11:13:39,910 [salt.loader      ][TRACE   ][20079] Loading core.ip4 grain
2016-08-10 11:13:39,961 [salt.loader      ][TRACE   ][20079] Loading core.saltversion grain
2016-08-10 11:13:39,961 [salt.loader      ][TRACE   ][20079] Loading core.saltpath grain
2016-08-10 11:13:39,962 [salt.loader      ][TRACE   ][20079] Loading core.pythonexecutable grain
2016-08-10 11:13:39,962 [salt.loader      ][TRACE   ][20079] Loading core.fqdn_ip4 grain
2016-08-10 11:13:39,963 [salt.loader      ][TRACE   ][20079] Loading core.fqdn_ip6 grain
2016-08-10 11:13:39,964 [salt.loader      ][TRACE   ][20079] Loading core.ip6_interfaces grain
2016-08-10 11:13:39,964 [salt.loader      ][TRACE   ][20079] Loading core.ip4_interfaces grain
2016-08-10 11:13:39,965 [salt.loader      ][TRACE   ][20079] Loading core.append_domain grain
2016-08-10 11:13:39,965 [salt.loader      ][TRACE   ][20079] Loading core.os_data grain
2016-08-10 11:13:40,194 [salt.modules.smbios][TRACE   ][20079] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
2016-08-10 11:13:40,218 [salt.modules.smbios][TRACE   ][20079] DMI system-uuid value /dev/mem: No such file or directory is an invalid UUID
2016-08-10 11:13:40,243 [salt.utils       ][TRACE   ][20079] 'virt-what' could not be found in the following search path: ['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin']
2016-08-10 11:13:40,244 [salt.loaded.int.grains.core][DEBUG   ][20079] Please install 'virt-what' to improve results of the 'virtual' grain.
2016-08-10 11:13:40,306 [salt.loader      ][TRACE   ][20079] Loading core.zmqversion grain
2016-08-10 11:13:40,306 [salt.loader      ][TRACE   ][20079] Loading core.saltversioninfo grain
2016-08-10 11:13:40,306 [salt.loader      ][TRACE   ][20079] Loading core.pythonpath grain
2016-08-10 11:13:40,307 [salt.loader      ][TRACE   ][20079] Loading core.id_ grain
2016-08-10 11:13:40,307 [salt.loader      ][TRACE   ][20079] Loading core.locale_info grain
2016-08-10 11:13:40,308 [salt.loader      ][TRACE   ][20079] Loading core.get_machine_id grain
2016-08-10 11:13:40,308 [salt.loader      ][TRACE   ][20079] Loading core.ip_interfaces grain
2016-08-10 11:13:40,309 [salt.loaded.int.grains.disks][TRACE   ][20079] Device md0 reports itself as an HDD
2016-08-10 11:13:40,310 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sda reports itself as an SSD
2016-08-10 11:13:40,310 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sdb reports itself as an HDD
2016-08-10 11:13:40,311 [salt.loaded.int.grains.disks][TRACE   ][20079] Device sdc reports itself as an HDD
2016-08-10 11:13:40,311 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-0 reports itself as an HDD
2016-08-10 11:13:40,312 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-1 reports itself as an HDD
2016-08-10 11:13:40,312 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-2 reports itself as an HDD
2016-08-10 11:13:40,312 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-3 reports itself as an HDD
2016-08-10 11:13:40,313 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-4 reports itself as an HDD
2016-08-10 11:13:40,313 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-5 reports itself as an HDD
2016-08-10 11:13:40,314 [salt.loaded.int.grains.disks][TRACE   ][20079] Device dm-6 reports itself as an HDD
2016-08-10 11:13:40,314 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop0 reports itself as an HDD
2016-08-10 11:13:40,314 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop1 reports itself as an HDD
2016-08-10 11:13:40,315 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop2 reports itself as an HDD
2016-08-10 11:13:40,315 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop3 reports itself as an HDD
2016-08-10 11:13:40,316 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop4 reports itself as an HDD
2016-08-10 11:13:40,316 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop5 reports itself as an HDD
2016-08-10 11:13:40,316 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop6 reports itself as an HDD
2016-08-10 11:13:40,317 [salt.loaded.int.grains.disks][TRACE   ][20079] Device loop7 reports itself as an HDD
2016-08-10 11:13:40,317 [salt.loaded.int.grains.mdadm][TRACE   ][20079] mdadm devices detected: md0
2016-08-10 11:13:40,328 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded jinja.render
2016-08-10 11:13:40,329 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded yaml.render
2016-08-10 11:13:40,330 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded state.sls
2016-08-10 11:13:40,331 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded saltutil.is_running
2016-08-10 11:13:40,332 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded grains.get
2016-08-10 11:13:40,334 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded roots.envs
2016-08-10 11:13:40,334 [salt.utils.gitfs ][DEBUG   ][20079] gitpython gitfs_provider enabled
2016-08-10 11:13:40,334 [salt.loader      ][TRACE   ][20079] Loaded gitfs as virtual git
2016-08-10 11:13:40,335 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded git.envs
2016-08-10 11:13:40,337 [salt.utils.lazy  ][DEBUG   ][20079] Could not LazyLoad roots.init
2016-08-10 11:13:40,342 [salt.fileserver  ][DEBUG   ][20079] Updating roots fileserver cache
2016-08-10 11:13:40,361 [salt.fileserver  ][DEBUG   ][20079] Updating git fileserver cache
2016-08-10 11:13:40,367 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:XXX.git'
2016-08-10 11:13:40,367 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:XXX.git'
2016-08-10 11:13:43,302 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:XXX.git'
2016-08-10 11:13:43,303 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:43,303 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:46,252 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:46,253 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:46,253 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:49,066 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:49,067 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:49,067 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:51,879 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:51,880 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:51,880 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:54,775 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:54,776 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:54,776 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:57,649 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:57,650 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:13:57,650 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:00,501 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:00,502 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:00,502 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:08,885 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:08,886 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:08,887 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:13,622 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:13,623 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:13,623 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:16,489 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:16,490 [salt.utils.gitfs ][DEBUG   ][20079] Set update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:16,490 [salt.utils.gitfs ][DEBUG   ][20079] Fetching gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:19,361 [salt.utils.gitfs ][DEBUG   ][20079] Removed update lock for gitfs remote 'git@bitbucket.org:YYY-formula'
2016-08-10 11:14:19,362 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,375 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,379 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,382 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,385 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,388 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,391 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,394 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,397 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,400 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,403 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,405 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,407 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,408 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,410 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,412 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,413 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,415 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,417 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,418 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,420 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,422 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,423 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,425 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,427 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,429 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,430 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,432 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,433 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,435 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,437 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,439 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,440 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,442 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,444 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,446 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,447 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,449 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,451 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,453 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,455 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,457 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,458 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,460 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,462 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,464 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,465 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,467 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,469 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,471 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,473 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,477 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,481 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,485 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,488 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,492 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,496 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,500 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,503 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,507 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,511 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,514 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,518 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,523 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,528 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,532 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,537 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,541 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,545 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,549 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,554 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,558 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,562 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,566 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,571 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,573 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,576 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,578 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,580 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,582 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,584 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,586 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,593 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,599 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
...
2016-08-10 11:14:19,798 [salt.pillar      ][DEBUG   ][20079] Determining pillar cache
2016-08-10 11:14:19,805 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded jinja.render
2016-08-10 11:14:19,805 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded yaml.render
2016-08-10 11:14:19,806 [salt.template    ][DEBUG   ][20079] compile template: /srv/salt/pillar/top.sls
2016-08-10 11:14:19,806 [salt.utils.jinja ][DEBUG   ][20079] Jinja search path: ['/srv/salt/pillar']
2016-08-10 11:14:19,812 [salt.template    ][PROFILE ][20079] Time (in seconds) to render '/srv/salt/pillar/top.sls' using 'jinja' renderer: 0.0059220790863
2016-08-10 11:14:19,812 [salt.template    ][DEBUG   ][20079] Rendered data from file: /srv/salt/pillar/top.sls:
base:
...
2016-08-10 11:14:19,851 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded config.get
2016-08-10 11:14:19,852 [salt.loaded.int.render.yaml][DEBUG   ][20079] Results of YAML rendering:  ...
2016-08-10 11:14:19,852 [salt.template    ][PROFILE ][20079] Time (in seconds) to render '/srv/salt/pillar/top.sls' using 'yaml' renderer: 0.0391838550568
2016-08-10 11:14:19,853 [salt.minion      ][DEBUG   ][20079] compound_match: ...
...
2016-08-10 11:14:19,879 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded file_tree.ext_pillar
2016-08-10 11:14:19,879 [salt.state       ][INFO    ][20079] Loading fresh modules for state activity
2016-08-10 11:14:19,888 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded jinja.render
2016-08-10 11:14:19,889 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded yaml.render
2016-08-10 11:14:19,895 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,902 [salt.fileclient  ][DEBUG   ][20079] Could not find file from saltenv 'base', u'salt://even_a_state_which_doesnt_exist.sls'
2016-08-10 11:14:19,907 [salt.fileserver  ][TRACE   ][20079] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
2016-08-10 11:14:19,915 [salt.fileclient  ][DEBUG   ][20079] Could not find file from saltenv 'base', u'salt://even_a_state_which_doesnt_exist/init.sls'
2016-08-10 11:14:19,915 [salt.template    ][DEBUG   ][20079] compile template: False
[ERROR   ] Template was specified incorrectly: False
2016-08-10 11:14:19,915 [salt.template    ][ERROR   ][20079] Template was specified incorrectly: False
2016-08-10 11:14:19,916 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded highstate.output
salt-master_master:
2016-08-10 11:14:19,917 [salt.output      ][TRACE   ][20079] data = {'salt-master_master': ["No matching sls found for 'even_a_state_which_doesnt_exist' in env 'base'"]}
    Data failed to compile:
----------
    No matching sls found for 'even_a_state_which_doesnt_exist' in env 'base'
2016-08-10 11:14:19,917 [salt.utils.event ][DEBUG   ][20079] Sending event - data = {'jid': '20160810111339046280', 'return': {'outputter': 'highstate', 'data': {'salt-master_master': ["No matching sls found for 'even_a_state_which_doesnt_exist' in env 'base'"]}}, 'success': True, '_stamp': '2016-08-10T09:14:19.916814', 'user': 'sudo_multani', 'fun': 'runner.state.orchestrate'}
2016-08-10 11:14:19,918 [salt.utils.lazy  ][DEBUG   ][20079] LazyLoaded local_cache.prep_jid
2016-08-10 11:14:19,919 [salt.client.mixins][INFO    ][20079] Runner completed: 20160810111339046280
2016-08-10 11:14:19,944 [salt.runner      ][DEBUG   ][20079] Runner return: {'outputter': 'highstate', 'data': {'salt-master_master': ["No matching sls found for 'even_a_state_which_doesnt_exist' in env 'base'"]}}

sudo salt-run --log-file-level=all state.orchestrate   2.98s user 1.03s system 9% cpu 41.831 total

So, even to trigger an orchestration which doesn't actually exist, it takes 41 seconds, of which 39 seconds are spent updating the Git repositories (which were refreshed seconds ago.)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2015.8.11

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: debian 8.5 
        machine: x86_64
        release: 3.16.0-4-amd64
         system: debian 8.5

(I don't want to upgrade to 2016 yet, as it still has some more serious issues that are affecting us.)

Ch3LL commented 8 years ago

ping @terminalmage Can I get your input on this issue.

I am able to replicate this every time I run state.orchestrat I see the attempt to update the gitfs cache:

[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] LazyLoaded git.envs
[DEBUG   ] Updating git fileserver cache
[DEBUG   ] Set update lock for gitfs remote 'https://github.com/saltstack-formulas/salt-formula.git'
[DEBUG   ] Fetching gitfs remote 'https://github.com/saltstack-formulas/salt-formula.git'
[DEBUG   ] gitfs remote 'https://github.com/saltstack-formulas/salt-formula.git' is up-to-date
[DEBUG   ] Removed update lock for gitfs remote 'https://github.com/saltstack-formulas/salt-formula.git'

Is there a way to set this so this does not occur each time?

terminalmage commented 8 years ago

There's no setting that lets you do this. I'll need to investigate.

jerrykan commented 8 years ago

In the duplicate issue (#35585) I requested that a option be added to disable the refresh of gitfs remotes with salt-call, but thinking about this some more it might be a good idea to instead implement an option the sets the how often the cache is invalidated and updated.

According to the documentation the fileserver backends are updated every 60 seconds. Even when running with master mode this seems a bit too eager (at least in our environment). It would be good if this setting could be adjusted to any value (with '0' meaning "never") with the setting being honoured no matter how salt is invoked (ie. using either master or standalone).

This way if I know that the salt formulas I'm using change only occasionally I could set the fileserver backends to update once a day, with the option to invoke runner.fileserver.update manually if I need to force an update of the backends.

multani commented 8 years ago

I also thought about it yesterday and although we are not there yet, the ideal situation would be to be able to completely disable preemptively polling the Git repositories and to have a way to trigger this polling on demand with a trigger when a push has been made. As @jerrykan said, 99% of the time there's no changes to pull (in our case at least).

jerrykan commented 8 years ago

@multani while I agree it would be ideal to only trigger an update once a push has been made this may not always be possible (ie. you may not be using a git host that supports triggering such an event). So polling would still be useful, but would ideally be optional (for the use-case you mention) and adjustable (so polling doesn't occur too often).

multani commented 7 years ago

FYI, I'm now running 2016.3.4 and running the orchestration still does a Git fetch on all the repositories configured on the master ... BUT, although there were changes to fetch from the Git remotes, these changes were NOT taken into account during the orchestration. Running first salt-run fileserver.update before the orchestration does work though.

(to be clear:

  1. I pushed a change on the Git remote
  2. I ran the orchestration state, which did the Fetching gitfs remote ... as explained in the first post, but they were no indication about pulled changes
  3. The orchestration completed without any changes
  4. I did a salt-run fileserver.update, which indicated this time it got the changes (but I'm 200% sure they were already on the remote at the time I did the first orchestration)
  5. I re-ran the orchestration state, which re-did the Git fetch on all the remote, still with no pulled changes
  6. The orchestration completed, this time with the expected changes.

)

Ch3LL commented 7 years ago

@multani correct me if i'm wrong but this newest issue you just brought up seems to be a seperate issue where the orchestartion is not updating the gitfs. Would you mind opening another issue with more details if this is the case?

Also for now i'll label this as a feature until terminalimage can dive into this more. Thanks

Hicks31 commented 6 years ago

Hi,

I found the same problem with svnfs: 

[DEBUG   ] Updating svn fileserver cache
[DEBUG   ] Set lock for http://mysvn
[DEBUG   ] svnfs is fetching from http://mysvn

Should i open a new issue? Is there a way to prevent this refresh on salt 2017.1 ?

junovitch commented 6 years ago

Get subscribed on this... I've seen my calls to things like salt-run manage.status get a lot slower moving from 2016.11 to 2018.3.0 recently and the gitfs syncing going on during every single salt-run looks like a bit part of the time. It looks like I was able to mitigate it a bit by using a blacklist of branches to exclude but I didn't time a before/after. Is this the same root cause under the hood?

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

jerrykan commented 5 years ago

This is still an issue that hasn't been addressed.

stale[bot] commented 5 years ago

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

jerrykan commented 4 years ago

This is still an issue that hasn't been addressed.

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

rosscdh commented 3 years ago

+1 still slow and tries to update on running state.orch as above

MurzNN commented 3 years ago

Same problem with very slow salt-ssh execution when using gitfs repos - it rebuilds them on every execution (even with gitfs_update_interval: 86400) - for example - simple command:

time salt-ssh brick-tmp1 test.ping

executes about 35 seconds!

Via adding -l debug suffix I see a lot of strings like:

[DEBUG   ] Current http.sslVerify for gitfs remote 'git://github.com/saltstack-formulas/nginx-formula': true (desired: true)
[DEBUG   ] Current fetch URL for gitfs remote 'git://github.com/saltstack-formulas/php-formula': git://github.com/saltstack-formulas/php-formula (desired: git://github.com/saltstack-formulas/php-formula)
[DEBUG   ] Current refspecs for gitfs remote 'git://github.com/saltstack-formulas/php-formula': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'])
...
[DEBUG   ] Updating gitfs fileserver cache
[DEBUG   ] Re-using gitfs object for process 77007
[DEBUG   ] Set update lock for gitfs remote 'git://github.com/saltstack-formulas/salt-formula'
[DEBUG   ] Fetching gitfs remote 'git://github.com/saltstack-formulas/salt-formula'
[DEBUG   ] Popen(['git', 'fetch', '-v', 'origin'], cwd=/var/cache/salt/master/gitfs/3c36313a8faa2a7343c915489b39c009120f2e71e3e41ce151f4d4386d09d7df, universal_newlines=True, shell=None, istream=None)
[DEBUG   ] Removed update lock for gitfs remote 'git://github.com/saltstack-formulas/salt-formula'
[DEBUG   ] Set update lock for gitfs remote 'git://github.com/saltstack-formulas/docker-formula'
[DEBUG   ] Fetching gitfs remote 'git://github.com/saltstack-formulas/docker-formula'
[DEBUG   ] Popen(['git', 'fetch', '-v', 'origin'], cwd=/var/cache/salt/master/gitfs/3a51c3f210b582f945814687f90f6c318c2461d03a7d9f7d2134a0ca340b55f9, universal_newlines=True, shell=None, istream=None)
[DEBUG   ] Removed update lock for gitfs remote 'git://github.com/saltstack-formulas/docker-formula'
[DEBUG   ] Set update lock for gitfs remote 'git://github.com/saltstack-formulas/pam-formula'
...
[DEBUG   ] Re-using gitfs object for process 76103
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/salt-formula duration=0.0003561973571777344 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/docker-formula duration=0.0002968311309814453 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/pam-formula duration=0.00031256675720214844 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/systemd-formula duration=0.00028514862060546875 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/users-formula duration=0.00030159950256347656 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/mysql-formula duration=0.0002944469451904297 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/postgres-formula duration=0.004677772521972656 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/prometheus-formula duration=0.00039696693420410156 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/grafana-formula duration=0.00029754638671875 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/nginx-formula duration=0.00029540061950683594 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/php-formula duration=0.0002942085266113281 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/apache-formula duration=0.00032329559326171875 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/node-formula duration=0.0002810955047607422 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/fail2ban-formula duration=0.000293731689453125 seconds
[PROFILE ] gitfs file_name cache rebuild repo=git://github.com/saltstack-formulas/letsencrypt-formula duration=0.0002884864807128906 seconds
[DEBUG   ] Re-using gitfs object for process 76103

Version is 3002.6+ds-1

sagetherage commented 3 years ago

This needs investigation as to what could be done - we will review a PR if anyone on this issue is willing and able to submit one! :)

jerrykan commented 1 year ago

I've just hit this issue again when trying to set up some CI stuff. After a short bit of time digging through the code using the debugger I noticed that the __fs_update config option gets set to True once the gitfs remotes are updated. So for a really hacky workaround you could place __fs_update: True in the Salt configuration to prevent updating the remotes each time salt is called. Though if you are considering this sort of work-around a better option might be to create a localconfig file and only include the setting as necessary.

ie, create the file /etc/salt/skip_remote_update with the following contents:

__fs_update: True

And include it using the localconfig option when you want to avoid updating the remotes.

salt-call state.apply --output-diff localconfig=/etc/salt/skip_remote_update

This may have other unintended side-effects, but may be a workable work-around until a better solution is implement in the Salt codebase.

fs-nicola-worthington commented 1 week ago

Not read the full comment history. Was directed to this by a colleague just after we implemented a workaround of this issue on 3004. We have 3 repos used with gitfs for files and pillars. gitfs was causing pretty much every minion action to be CPU bound on the master on a single CPU core. With ~24GB/20,000 files in Git and 1,300 minions on a huge Salt master (64 cores, 128GB RAM), this was completely unusable.

Ended up ripping all of the gitfs out and replacing it with a very small shell script which will pull the updates from our repositories only when our CI pipeline notices a new commit.

This results in an order or magnitude better performance.