search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] GitFS hang until longest update_interval #64615

Open twangboy opened 1 year ago

twangboy commented 1 year ago

Description GitFS will hang after a certain amount of time. On my machine, it was about 5 minutes. I don't know if it's based on time or the number of refreshes, which on my machine was about 10 refreshes. It will do no more fetches until the last defined GitFS remote update_interval fires. It is then issued a new PID and the cycle starts over. This would not be noticeable if your longest update_interval is close to the other update intervals. But if you have an update interval of a day (86400), then your GitFS remotes will update for 5 minutes every day.

Setup master config

loop_interval: 15

log_level_logfile: all

gitfs_disable_saltenv_mapping: True
gitfs_provider: pygit2
gitfs_remotes:
  - https://github.com/saltstack/salt-bootstrap.git:
    - base: stable
    - mountpoint: salt://bootstrap
    - update_interval: 360
  - https://github.com/DataDog/datadog-formula.git:
    - base: 3.4
    - mountpoint: salt://datadog
    - root: datadog
    - update_interval: 30
fileserver_backend:
  - roots
  - gitfs
# Files on GitHub to ignore on the SaltStack fileserver
file_ignore_regex:
  - '/\.git($|/)'
  - '/\.gitignore'
  - '/\.pre-commit-config.yaml'

Steps to Reproduce the behavior Start the master:

sudo salt-master -l debug

In another terminal on the same machine, tail the master log grepping for gitfs:

sudo tail -f /var/log/salt/master | grep gitfs

You will see the following output:

[shane@localhost ~]$ sudo tail -f /var/log/salt/master | grep gitfs
[sudo] password for shane:

>>>>>>>> Notice that the refresh occurs every 30 seconds as expected

2023-07-07 15:51:38,602 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:51:38,602 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:51:38,603 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:51:38,604 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:51:38,956 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:51:38,958 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:08,960 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:52:08,961 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:52:08,961 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:08,962 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:09,323 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:52:09,326 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:39,327 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:52:39,328 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:52:39,328 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:39,328 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:52:39,668 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:52:39,671 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:09,672 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:53:09,673 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:53:09,674 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:09,674 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:10,010 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:53:10,012 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:40,014 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:53:40,015 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:53:40,016 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:40,017 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:53:40,354 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:53:40,355 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:10,357 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:54:10,357 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:54:10,358 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:10,359 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:10,732 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:54:10,735 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:40,737 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:54:40,737 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:54:40,739 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:40,739 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:54:41,091 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:54:41,094 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:11,096 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:55:11,097 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:55:11,098 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:11,098 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:11,431 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:55:11,432 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:41,433 [salt.master      :450 ][DEBUG   ][171559] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:55:41,434 [salt.utils.gitfs :2885][DEBUG   ][171559] Re-using gitfs object for process 171559
2023-07-07 15:55:41,435 [salt.utils.gitfs :919 ][DEBUG   ][171559] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:41,435 [salt.utils.gitfs :817 ][DEBUG   ][171559] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:55:41,794 [salt.utils.gitfs :1917][DEBUG   ][171559] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:55:41,796 [salt.utils.gitfs :697 ][DEBUG   ][171559] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'

>>>>>>>> 30-second refreshes stop here for 1.5 minutes, then the PID changes from 171559 to 171362

2023-07-07 15:57:18,335 [salt.utils.gitfs :2775][DEBUG   ][171362] pygit2 gitfs_provider enabled
2023-07-07 15:57:18,335 [salt.utils.gitfs :2879][DEBUG   ][171362] Created gitfs object with uninitialized remotes
2023-07-07 15:57:18,335 [salt.loader.lazy :963 ][DEBUG   ][171362] The functions from module 'gitfs' are being loaded by dir() on the loaded module
2023-07-07 15:57:18,335 [salt.utils.lazy  :99  ][DEBUG   ][171362] LazyLoaded gitfs.envs
2023-07-07 15:57:18,343 [salt.utils.gitfs :2885][DEBUG   ][171362] Re-using gitfs object for process 171362

>>>>>>>>>> Now it will refresh the bootstrap, and expected behavior resumes
>>>>>>>>>> With an interesting PID change again, from 171362 to 176464
>>>>>>>>>> You can see this happen one more time during this log dump, just watch the PID changes

2023-07-07 15:57:18,378 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/saltstack/salt-bootstrap.git', None)]
2023-07-07 15:57:18,379 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:57:18,382 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:57:18,382 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:57:18,387 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 15:57:18,388 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 15:57:18,389 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:57:18,393 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:57:18,732 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:57:18,735 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:57:18,895 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/saltstack/salt-bootstrap.git' is up-to-date
2023-07-07 15:57:18,900 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 15:57:48,737 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:57:48,737 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:57:48,738 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:57:48,738 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:57:49,099 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:57:49,101 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:19,103 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:58:19,103 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:58:19,104 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:19,105 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:19,451 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:58:19,453 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:49,455 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:58:49,456 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:58:49,458 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:49,458 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:58:49,802 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:58:49,805 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:19,807 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:59:19,807 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:59:19,808 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:19,809 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:20,160 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:59:20,162 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:50,164 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 15:59:50,165 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 15:59:50,166 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:50,167 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 15:59:50,502 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 15:59:50,503 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:20,505 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:00:20,505 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 16:00:20,506 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:20,506 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:20,859 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:00:20,861 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:50,862 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:00:50,863 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 16:00:50,864 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:50,865 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:00:51,212 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:00:51,214 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:21,216 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:01:21,217 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 16:01:21,218 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:21,218 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:21,593 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:01:21,594 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:51,595 [salt.master      :450 ][DEBUG   ][176464] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:01:51,595 [salt.utils.gitfs :2885][DEBUG   ][176464] Re-using gitfs object for process 176464
2023-07-07 16:01:51,596 [salt.utils.gitfs :919 ][DEBUG   ][176464] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:51,596 [salt.utils.gitfs :817 ][DEBUG   ][176464] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:01:51,935 [salt.utils.gitfs :1917][DEBUG   ][176464] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:01:51,938 [salt.utils.gitfs :697 ][DEBUG   ][176464] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:28,736 [salt.utils.gitfs :2775][DEBUG   ][171362] pygit2 gitfs_provider enabled
2023-07-07 16:03:28,736 [salt.utils.gitfs :2879][DEBUG   ][171362] Created gitfs object with uninitialized remotes
2023-07-07 16:03:28,736 [salt.loader.lazy :963 ][DEBUG   ][171362] The functions from module 'gitfs' are being loaded by dir() on the loaded module
2023-07-07 16:03:28,737 [salt.utils.lazy  :99  ][DEBUG   ][171362] LazyLoaded gitfs.envs
2023-07-07 16:03:28,748 [salt.utils.gitfs :2885][DEBUG   ][171362] Re-using gitfs object for process 171362
2023-07-07 16:03:28,780 [salt.master      :450 ][DEBUG   ][178713] Updating gitfs fileserver cache for the following targets: [('https://github.com/saltstack/salt-bootstrap.git', None)]
2023-07-07 16:03:28,783 [salt.master      :450 ][DEBUG   ][178713] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:03:28,784 [salt.utils.gitfs :2885][DEBUG   ][178713] Re-using gitfs object for process 178713
2023-07-07 16:03:28,784 [salt.utils.gitfs :2885][DEBUG   ][178713] Re-using gitfs object for process 178713
2023-07-07 16:03:28,787 [salt.utils.gitfs :919 ][DEBUG   ][178713] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:28,788 [salt.utils.gitfs :817 ][DEBUG   ][178713] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:28,788 [salt.utils.gitfs :919 ][DEBUG   ][178713] Set update lock for gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 16:03:28,789 [salt.utils.gitfs :817 ][DEBUG   ][178713] Fetching gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 16:03:29,137 [salt.utils.gitfs :1917][DEBUG   ][178713] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:03:29,140 [salt.utils.gitfs :697 ][DEBUG   ][178713] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:29,272 [salt.utils.gitfs :1917][DEBUG   ][178713] gitfs remote 'https://github.com/saltstack/salt-bootstrap.git' is up-to-date
2023-07-07 16:03:29,276 [salt.utils.gitfs :697 ][DEBUG   ][178713] Removed update lock for gitfs remote 'https://github.com/saltstack/salt-bootstrap.git'
2023-07-07 16:03:59,142 [salt.master      :450 ][DEBUG   ][178713] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:03:59,143 [salt.utils.gitfs :2885][DEBUG   ][178713] Re-using gitfs object for process 178713
2023-07-07 16:03:59,145 [salt.utils.gitfs :919 ][DEBUG   ][178713] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:59,146 [salt.utils.gitfs :817 ][DEBUG   ][178713] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:03:59,504 [salt.utils.gitfs :1917][DEBUG   ][178713] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:03:59,506 [salt.utils.gitfs :697 ][DEBUG   ][178713] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:04:29,508 [salt.master      :450 ][DEBUG   ][178713] Updating gitfs fileserver cache for the following targets: [('https://github.com/DataDog/datadog-formula.git', None)]
2023-07-07 16:04:29,509 [salt.utils.gitfs :2885][DEBUG   ][178713] Re-using gitfs object for process 178713
2023-07-07 16:04:29,510 [salt.utils.gitfs :919 ][DEBUG   ][178713] Set update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:04:29,510 [salt.utils.gitfs :817 ][DEBUG   ][178713] Fetching gitfs remote 'https://github.com/DataDog/datadog-formula.git'
2023-07-07 16:04:29,874 [salt.utils.gitfs :1917][DEBUG   ][178713] gitfs remote 'https://github.com/DataDog/datadog-formula.git' is up-to-date
2023-07-07 16:04:29,877 [salt.utils.gitfs :697 ][DEBUG   ][178713] Removed update lock for gitfs remote 'https://github.com/DataDog/datadog-formula.git'

Expected behavior I would expect all GitFS remotes to sync on their specified interval without interruption, regardless of the update_interval of other remotes. I find it strange that syncing resumes after the repo with the longest update interval syncs.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3005.1 Dependency Versions: cffi: 1.15.1 cherrypy: Not Installed dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: 1.1.0 M2Crypto: 0.35.2 Mako: 1.0.6 msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.21 pycrypto: Not Installed pycryptodome: Not Installed pygit2: 1.6.1 Python: 3.6.8 (default, Jan 23 2023, 22:31:05) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 20.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: rhel 8.8 Ootpa locale: UTF-8 machine: x86_64 release: 4.18.0-477.13.1.el8_8.x86_64 system: Linux version: Red Hat Enterprise Linux 8.8 Ootpa ```
twangboy commented 1 year ago

I removed the longer running update interval and left just the DataDog GitFS remote that updates every 30 seconds. I ran the master and observed the continuous syncing for over 15 minutes. However, I did notice that the PID changed every 5 minutes.

twangboy commented 1 year ago

I tested this on a Windows master and did not observe the GitFS hang. The PID never changes.

knine commented 1 year ago

Thanks for looking into this for me and for all your help.