search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.15k stars 5.48k forks source link

[BUG] git_pillar pulling from git at abnormally high interval. #58786

Closed whytewolf closed 4 years ago

whytewolf commented 4 years ago

Description after upgrading to 3002 I'm seeing that git_pillar is hitting my git backend for pillar at an abnormal speed.

Setup install salt-master and setup git_pillar

Steps to Reproduce the behavior have git_pillar set up on a master and set the logs to trace to watch the fun. 

[root@salt00 salt]# salt-run config.get loop_interval
60

without trace-level logs, I see this. but if the lock is turned off you won't see this error.

2020-10-22 08:49:51,034 [salt.loaded_139990193331056.int.returner.appoptics_return:100 ][ERROR   ][3297] The appoptics_return module couldn't load the appoptics_metrics module.
2020-10-22 08:49:51,034 [salt.loaded_139990193331056.int.returner.appoptics_return:102 ][ERROR   ][3297] please make sure it is installed and is in the PYTHON_PATH.
2020-10-22 08:49:53,301 [salt.utils.gitfs :878 ][WARNING ][3297] git_pillar_global_lock is enabled and update lockfile /var/cache/salt/master/git_pillar/bd8b01133547c4630de11f5e478ceb20a846109c8bb7c49d27813bc61ac6a163/.git/update.lk is present for git_pillar remote 'master git@gitserver.private'. Process 6506 obtained the lock but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.
2020-10-22 08:49:53,301 [salt.utils.gitfs :833 ][WARNING ][3297] Update lock file is present for git_pillar remote 'master git@gitserver.private', skipping. If this warning persists, it is possible that the update process was interrupted, but the lock could also have been manually set. Removing /var/cache/salt/master/git_pillar/bd8b01133547c4630de11f5e478ceb20a846109c8bb7c49d27813bc61ac6a163/.git/update.lk or running 'salt-run cache.clear_git_lock git_pillar type=update' will allow updates to continue for this remote.
2020-10-22 08:50:54,750 [salt.loaded_139990146493632.int.returner.appoptics_return:100 ][ERROR   ][3297] The appoptics_return module couldn't load the appoptics_metrics module.
2020-10-22 08:50:54,750 [salt.loaded_139990146493632.int.returner.appoptics_return:102 ][ERROR   ][3297] please make sure it is installed and is in the PYTHON_PATH.
2020-10-22 08:50:56,978 [salt.utils.gitfs :878 ][WARNING ][3297] git_pillar_global_lock is enabled and update lockfile /var/cache/salt/master/git_pillar/bd8b01133547c4630de11f5e478ceb20a846109c8bb7c49d27813bc61ac6a163/.git/update.lk is present for git_pillar remote 'master git@gitserver.private'. Process 6506 obtained the lock but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.
2020-10-22 08:50:56,979 [salt.utils.gitfs :833 ][WARNING ][3297] Update lock file is present for git_pillar remote 'master git@gitserver.private', skipping. If this warning persists, it is possible that the update process was interrupted, but the lock could also have been manually set. Removing /var/cache/salt/master/git_pillar/bd8b01133547c4630de11f5e478ceb20a846109c8bb7c49d27813bc61ac6a163/.git/update.lk or running 'salt-run cache.clear_git_lock git_pillar type=update' will allow updates to continue for this remote.
2020-10-22 08:51:58,399 [salt.loaded_139990142644688.int.returner.appoptics_return:100 ][ERROR   ][3297] The appoptics_return module couldn't load the appoptics_metrics module.
2020-10-22 08:51:58,399 [salt.loaded_139990142644688.int.returner.appoptics_return:102 ][ERROR   ][3297] please make sure it is installed and is in the PYTHON_PATH.
2020-10-22 08:52:00,765 [salt.utils.gitfs :878 ][WARNING ][3297] git_pillar_global_lock is enabled and update lockfile /var/cache/salt/master/git_pillar/bd8b01133547c4630de11f5e478ceb20a846109c8bb7c49d27813bc61ac6a163/.git/update.lk is present for git_pillar remote 'master git@gitserver.private'. Process 6506 obtained the lock but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.

Expected behavior git_pillar pull from git about once at loop_interval. instead of every second.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3002 Dependency Versions: cffi: 1.14.0 cherrypy: unknown dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.11.1 libgit2: 0.26.6 M2Crypto: 0.35.2 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 1.0.0 mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 0.26.4 Python: 3.6.8 (default, Apr 2 2020, 13:34:55) python-gnupg: 2.3.1 PyYAML: 3.13 PyZMQ: 17.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: centos 7 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1127.13.1.el7.x86_64 system: Linux version: CentOS Linux 7 Core ```

Additional context Add any other context about the problem here.

whytewolf commented 4 years ago

nevermind. it has been a difficult morning and reading a clock just isn't apperently a thing for me this morning.