search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

Dynamic Git pillar race condition in 2016.11.2 #39259

Closed peter-slovak closed 7 years ago

peter-slovak commented 7 years ago

Description of Issue/Question

We are using Git external pillar on our master and this is the only source of pillar data configured. Instead of enumerating each branch and its environment respectfully, we are leveraging the __env__ branch name that maps to the same branch as the requesting minion's environment (or pillarenv) setting:

ext_pillar:
  - git:
    - __env__ ssh://git@git.domain.tld/project/salt.git:
      - root: pillar

Note that this feature has been dropped somewhere along the 2016 release and then brought back and reimplemented. Anyway, when the GitProvider initializes the remotes and checks out branches, it should create a separate directory for each remote branch in cachedir (e.g. /var/cache/salt/master/git_pillar). Instead, it reads __env__ as if it were one branch and only creates one directory that is also the only one in remote_map.txt:

# git_pillar_remote map as of 08 Feb 2017 22:29:41.407132
b4092923fbdfd694387b13fcfbb90712d13692542d47e70913049848f672b8a6 = __env__ ssh://git@git.domain.tld/project/salt.git

Actually, if you have a reasonable amount of minions spread into various environments, you can watch the contents of the single pillar directory changing in real-time:

(production) root@saltMaster:/var/cache/salt/master/git_pillar/b4092923fbdfd694387b13fcfbb90712d13692542d47e70913049848f672b8a6$ ls -l pillar/
total 72
drwxr-xr-x  7 root root 4096 Feb  8 22:30 backup
drwxr-xr-x  2 root root 4096 Feb  8 22:31 common
drwxr-xr-x 10 root root 4096 Feb  8 22:30 database
drwxr-xr-x 40 root root 4096 Feb  8 22:31 internal
drwxr-xr-x  6 root root 4096 Feb  8 22:30 loadbalancer
drwxr-xr-x  6 root root 4096 Feb  8 22:31 logserver
drwxr-xr-x  9 root root 4096 Feb  8 22:31 mail
drwxr-xr-x  4 root root 4096 Feb  8 22:30 memcache
drwxr-xr-x  2 root root 4096 Feb  8 22:31 network
drwxr-xr-x  9 root root 4096 Feb  8 22:30 openstack
-rw-r--r--  1 root root 2694 Feb  8 22:31 roles.yaml
drwxr-xr-x  4 root root 4096 Feb  8 22:30 saltmaster
drwxr-xr-x  4 root root 4096 Feb  8 22:30 shellserver
drwxr-xr-x  4 root root 4096 Feb  8 22:31 storage
-rw-r--r--  1 root root  388 Feb  8 22:31 top.sls
drwxr-xr-x  6 root root 4096 Feb  8 22:31 webserver
drwxr-xr-x  7 root root 4096 Feb  8 22:30 workshop
(production) root@saltMaster:/var/cache/salt/master/git_pillar/b4092923fbdfd694387b13fcfbb90712d13692542d47e70913049848f672b8a6$ ls -l pillar/
total 4
-rw-r--r-- 1 root root 90 Feb  8 22:33 top.sls

Surprisingly, it looks like this inconsistency only affects pillar.items and not pillar.item someitem. The former returns a pseudorandom version of pillar depending on what branch is currently cecked out in cachedir; the latter seems to be always right. One of our branches has just a placeholder top.sls so I was initially suspecting an incorrectly configured merging behavior, but after trying every combination (and purging caches) nothing has changed.

Other gitfs and pillar related options in master config:

top_file_merging_strategy: same
pillar_source_merging_strategy: none
fileserver_backend:
  - git
gitfs_provider: gitpython

Can anyone please explain if/why this is needed or what am I missing? Thank you for any help.

Versions Report

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-24-generic
         system: Linux
        version: Ubuntu 16.04 xenial
peter-slovak commented 7 years ago

Turns out that pillarenv minion config option is the key here; I was used to environment which covered both gitfs and git_pillar. When I specified the correct pillarenv, everything started to work fine.