search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] gitfs fileserver does not sync to all_saltenvs for remote #62077

Open dapak opened 2 years ago

dapak commented 2 years ago

Description When utilizing the all_saltenvs option for a gitfs remote resource, you would expect that the resource would be synced to all environments identified by gitfs specified by file_roots in the master configuration. The resource however is only synced to the base environment.

Setup The following is the master configuration:

git_pillar_provider: pygit2

fileserver_backend:
  - roots
  - git

gitfs_remotes:
  - https://github.com/saltstack-formulas/vim-formula:
    - all_saltenvs: v0.15.5
    - root: vim
    - mountpoint: salt://vim

file_roots:
  base:
    - /srv/salt/states
  dev:
    - /srv/salt/states

pillar_roots:
  base:
    - /srv/salt/pillar/base
  dev:
    - /srv/salt/pillar/dev
    - /srv/salt/pillar/base

pillar_source_merging_strategy: recurse
pillarenv_from_saltenv: True
top_file_merging_strategy: same

The following is the additional minion configuration:

pillarenv: dev
saltenv: dev

The following is the directory structure of /srv/salt:

[root@lima-salt-master salt]# pwd
/srv/salt
[root@lima-salt-master salt]# tree
.
├── formulas
├── pillar
│   ├── base
│   │   ├── hello
│   │   │   └── world.sls
│   │   ├── nginx
│   │   │   └── init.sls
│   │   └── top.sls -> /srv/salt/pillar/top.sls
│   ├── dev
│   │   └── top.sls -> /srv/salt/pillar/top.sls
│   └── top.sls
└── states
    └── top.sls

7 directories, 6 files

The contents of the states/top.sls:

{{ saltenv }}:
  '*':
    - vim

Please be as specific as possible and give set-up details.

Steps to Reproduce the behavior The above configuration syncs the vim formula resource to the base environment correctly:

[root@lima-salt-master salt]# salt-run fileserver.file_list
- top.sls
- vim/_mapdata/_mapdata.jinja
- vim/_mapdata/init.sls
- vim/absent.sls
- vim/editor.sls
- vim/files/nerdtree/doc/NERD_tree.txt
- vim/files/nerdtree/nerdtree_plugin/exec_menuitem.vim
- vim/files/nerdtree/nerdtree_plugin/fs_menu.vim
- vim/files/nerdtree/plugin/NERD_tree.vim
- vim/files/nerdtree/syntax/nerdtree.vim
- vim/files/pyflakes/ftplugin/python/pyflakes.vim
- vim/files/pyflakes/ftplugin/python/pyflakes/LICENSE
- vim/files/pyflakes/ftplugin/python/pyflakes/NEWS.txt
- vim/files/pyflakes/ftplugin/python/pyflakes/README.rst
- vim/files/pyflakes/ftplugin/python/pyflakes/bin/pyflakes
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/__init__.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/checker.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/messages.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/scripts/__init__.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/scripts/pyflakes.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/__init__.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/harness.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/test_imports.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/test_other.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/test_script.py
- vim/files/pyflakes/ftplugin/python/pyflakes/pyflakes/test/test_undefined_names.py
- vim/files/pyflakes/ftplugin/python/pyflakes/setup.py
- vim/files/salt/ftdetect/jinja.vim
- vim/files/salt/ftdetect/sls.vim
- vim/files/salt/ftplugin/rst.vim
- vim/files/salt/ftplugin/sls.vim
- vim/files/salt/syntax/jinja.vim
- vim/files/salt/syntax/sls.vim
- vim/files/vimrc
- vim/init.sls
- vim/map.jinja
- vim/nerdtree.sls
- vim/pyflakes.sls
- vim/salt.sls

It does not sync the resource to the dev environment:

[root@lima-salt-master salt]# salt-run fileserver.update
True
[root@lima-salt-master salt]# salt-run fileserver.file_list saltenv=dev
- top.sls

Additional list of the environments on the fileserver:

[root@lima-salt-master salt]# salt-run fileserver.envs
- base
- dev
- v0.14.2
- v0.15.0
- v0.15.1
- v0.15.2
- v0.15.3
- v0.15.4
- v0.15.5

Expected behavior I would expect that the vim formula also be present within the dev environment.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.11.5 cherrypy: unknown dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: 0.26.8 M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.14 pycrypto: Not Installed pycryptodome: Not Installed pygit2: 0.26.4 Python: 3.6.8 (default, Nov 10 2021, 06:50:23) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 19.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: oracle 8.5 locale: UTF-8 machine: x86_64 release: 5.4.17-2136.302.7.2.2.el8uek.x86_64 system: Linux version: Oracle Linux Server 8.5 ```
welcome[bot] commented 2 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

OrangeDog commented 2 years ago

This was discussed on Slack, and this is not what the problem was. Indeed, this is the config that you said worked correctly,

What wasn't working was using __env__ in the file_roots.

dapak commented 2 years ago

@OrangeDog

This config does not work correctly per the output of fileserver.file_list for the dev environment above. It should have the vim formula synced to it, assuming all_saltenvs is designed to sync a resource to all salt environments. Like many things, the documentation isn't 100% clear on that so I could be quite wrong.

The __env__ issue I think might be related to this problem. I can file a separate bug for that if you prefer, but I'm still having the problem with specifying even dev in and not even using a configuration with __env__.

OrangeDog commented 2 years ago

Oh, I thought it was solved in the end. Still no idea why this doesn't work for you then.

dapak commented 2 years ago

@OrangeDog

Unfortunately nope, I'm still scratching my head on this one. I feel like I've tried every combination of configuration at this point to get this to work and I've given up. I've been doing configuration management work with other tooling for over a decade and Salt's breaking my brain on this particular feature.

Lucianovici commented 1 year ago

Hi everybody, I am also struggling with this. It is very strange. I have a simple master config combining roots and git fileserver backends. I use gitfs for a bunch of formulas. But when I run salt to apply states with a different saltenv (other than base) it is going to crash and burn - all my states from formulas are "not available" anymore.

    Specified SLS postgres in saltenv staging is not available on the salt master or through a configured fileserver

Master config snippet:

fileserver_backend:
  - roots
  - git

file_roots:
  staging:
    - /staging/path/to/my/salt/states
  prod:
    - /prod/path/to/my/salt/states

gitfs_provider: pygit2
gitfs_remotes:
  - https://github.com/saltstack-formulas/postgres-formula:
      - all_saltenvs: master

pillar_roots:
  staging:
    -  /staging/path/to/my/salt/pillar
  prod:
    -  /prod/path/to/my/salt/pillar
leifb commented 1 year ago

I'm still having the same problem with salt-master 3006.1. The all_saltenvs option is not working at all.

After a lot of trail and error I was able to figure out two things:

  1. Some configuration changes seem to not have any effect unless you manually delete the gitfs cache.
    On my install, the cache is located here: /var/cache/salt/master/gitfs.
    I did not test which of the options were not applied, I just remember that salt-run fileserver.file_list saltenv=development suddenly had an output after deleting the cache manually.
  2. I was able to find a workaround by defining the ref for every saltenv I use.

Here is the full config I use that works for me:

fileserver_backend:
  - roots
  - gitfs

gitfs_provider: pygit2  
gitfs_disable_saltenv_mapping: True
top_file_merging_strategy: same

pillar_roots:
  __env__:
    - /srv/salt/pillar

file_roots:
  __env__:
    - /srv/salt/states

gitfs_remotes:
  - https://github.com/saltstack-formulas/letsencrypt-formula.git:
    - saltenv:
      - development:
        - ref: v3.2.0
      - production:
        - ref: v3.2.0
      - test:
        - ref: v3.2.0