Follow Salt Formula conventions?

[renoirb]

Hi,

I´m unsure if its a deliberate decision but this formula seem to not follow SaltStack Formulas conventions.

(I wasn’t aware that you documented the reasons and I’ll take a better look at it ASAP)

The nice thing about such convention is that we can make a simple list of formulas without worrying of where the formula’s init.sls is.

For example, I’m using a pillar listing all the formulas I want to use. A state either ensures they are cloned and added as file_roots: OR as gitfs_remotes: entry.

For example, In a development VM;

file_roots:
  base:
    - /srv/salt
    - /srv/local/states
    - /srv/formulas/sysctl
    - /srv/formulas/foo
    - /srv/formulas/bar

And then in another salt master that uses gitfs;

gitfs_remotes:
  - /srv/formulas/sysctl
  - /srv/formulas/foo
  - /srv/formulas/bar

[renoirb]

After reading the reasons. I see why you call the main state folder states instead of using the formula name. My concern becomes now how we can systematically configure a formula in that format AND support SaltStack official format too.

[arnisoph]

@renoirb This is a complex topic and I'll try to give feedback as soon as I find some free time.

In short: I'm using an approach that merges the formula git repositories all together to lower number of repos because with the increasing amount of attached git repos, Gitfs feels slow. The are pros and cons following my approach or the official ones. I'm going to elaborate more details to explain why I think the official approach isn't useful for me soon.

[renoirb]

Hi @bechtoldt I just want to understand how you do it, I’d love to help you make it systematic.

I’ve been working on a vagrant workbench to work on salt states locally and I agree with you with relying on GitFS can sometimes be annoying if the remote doesn’t exist. Compensating to this requires to create a mirror of a formula which also adds overhead. Hence this discussion :)

If I may, take a look at webplatform/ops.

Idea of that repo is to encapsulate salt-state maintenance by allowing it to work on a Vagrant VM.

1. salt-master/ has scripts taking care of pulling git repos of config and ensuring file ownership
2. vagrant-workbench/ allows to have a local VM to work salt states in
3. vagrant-minions/ to boot minions that’ll report to the local salt master

As for the formula automation macros, I’ve made them in what I called a "basesystem" that any VM would use.

The way I configured it is described in this issue. Look at the pillar example which illustrate how you can list your formulas.

I’ll push later today a full implementation in new_master.sls which basically loops using basesystem/macros/git.sls (I’m still working on it A.T.M.)

[arnisoph]

Cool thing! I added some magic to my custom Vagrant provision scripts that take care of linking the states directories into the local Vagrant file system:

https://github.com/bechtoldt/vagrant-assets/blob/master/scripts/provision/rhel/custom/saltstack_formulas/A_base.sh

This works for the old styled formulas but also for newer ones and prevents duplicating code. Maybe there's a chance to merge our efforts together in a sane way.

[renoirb]

I’d love to participate for sure!

I see what you are doing in the shell script to initialize a machine that you run once. I did the same and I recently changed it to use pillars.

Problem is that I don’t want to have to change the script to add things, nor have too many copies of code around. Scripts are great, but there must be a way to improve by leveraging Salt there too.

Problem with GitFS is that it tend to run at moments we dont want. Your proposal is to have our own mirror and sync ourselves.

My thought was; why not treating every git repo the same way. We already manage things in DVCS; web app code, configuration manifest, formulas, (???), they’re all all git repos after all.

We can leverage salt and pillars and turn all the repos we need into a number of list of repos by purpose. Each of them would likely have more than one origin (i.e. upstream, your fork, on Gerrit, etc) we can automate the clone, fetch, reset, set remotes in state files.

I’m on freenode IRC with same nick if you want to talk about it.

[renoirb]

(after some time reading the code you gave, and some time to think about it)

So how do you create the file_roots and pillar_roots exactly? I couldn’t find where you set them.

Problem with the fact that each formula has states/ then the init.sls in it, how do you refer to it by making the roots?

In GitFS, HGFS, SVNFS, etc. they do have *_mount;

# e.g. /etc/salt/master.d/gitfs.conf

gitfs_remotes:
  base:
    - /srv/salt
    - /srv/formulas/sysctl
      - root: states
    - /srv/formulas/foo
      - root: states
    - /srv/formulas/bar
      - root: states

So, how would you do it for file_roots: and how would you do it for pillars?

[arnisoph]

@renoirb I'm going to handle this topic in one of my next blog posts at https://arnoldbechtoldt.com/.

In summary: Everything vagrant+salt related can be found in https://github.com/bechtoldt/vagrant-assets/blob/master/scripts/provision/rhel/custom/saltstack_formulas/A_base.sh . (file_)roots dirs will be created, subfolders are symlinks to separate Git repos. This is done for states, pillars and contrib files (unmaintained states or templates). See also https://asciinema.org/~arnoldb.

For production environments I practise and recommend merging (more or less) all required Git repos that should be reachable by Salt Master/ Salt Syndic Master.

autosign_file: /etc/salt/autosign.conf
ext_pillar:
- git: master git@gitlab:matchbit/salt-pillar.git
failhard: true
fileserver_backend:
- roots
- git
gitfs_provider: gitpython
gitfs_remotes:
- git+ssh://git@gitlab/matchbit/salt-env.git
- git+ssh://git@gitlab/matchbit/salt-custom.git
- git+ssh://git@gitlab/matchbit/salt-modules.git
order_masters: true
pillar_opts: false
pillar_source_merging_strategy: recurse
runner_dirs:
- /home/arbe/runners
- /usr/local/share/salt-runners

• salt-env: contains the states of all formulas
• salt-custom: contains private states/ file templates/ binaries
• salt-modules: contains custom modules, see github.com/bechtoldt/salt-modules
• salt-pillar: contains all pillars

In the end it's just simple rsync'ing between formulas and salt-env:

#!/bin/bash

formulas=~/Documents/dev/github
fdir=~/Documents/dev/salt-infra/saltup-env

#if [[ -z "$test" ]]; then
#  dotest=true
#elif [[ "$test" == '0' ]]; then
#  dotext=false
#fi

for d in ${formulas[@]}/*-formula; do
  echo ============
  formula="${d##*/}"
  formula="${formula%*-formula}"
  formula_prefix=''

  if [[ "$formula" =~ saltstack-* ]]; then
    formula="${formula#saltstack-*}"
    formula_prefix='saltstack-'
  fi

  echo Formula: $formula

  if [[ -d ${formulas}/${formula_prefix}${formula}-formula/states/ ]]; then
    rsync --exclude='*.slsc' --exclude='*.swp' --delete -q -a ${formulas}/${formula_prefix}${formula}-formula/states/ ${fdir}/${formula}/
  else
    rsync --exclude='*.slsc' --exclude='*.swp' --delete -q -a ${formulas}/${formula_prefix}${formula}-formula/${formula}/ ${fdir}/${formula}/
  fi

  [[ -d ${formulas}/${formula_prefix}${formula}-formula/contrib/states/files/ ]] && rsync --exclude='*.slsc' --exclude='*.swp' -q -a ${formulas}/${formula_prefix}${formula}-formula/contrib/states/files/ ${fdir}/${formula}/files/
  [[ -d ${formulas}/${formula_prefix}${formula}-formula/contrib/files/ ]] && rsync --exclude='*.slsc' --exclude='*.swp' -q -a ${formulas}/${formula_prefix}${formula}-formula/contrib/files/ ${fdir}/${formula}/files/
done

echo -e "\n"

This is a quick&dirty script. I'll use https://github.com/bechtoldt/vcs-gather (will be refactored using python) as base to make full-features git clone/rsyncing for salt states/pillars but also for puppet, generic Git repos and much more.

Sometimes it's easier to take the KISS approach instead of the cool/mightiest/over-engineered solution.

Hope this helps.

[arnisoph]

Memo to myself: I need to file an issue on https://github.com/bechtoldt/formula-docs that will document future work on this topic.