Open myii opened 5 years ago
Just an idea: document how to create a formula from this template one, inside this repository or/and in saltstack docs
@daks I've added that to the task list above -- thanks for that suggestion.
@myii I just wanted to say „Thank you!“ :-) This formula enabled me to implement https://github.com/saltstack-formulas/prometheus-formula really fast. I found the structure and especially your turnkey Travis/kitchen configuration very helpful.
@alxwr That's really good to hear and it makes all of this effort worthwhile. There were lots of contributions involved, including your own! Good to see that you've got semantic-release
running as well.
We've got an upcoming issue due to wanting to implement config.get
properly, instead of pillar.get
. The only solution I've been able to find so far is using defaults.merge
so I'm going to submit a PR in order to open up discussion. Since you're familiar with salt-ssh
, I'll request a review from you as well, so that you have the opportunity to argue against my proposal! Actually, the alternative is to go back to pillar.get
and that may be the final conclusion anyway...
@myii AFAIR we could work around the defaults.merge
issue, because we can detect whether salt
or salt-ssh
is used. But this is not pretty. :-)
Maybe it's time for a libworkaround.jinja
which would be living documentation of several incompatibilities and shortcomings that need to go. :-)
@alxwr So there's a reliable way to check for salt-ssh
? That's excellent news. So we can have both methods of merging the map available and then no-one loses out. Do you mind sharing it here? Then I can include it in the PR immediately, so that everyone can get on with reviewing it, rather than having a long discussion first. I will ensure that you are credited as the author of that specific commit, identifying whether salt
or salt-ssh
is being used.
@myii Sry for the delay.:-)
The way I found most straightforward, is inserting grains:salt_ssh: True
in the roster
file. But I get, that this is not exactly portable.
The second most reliable way is to compare paths:
salt host grains.itms
:
pythonpath:
- /usr/local/bin
- /usr/local/lib/python36.zip
- /usr/local/lib/python3.6
- /usr/local/lib/python3.6/lib-dynload
- /usr/local/lib/python3.6/site-packages
saltpath:
/usr/local/lib/python3.6/site-packages/salt
salt-ssh host2 grains.itms
:
pythonpath:
- /var/tmp/.root_99b0b6_salt/py3
- /var/tmp/.root_99b0b6_salt/pyall
- /var/tmp/.root_99b0b6_salt
- /usr/lib/python35.zip
- /usr/lib/python3.5
- /usr/lib/python3.5/plat-x86_64-linux-gnu
- /usr/lib/python3.5/lib-dynload
- /usr/local/lib/python3.5/dist-packages
- /usr/lib/python3/dist-packages
saltpath:
/var/tmp/.root_99b0b6_salt/pyall/salt
salt-ssh
is run from a temporary directory on the minion. This shows in the grain saltpath
. (We could also correlate pythonpath[0]
with saltpath
for higher certainty, but I think focusing on saltpath
is sufficient.)
@alxwr How about the method in #102? How does that work for you? I got a salt-ssh
minion prepared and tested the map both ways. The __cli
option appears to be reliable. I did notice the path differences you mentioned above but this appears to be even simpler.
@myii there is another way to detect salt-ssh
, but this involves the py renderer.
__opts__
differs when using salt-ssh
:
def config_dir():
if '__master_opts__' in __opts__:
# run started via salt-ssh
return __opts__['__master_opts__']['config_dir']
else:
# run started via salt
return __opts__['config_dir']
Taken from: https://github.com/saltstack-formulas/openssh-formula/blob/master/_pillar/known_hosts_salt_ssh.sls
@alxwr Do you mind if we discuss this in #102? It's a bit off-topic for this issue and then it opens up the discussion with the others, so that we can reach a resolution. It's my fault for bringing it up here first!
@myii I was just going to suggest that. :-)
Does the merge of saltstack/salt#52156 affect map.jinja
and similar in any way?
Might be nice as a replacement to tplroot
and/or things like include: - template.X
.
@5paceToast Appreciate you bringing that merge to our attention. A cursory look suggests that we will be able to improve our use of include
at the very least. The problem is that this will take a while to reach us; we generally have to support the three versions of Salt at any given time. So right now, that means back to 2017.7
. It all depends on how far back that merge is backported.
In terms of tplroot
, that's in the queue at https://github.com/saltstack/salt/pull/51814.
@myii in that case, it might be nice to also have a template-formula variant that applies only to the latest release, for people interested in that (i.e for development, learning and other purposes). Would that be politically feasible at all?
@5paceToast There have been conversations about having various types of template-formula
. There are no right or wrong answers as such but a lot comes down to keeping everything maintained and synchronised. The general direction is to focus on this template-formula
for the time being, probably until most of the tasks laid out above are completed. Not to mention the amount of time and effort it requires to propagate and review the changes when applying to the rest of the formulas, which has already been taking place thanks to numerous contributors.
In terms of a template-formula
for the latest release only, I would surmise that would be fairly low priority. Not because it isn't a good idea; rather, it would be an extremely useful reference as we bring each formula up-to-date over time. However, the issue remains that we have to support all of the current Salt versions across all 300+ formulas in this org. I'm not sure if there is a formula out there that would benefit from a template-formula
for the latest release only.
If someone had the desire to go ahead with this idea, I'm sure there are contributors here who would be happy to offer advice along the way, either here in GitHub or in our Slack/IRC/Matrix room. An actual functioning repo in place becomes an excellent stimulus for further discussions about the merits and pitfalls of such an approach and whether it would be something worth supporting at an org level.
Going back to https://github.com/saltstack/salt/pull/52156 for a moment: this is only available in develop
(so far). If it does reach 2019.2
, what will the latest release template-formula
do? Meaning, the earlier versions of the 2019.2
series (e.g. 2019.2.0
, 2019.2.1
) won't contain that functionality, so it would be no better than where we are right now.
But from another angle, how about the idea of a template-formula
that's actually based on the develop
branch, rather than the latest release? Now that would definitely generate some interest. In fact, it could be worth having a develop
branch in this repo, to track exactly that. I'll add that to the list above -- thanks for the idea!
@5paceToast Thanks to @javierbertoli, we've now got testing images for develop
. I've already run initial tests with tojson
, which shows why we can't use it yet due to having to support 2017.7
. So this is an excellent development.
The funny thing is when I looked at the upstream merge again in more detail, I realised that there isn't any way it can be used within this formula, since it is about pillar includes. In any case, at least the discussion led to some useful outcomes.
The funny thing is when I looked at the upstream merge again in more detail, I realized that there isn't any way it can be used within this formula, since it is about pillar includes. In any case, at least the discussion led to some useful outcomes.
I noticed that too, but shortly after also noticed the include/exclude docs which claim that relative includes were added way back in 0.16.0 while parent-based (grandparent etc) includes in 2015.8. (something that might be usable in template.config.file
, for example!)
It seems like some new features sometimes get missed, and some don't even make it to the docs (salt.fileserver.s3fs
barely even makes sense without salt.pillar.s3
as context, for example).
Which is why I think a sort of "latest" version of template-formula really makes sense (a develop
branch does indeed sound great to me)!
This issue is a collection of ideas initially taken from the log of discussions[1][2][3][4] captured in the wiki.
Visibility
template-formula
a sticky formula for the whole SaltStack Formulas project.Release tagging (for each merged PR)
bump:...
labels to use when evaluating the extent of change each PR will have on the formula. Helps when bumping version numbers for releases.bump:major
bump:minor
bump:patch
CHANGELOG
to use the format proposed by https://keepachangelog.com/en/1.0.0/:django-formula
repo.AUTHORS
file, with the primary purpose of identifying the key maintainers of the repo.salt-formula
repo.semantic-release
for this formula, including updating the changelog and updating theFORMULA
file -- #35.commitlint
to warn when commit messages aren't appropriate for bumping the version number -- #41.semantic-release
(keeping this list in chronological order for the most part):vault-formula
collectd-formula
nginx-formula
cert-formula
chrony-formula
rkhunter-formula
postgres-formula
-- specific commitkeepalived-formula
systemd-formula
ufw-formula
salt-formula
fail2ban-formula
bind-formula
syslog-ng-formula
apt-formula
locale-formula
sudoers-formula
postfix-formula
iptables-formula
golang-formula
-- with follow-uplogrotate-formula
php-formula
-- with follow-updeepsea-formula
mysql-formula
libvirt-formula
openvpn-formula
sysstat-formula
dhcpd-formula
packages-formula
prometheus-formula
grafana-formula
pre-commit_semantic-release.sh
could check for the existence of thedocs
dir.semantic-release
(and/or the TOFS pattern?) can be used to mitigate the use offormula:ng
states:php-formula
:YAML & Jinja
map.jinja
style -- #20.map.jinja
style due todefaults.merge
vs.salt-ssh
-- Upstream bug report: https://github.com/saltstack/salt/issues/51605.map.jinja
style for the time being -- #25.map.jinja
accordingly.oscodename
-- e.g.postgres-formula
.osfinger
-- e.g.apache-formula
-- #25.macros.jinja
template with common macros that "Any formulae would find useful":extra_yamls
approach outlined by @javierbertoli is another approach to solve scalability issues (awaiting sample PR).jinja_lstrip_blocks
and more specficallyjinja_trim_blocks
until the upstream issue is resolved.~
for string concatenation instead of+
.| json
instead of| tojson
, at least until2017.7
support is required -- refer back to this comment.2018.3.3
: https://github.com/saltstack/salt/commit/1499c6abcf8076a2fab43c608cc46ed93c9e6635| json
and| tojson
, so| yaml
may be preferable (at least in certain situations), see:{%- ... %}
as a standard due to linked issues raised in this comment.osmajorrelease
andosfinger
on Arch Linux.map.jinja
:{%- do template.update({'osmajorrelease': salt['grains.get']('osmajorrelease', 0)|int }) %}
comp_lzo:
-- insist on explicit values at all times by modifying ouryamllint
configurations based on https://yamllint.readthedocs.io/en/stable/rules.html#module-yamllint.rules.empty_values?yaml_dump.sls
(via.ssf-formula
) in all formulas so that changes to the map before and after can easily be picked up in the CI logs.TOFS pattern vs. pillar
files_switch
) -- #19.source_files
instead offiles
in the pillar (otherwise there are twofiles
, which will cause confusion) -- #76.libtofs.jinja
instead ofmacros.jinja
, to work towards the simplest drop-in solution for other formulas -- #79.Other pillar-avoidance methods to consider
pillar.get
toconfig.get
globally (as in, for all formulas eventually).Make the formula portable
tpldir
not working in template files, see https://github.com/saltstack/salt/issues/41195.tplroot
throughout the formula and then changing the root directory to ensure the whole formula is using the new namespace.Documentation
docs
sub-directory..keep
to push directory structure to Git..gitignore
to preventassets
being committed (also need the proper solution for these).template-formula
wiki the central location for planning/organisation/reference.tmux-formula
postgres-formula
formula:ng
toformula
-- see https://github.com/saltstack-formulas/nginx-formula/pull/236.Tests and CI
kitchen-ci
andinspec
(ortestinfra
) examples for the basic tasks (add/remove package, check service, etc) -- want to replace oldserverspec
tests -- #34.inspec
.Goss
andsaltcheck
also mentioned -- perhaps alongside for the latter?saltscaffold
.nginx-formula
in this pull.~opensuse
not working -- #45.opensuse
usually works the second time its run), evaluate ways of minimising these delays:travis_retry
.matrix: allow_failures: - os: opensuse-leap
.clean
stages -- @javierbertoli looking into this.Gemfile
, to avoid test failures -- #81.provision_command
for each platform, as used in thesystemd-formula
-- only if actually useful, not just for the sake of it..kitchen.yml
=>kitchen.yml
-- done in #40.pillars-from-files
=>pillars_from_files
-- done in #74.States
template-formula
to not be too focussed onpkg
:pkg/install.sls
,pkg/clean.sls
,repo/install.sls
,repo/clean.sls
, etc.Repo standards
develop
branch to allow PRs with the very latest changes, that can be merged back intomaster
at the appropriate times.Other resources to evaluate for inclusion