Closed driskell closed 9 years ago
OK, I just about have a PR ready to go. Here are the changes I propose making:
After reviewing this in more detail, I think @driskell had the right idea in his earlier proposal. Therefore, I've done a few things:
1) Configuration files are now rendered as ordered dictionaries. I have added a new configuration file option called ordered_top_files
as suggested by @driskell . It defaults to False, but when it is turned on, it will process environments in the order they are written in the configuration file.
2) I have added a configuration option called top_file_merging_strategy
. This can be set to either merge
or same
(currently). If set to merge, the current behaviour is preserved. However, this can be influenced by the ordered_top_files
setting above, in what should hopefully be an obvious way. If it's not, please let me know. :]
I did not implement fallback options because I feel like environments are already very difficult to reason about and I feel like adding fallback behaviour will only increase that complexity. Differing opinions are welcome of course. :]
3) I created a default_top
option. If this is set, and no environment is speficially requested (via configuration or via a saltenv
argument) and the same
value is applied to the top_file_merging_strategy
option, this is the environment that will be used.
Thoughts? I'll work on cleaning this up and submitting a PR, probably tomorrow.
@cachedout I'd be very interested in seeing your PR. Being able to disable the topfile merging would greatly simplify our environment setup and test process.
@mathom I was just waiting for any feedback. :] I am planning on getting this cleaned up and submitted today. Thanks for being willing to review it. Much appreciated.
@cachedout I will love to test this.
I assume that top_file_merging_strategy
being set to merge by default and ordered_top_files
being set to True
would pull them from the top down.
Using this method, we would need to define the most generic first and then the most custom at the bottom to get a proper merge overwrite or is this resolved otherwise with the change to using ordered dictionaries? I'm interested in seeing how much would need to be altered to use different branches in our git repositories to deploy our code base from a single top file for definition.
Thanks for all the work put in as this is a large problem that is quite complex from what I can understand.
@jmelfi I ended up running into a problem with ordered_top_files
and so I'm changing it to support an order:
flag for each env in file_roots
. In the end, I think this is better since depending on ordering in a YAML dictionary is just asking for trouble. To answer your question though, yes, with the order flag set in your envs, the behaviour you describe is what should happen.
OK @jmelfi and @mathom. Have a look at #24878 and let me know what you think.
@cachedout I'll give it a review over the next hour or so. Thanks for the update!
@jmelfi Super. Be sure to configure things as described in the PR. It won't (or shouldn't) change behaviour out of the box. I'll be here if you have questions. Thanks for the testing help!
wow, this was a long issue to read :)) Would love to see this merged soon, thx for the effort!
I have merged this fix into develop and it should be backported into the 2015.8 branch in the next day or two.
:+1:
Back-port is in #26214. :)
@cachedout sorry for a late reply here. I was able to get the sorting to work properly after using the PR work. Thanks you guys for this hard work. I've been working the last week on moving from our puppet code completely into saltstack.
You guys are awesome!
@jmelfi I'm so glad! Thanks for testing this. For the time being, I am going to close this issue since we're using it to track progress toward our release. That doesn't mean, however, that people can't continue to post feedback here. We'll re-open it if we need to. Thanks again!
Hi
(This is taken from my comments on closed issue #5440 - @basepi recommended a new issue so here it is! Thanks.)
I have 4 environments, "base", "qa", "dev", "master", in that order defined in file_roots. These are branches in a git and I have my own deployment of this to the salt server (I do not use gitfs).
Each has a version of the top.sls file containing all environments (as you'd expect since it same file, just different revisions different branches)
The documentation describes that if an environment has info in top.sls in "base", it will always override all other top.sls. This is not the case - and checking the code there is no distinction between "base" and any other name, and no specific decision on ordering (and it seems alphabetical) - so in my case, even though in the top.sls and file_roots, "qa" is second down, because it is last alphabetically, that overrides all other top.sls files in all other environments.
Based on the docs, it seems to give the idea that "base" is a special name and is always authoritative. It also explicitly says that if an environment is defined in multiple top.sls files - the definition in the top.sls for that environment will be preferred. But the code seems to say otherwise.
The code seems to simply merge and overwrite, in alphabetical order. Which contradicts entirely the documentation.
For reference, from http://docs.saltstack.com/en/latest/ref/states/top.html:
The following is what lead me to think "base" was a special name (it doesn't say the FIRST environment wins, it says the "base" environment wins.)
The following is the bit saying a top.sls in an environment is authoritative for that environment if it contains config for it:
Although if I'm honest, point 3 does seem to contradict point 2 (below) - I would've expected point 2 to at least make an exception for point 3 - or point 3 come first
@basepi:
I agree the documentation is (mostly) a good way, and the behaviour should match. Assuming I haven't read the doc wrong though, and "base" is meant to be special and authoritative and overriding, then I think that should be configurable so I can call it something else, so maybe a "main_env: base" option with that as default.
Thanks guys!