search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.14k stars 5.47k forks source link

Merging a list in pillars #28394

Closed ldelossa closed 8 years ago

ldelossa commented 8 years ago

I'm noticing that simply merging lists within pillars is not possible. I'm trying to take two pillar files with the following lists:

packages:
  - traceroute
  - nmap
packages:
  - mtr
  - ping

I would like to apply these two pillars to a minion, and have them merge into one conhesive list.

This seems to be only possible in a hackish way using nested dictionaries. Are there any plans to do this with the simple merge data struct? 

packages:
  install:
    open-vm-tools:
    lsof:
    tcpdump:
    mtr:
    traceroute:
    telnet:
    bind-utils:
    curl:
    wget:
    ftp:
    tftp:
    samba:
    samba-client:
    ntp:
    yum-utils:
  remove:
    postfix:
    NetworkManager:

This is the structure that will allow merging. I cannot have similar keys anywhere. Also seems to break with Ints.

mlalpho commented 8 years ago

dupe of #3991 ?

ldelossa commented 8 years ago

Looks like that is the case. It's cool to close this one out then.

jfindlay commented 8 years ago

@ldelossa, @mlalpho, thanks for reporting.

uvsmtid commented 8 years ago

Strictly speaking, issue #3991 (based on examples in the initial posts here and there) is supposed to solve completely different problem of merging dictionaries.

The problem of merging dictionaries is solved as I commented there. If two dictionaries from different pillar files contain conflicting key names, the value of the first one gets overwritten by the second one (however, the order which one is the 1st is not known to me). If there is no key name conflicts, both key-values are simply accepted in the combined dictionary.

The problem of merging lists is not solved. Lists are just value objects like strings, numbers, or anything else (except dictionaries which recurse during merging). However, default strategy is not clear:

Dictionaries are more straightforward in this sense as deep merging (with value overwrites in case of conflicting keys) is almost always acceptable default behaviour.

Should this issue be reopened and be focused strictly on merging list items?

ldelossa commented 8 years ago

I believe so.

The idea of having a hierarchical pillar structure is difficult without the merging of lists. I can merge dictionaries but I'm left with valueless keys in some situations which is more of a hack then anything.

Example:

/core/packages.sls has list of 14 packages

/rabbitmq/packages.sls has 3 more packages

If both the core.packages and rabbitmq.packages pillars are applied to the host, then all 17 list items should be applied to the host.

It would be nice if in the actual top.sls we can define if two pillars with the same .sls name are to be merged or are they to overwrite. Since overwriting is still a very functional procedure in a hierarchical design.

On Saturday, November 21, 2015, Alexey Pakseykin notifications@github.com wrote:

Strictly speaking, issue #3991 https://github.com/saltstack/salt/issues/3991 (based on examples in the initial post here and there) is supposed to solve completely different problem of merging dictionaries.

The problem of merging dictionaries is solved as I commented there https://github.com/saltstack/salt/issues/3991#issuecomment-158651913. Currently, if dictionary keys from two different pillars match, their values gets overwritten (I'm not sure if there is a defined order which gets overwritten by which).

The problem of merging lists is not solved. Lists are just value objects like strings, numbers, or anything else (except dictionaries which recurse during merging). However, default strategy is not clear:

  • Should lists be overwritten (as it is now)?
  • Should they be combined? If yes, then which order they should be combined in (because order of list items is often used as a feature).
  • And if different strategies are required, there should be a way to control it.

Dictionaries are more straightforward in this sense as deep merging (with value overwrites in case of conflicting keys) is almost always acceptable default behaviour.

Should this issue be reopened and be focused strictly on merging list values?

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/28394#issuecomment-158652721.

uvsmtid commented 8 years ago

@ldelossa You see (I refer to my points above), making it "nice" may unnecessarily complicate rules and add new keywords just for the sake of merging lists while valueless keys in dictionaries already provide acceptable solutions in most of the cases.

Technically speaking, I don't feel urge for this feature (and I'm OK if this issue stay closed until something escalates the case).

In fact, there is only one unresolved (rare?) need - combining items in the order they appear:

# file_a.sls:
some_values:
    one: ~
    two: ~
# file_a.sls:
some_values:
    three: ~
    four: ~
# Result of `pillar.items`:
some_values:
    four: ~
    one: ~
    two: ~
    three: ~

When combined dictionary is traversed, the order of keys is undefined. Lists could solve the order issue.

What I really want (just to make public aware to strive for pointful and focused discussion) is:

ldelossa commented 8 years ago

I understand you're point. However to I do believe valueless keys is not an intuitive method for merging. A new user to salt would have trouble understanding why this is the method when a list is a simple collection of items.

I'm happy with the solution and work around and can live with it, but in my opinion it does add to the already steep learning curve of salt

On Saturday, November 21, 2015, Alexey Pakseykin notifications@github.com wrote:

@ldelossa https://github.com/ldelossa You see (I refer to my points above <#1512ad8377c7e0aa_issuecomment-158652721>), making it "nice" may unnecessarily complicate rules and add new keywords just for the sake of merging lists while valueless keys in dictionaries already provide acceptable solutions in most of the cases.

Technically speaking, I don't feel urge for this feature (and I'm OK if this issue stay closed until something escalates the case).

In fact, there is only one unresolved (rare?) need - combining items in the order they appear:

file_a.sls:

some_values: one: ~ two: ~

file_a.sls:

some_values: three: ~ four: ~

Result of pillar.items:

some_values: four: ~ one: ~ two: ~ three: ~

When combined dictionary is traversed, the order of keys is undefined. Lists could solve the order issue.

What I really want (just to make public aware to strive for pointful and focused discussion) is:

  • Disambiguation: dicts and lists are different topics.
  • Known solutions: dicts have already been solved and lists have known acceptable workaround.

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/28394#issuecomment-158659657.