Implement merge magic

[lowell80]

This is mostly a brain dump, probably need to split this into more specific tickets

Here are my thoughts on what this could look like:

• Allow certain keys to be suppressed by subsequent layers by using some kind of sentinel value, like <<UNSET>> or something. Use case: Disabling a transformer defined in the upstream layer. So instead of TRANSFORMS-class= being shown in the output, the entire key is removed from the destination.)
• Allow stanzas to be suppressed with some kind of special key value token. Use case: Removing unwanted eventgen related source matching stanza's like [source::...(osx.*|sample.*.osx)] instead of having to suppress individual keys (difficult to maintain).

• Allow a special wildcard stanzas that globally apply one or more keys to all subsequent stanzas. Use Case: set index=os_<ORG> on all existing stanzas in an inputs.conf file. Perhaps this could be setup like

  [*]
  index=os_prod
     --  or --
  [<<COPY_TO_ALL>>]
  index=os_prod

  Assume this will apply to all currently-known stanzas. (Immediately applied, during the merging process, not held and applied after all layers have been combined.)

• Possibly allow list augmentation (or subtraction?) to comma or semicolon separated lists.

  TRANSFORMS-syslog = <<APPEND>> ciso-asa-sourcetype-rewrite
      -- or -- 
  TRANSFORMS-syslog = <<REMOVE>> ciso-asa-sourcetype-rewrite

  Possible filter operators include:

  <<APPEND>> string           Pure string concatenation
  <<APPEND(sep=",")>> i1,i2   Handles situation where parent may be empty (no leading ",")
  <<REMOVE>> string           Removes "string" from parent value; empty if no parent
  <<ADD(sep=";")>> i1;i2      Does SET addition.  Preserves order where possible, squash dups.
  <<SUBTRACT(sep=",")>> i1    Does SET reduction.  Preserves order where possible, squash dups.

  Down the road stuff: (Let's be careful NOT to build yet another template language...)

  <<LOOKUP(envvar="HOSTNAME")>>               <<-- setting host in inputs.conf, part of a monitor stanza

  NOTE: This is possibly bad or misleading example as HOSTAME maybe on a deployer not on the SHC member, for example.

  <<LOOKUP(file="../regex.txt", line=2)>>     <<-- Can't thing of a legitimate use, currently
  <<RE_SUB("pattern", "replace", count=2)>>   <<-- Parsing this will be a challenge, possibly allow flexible regex start/stop characters?
  <<INTROSPECT(key, stanza="default")>>       <<-- Reference another value located within the current config file.

It may be nice to set these up and register them via decorators. Make all filter calls align with Python function calling norms (like what Jinja2 does)

 @register_filter("APPEND")
 def filter_op_APPEND(context, sep=","):
     # context.op       name of the filter, "APPEND" in this case.
     # context.stanza   name of the current stanza
     # context.key      the key who's value is being operated upon
     # context.parent   original string value
     # context.payload  raw value of the current key, without magic text (may be blank)

Allow for standard python style function argument passing; allowing for both positional and named parameters (that is *arg and **kwargs)

[crumpetcrusher]

An additional feature would be tombstoning an entire .conf file (or dir!), something like this.

├─bin
├─conf.d
│ ├─10-upstream
│ │ ├─data
│ │ │ └─ui
│ │ │   └─views
│ │ │     └─dashboard.xml
│ │ ├─inputs.conf
│ │ └─indexes.conf
│ ├─40-idx_layer
│ │ ├─data.disabled # empty file, remove data dir entirely when included in combine/package/etc.
│ │ ├─inputs.conf.disabled # not needed at idx if using heavy forwarder
│ │ └─indexes.conf # maybe edits for retention or volumes
│ └─50-shc_layer
│   └─indexes.conf.disabled # empty file, remove it entirely.
├─lookups

I think this is what slim partition is intending to do, but I can't confirm it's doing anything in its current state. It's never generated proper layers for me. It seems to be intended as straight magic in that it just knows you wouldn't have views on an indexer so drop all those. 🤷‍♂️

[lowell80]

@crumpetcrusher. That's a really good point. Yeah slim's @placement XXX idea is really interesting, but somewhat problematic imho.

I'm going to open this as a new enhancement ticket and put a proper reply in there.