Add, subtract, or override semantics for repeated config items

[hjoliver]

Motivation: the special task lists don't behave nicely if you want to divide the whole suite into logically distinct parts. E.g. NIWA has dual use (operational and research) suites in which the operational product generation and ingestion graph and runtime is ~hidden from non-operational users in an include-file at the bottom of the suite definition. This works well for graph and runtime, but not special tasks - if I want to add a new operational clock-triggered task I have to repeat all previously declared clock-triggered tasks again in the new declaration.

There aren't many list-valued items. Are there occasions when the current override semantics would be preferred over append?

Note also, that graph items are currently treated differently: they have append semantics (in effect) in spite of not being list items. If we implement this proposal, consistency could be restored by simply making graph into a list item.

[matthewrmshin]

Ideally, however, we should have syntaxes to support both append and override semantics.

[trwhitcomb]

Another area where this is handy is in the specification of PBS/scheduler options. For example, in PBS the node count, walltime, etc. is all wrapped up in the -l option. This means that we need to have only a single -l for each task rather than allowing them to be built up.

We've found the override semantics useful by default - maybe if it doesn't cause too many problems with the parser, you could choose to either support += for explicit appends or == for explicit override

[hjoliver]

@trwhitcomb & @matthewrmshin - agreed, we'll allow both. I like += for append (list or string) and = for override (since == is typically a comparison operator).

[dpmatthews]

@trwhitcomb - note that you can split the PBS -l options into separate directives, see http://cylc.github.io/cylc/html/multi/cug-htmlse11.html#11.2.5

[arjclark]

Appended here from #1734 (closed as duplicate):

In some suites we've seen it'd definitely be useful to be able to define clock-triggered entries in an additive manner (as we do for graphing sections).

This is particularly useful for suites where graphing sections are brought in from include files as it would mean we could keep all the graphing related entries in one place rather than in the "master" suite.rc file with various switches etc.

i.e. we should be able to do this: suite.rc.processed:

# main suite.rc
[scheduling]
[[special tasks]]
  clock-triggered = foo
...
# main model graphing
...

# included sub-model.rc file
[scheduling]
[[special tasks]]
  clock-triggered = bar

1364 relates as a by-product of being able to put this sort of thing in graphing.

[hjoliver]

Title changed to reflect the growing consensus on this issue.

[hjoliver]

I would be keen to up the priority of this one, as it comes up in making site-portable suites too. However, we're not exactly short of things to work on at the moment :grimacing:

[hjoliver]

[meeting]

• need to support += and -= (list and string)
• and obliteration of items (not just unsetting) - e.g. for environment vars and job directives
• after this is done, similar functionality required for cylc broadcast

[TomekTrzeciak]

I agree that == is too synonymous with comparison syntax and I think := would be be better as a redefine operator and it wouldn't conflict with current usage of = for default assignment behaviour. To unset items completely perhaps -= with no value could used. Additionally, it could be useful to have syntax to set items if they are not set yet, maybe with ?=.

[matthewrmshin]

See also https://github.com/cylc/cylc/issues/2534#issuecomment-360739139

[ColemanTom]

Just curious if this was still a Cylc9 thing? It would be great being able to do

inherit = ....

...
{% include "site/..." %}
#which has
inherit += MY_LOCAL_PLATFORM_SETTINGS

type of thing.

[hjoliver]

IMO this would be useful, but we still have many higher priorities.