Cylc config with multiple items - output upgrades idea

[ColemanTom]

Running cylc config -i something -i something_else ..., the results will be

1. Everything is present - something is printed for each item
2. Something is missing - up to the "something is missing", items get printed. After the "something is missing", nothing gets printed

My question is, should cylc config abort on the first missing item, or should it print something for each item?

At the moment, given the way things are printed, with no differentiation, I guess aborting immediately is the only sane option. However, could there be a more grep like implementation? Below I've suggested an option --add-prefix

$ cylc config workflow --add-prefix -i '[runtime][task][meta]' -i '[runtime][second_task][meta]'
[runtime][task][meta]: title = My Title
[runtime][task][meta]: description = My longer description
[runtime][second_task][meta]: title = My Second Title
[runtime][second_task][meta]: description = My longer description for the second task

If this approach is used, then we could then have

$ cylc config workflow --add-prefix  --continue-if-missing -i '[meta]foo' -i '[runtime][task][meta]' -i '[runtime][missing_task][meta]' -i '[runtime][second_task][meta]'
ItemNotFoundError: You have not set "[meta]foo" in this config.      # stderr
[runtime][task][meta]: title = My Title
[runtime][task][meta]: description = My longer description
InvalidConfigError: "[runtime][missing_task][meta]" is not a valid configuration for flow.cylc   # stderr
[runtime][second_task][meta]: title = My Second Title
[runtime][second_task][meta]: description = My longer description for the second task

Why do this? Well, it allows you to query a bunch of stuff in one request, and obtain information which may not exist without aborting. This can then be examined afterwards to check if information is missing.

The problem I was trying to solve is - I want to make sure all tasks in my workflow have a [meta]URL defined and the workflow itself has a [meta]URL defined. The only way I can think of to do that check is to parse the results from cylc config. Noting that ideally the workflow doesn't need to be installed to perform this check as it should be doable vi CI.

[oliver-sanders]

My question is, should cylc config abort on the first missing item, or should it print something for each item?

I think this behaviour is sensible and can be useful in some cases. E.G. I sometimes scan workflows looking for particular config items.

If you don't want this behaviour, the --defaults option is your best bet, this will pull in the config default if the setting is not defined in the workflow. With the --default option the command should never abort.

cylc config --default -i something -i something_else ...

I want to make sure all tasks in my workflow have a [meta]URL defined and the workflow itself has a [meta]URL defined

This'll do it:

cylc config <workflow> --default -i [meta]URL $(cylc list <workflow> | sed 's/\(.*\)/-i [runtime][\1][meta]URL/')

I've been interested in the idea of plugin validators which would be an interesting option for this going forward. Essentially a plugin Python function that could operate on the config and raise an exception if it wants to fail validation. We would probably need to do a bit of groundwork before we can present an API for this that we would be willing to maintain though.

[ColemanTom]

If you don't want this behaviour, the --defaults option is your best bet, this will pull in the config default if the setting is not defined in the workflow. With the --default option the command should never abort.

Default doesn't seem to do anything for custom fields. For other things, like err-script, it will return an empty line.

$ cylc config access_g4_pp_cylc835_grp21 --defaults -i '[meta]URL' -i '[meta]OtherItemDoesNotExist' -i '[meta]description'
https://website/latest
ItemNotFoundError: You have not set "[meta]OtherItemDoesNotExist" in this config.

p.s. is it --default or --defaults? Using both doesn't lead to a failure about an unrecognised option being passed in, but only --defaults is documented in the help area.

[ColemanTom]

cylc config --default -i [meta]URL $(cylc list | sed 's/(.*)/-i [runtime][\1][meta]URL/')

Does Cylc guarantee the order is preserved for the -i order provided to cylc config? The help does not say it is, so I feel it would be unsafe to assume it is and will remain that way.

[ColemanTom]

Looking at it, without knowing which line belongs to which -i option, a multi-line thing would cause problems for checking anything that might be missing I think.

$ cylc config access_g4_pp_cylc835_grp21 --defaults -i '[runtime][convparams_000_019]pre-script' -i '[runtime][convparams_000_020]pre-script'
if [[ ... ]]; then
    do_some_setup
fi
if [[ ... ]]; then
    do_some_setup
fi