search

JCSDA-internal / yamlprocessor

Process YAML files, handling sub-data-structure include and string value variable substitution
https://jcsda-internal.github.io/yamlprocessor
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

Add a pre-process step #25

Open mikecooke77 opened 1 week ago

mikecooke77 commented 1 week ago

Description

In order to allow for inclusion of stationlist type settings into the yaml file I have a suggestion that we include a pre-process step. This step would be run at the start of the yaml processor and would just cat the contents of a file at a suitable hierarchy.

A suggested use would be to split a file such as: https://github.com/MetOffice/jjdocs/blob/develop/docs/share/jopa/um-gl/gl_ops-um-jedi_process_iasi.yaml

Into one file which includes lines 1-117. This would go into jjaux and be called something like iasi_anchors.yaml. The current file would then have something like:

PREPROCESS: <file path to file in jjaux>

This would then explicitly separate out the types of includes from a parsed file to a straight include.

The alternative I was thinking of would be to cat the files in the shell script which runs the task. This would be less explicit than using the yaml parser.

matthewrmshin commented 1 week ago

Maybe the easiest way is to change yp-data's usage to something like this?

yp-data -o out.yaml in-one.yaml in-two.yaml ...  # ... other options and arguments
mikecooke77 commented 1 week ago

Yes that makes sense to me. Just for confirmation two gets concatenated onto the end of one?

matthewrmshin commented 1 week ago

... Just for confirmation two gets concatenated onto the end of one?

@mikecooke77 I would imagine so. In the presence of the -o output-file option, all arguments will become input files, and they will be concatenated in the same order as they are specified before being parsed.

matthewrmshin commented 5 days ago

From https://github.com/JCSDA-internal/yamlprocessor/pull/27#issuecomment-2346295644, we should consider the more complex preprocess solution before we can close this.

However, I'd need to better understand the use case and the nature of this more complex solution before designing a suitable syntax (or adopting a suitable templating language). Otherwise, we risk creating a system that makes our configuration really hard to understand.