arcalot / arcaflow-engine

Arcaflow is a highly-portable workflow engine enabling modular and validated pipelines through containerized plugins.
https://arcalot.io/arcaflow/
Apache License 2.0
6 stars 9 forks source link

Add support for YAML anchors in input files #132

Open dustinblack opened 6 months ago

dustinblack commented 6 months ago

Please describe what you would like to see in this project

In writing a complex multi-level workflow, some input parameters may be repeated a number of times. It would simplify writing and maintaining the workflow inputs if YAML anchors were supported.

Please describe your use case

An example input snippet:

sysbench_cpu_tests:
  - sysbench_params:
      threads: 1
      time: 10
      percentile: 99
      cpu-max-prime: 500000
    pcp_params:
      pmlogger_interval: 1.0
      pmlogger_metrics: "kernel.all.load kernel.percpu.cpu.vuser kernel.percpu.cpu.sys kernel.percpu.cpu.wait.total mem.util.used mem.vmstat.pgpgin mem.vmstat.pgpgout swap.pagesin swap.pagesout"
  - sysbench_params:
      threads: 4
      time: 10  
      percentile: 99
      cpu-max-prime: 500000
    pcp_params:
      pmlogger_interval: 1.0
      pmlogger_metrics: "kernel.all.load kernel.percpu.cpu.vuser kernel.percpu.cpu.sys kernel.percpu.cpu.wait.total mem.util.used mem.vmstat.pgpgin mem.vmstat.pgpgout swap.pagesin swap.pagesout"
  - sysbench_params:
      threads: 8
      time: 10
      percentile: 99
      cpu-max-prime: 500000
    pcp_params:
      pmlogger_interval: 1.0
      pmlogger_metrics: "kernel.all.load kernel.percpu.cpu.vuser kernel.percpu.cpu.sys kernel.percpu.cpu.wait.total mem.util.used mem.vmstat.pgpgin mem.vmstat.pgpgout swap.pagesin swap.pagesout"

To simplify this, I'd like to do something like this:

sysbench_cpu_tests:
  - sysbench_params:
      threads: 1
      time: &time 10
      percentile: &percentile 99
      cpu-max-prime: &cpumaxprime 500000
    pcp_params:
      pmlogger_interval: &pmlinterval 1.0
      pmlogger_metrics: &pmlmetrics "kernel.all.load kernel.percpu.cpu.vuser kernel.percpu.cpu.sys kernel.percpu.cpu.wait.total mem.util.used mem.vmstat.pgpgin mem.vmstat.pgpgout swap.pagesin swap.pagesout"
  - sysbench_params:
      threads: 4
      time: *time  
      percentile: *percentile
      cpu-max-prime: *cpumaxprime
    pcp_params:
      pmlogger_interval: *pmlinterval
      pmlogger_metrics: *pmlmetrics
  - sysbench_params:
      threads: 8
      time: *time  
      percentile: *percentile
      cpu-max-prime: *cpumaxprime
    pcp_params:
      pmlogger_interval: *pmlinterval
      pmlogger_metrics: *pmlmetrics