Overwriting parameters in "included" DAGs?

[FredericoCoelhoNunes]

I have a main DAG that looks like this:

+dag:
  +step:
    !include : 'workflows/echo.dig'

echo.dig looks like the following:

+subdag:
  +echo:
    p: 10
    echo>: "hello ${p}"

Now I need to run this DAG using a different value for the p parameter, in the echo task. I haven't been able to figure out if this is even possible, and if so, how I can do it. For context: I am trying to write a test for one of my DAGs, and I want to override some of the production parameters with test parameters.

Thank you.

[hiroyuki-sato]

Hello, @FredericoCoelhoNunes

This comment may help.

https://github.com/treasure-data/digdag/issues/990#issuecomment-463544061

See also: #990 #749 #847

[FredericoCoelhoNunes]

Hi @hiroyuki-sato , thanks but unfortunately I still don't understand. I have tried setting the parameter with -p p="test value", I have also tried -p step.subdag.echo.p="test value" and other variations of this, but couldn't find the right way.

[hiroyuki-sato]

Hello, @FredericoCoelhoNunes

IIUC, Unfortunately, Digdag use the parameter which defined in dig file even if you use -p option. (Priority: -p opiton < dig file param).

If your dig file not define p parameter, you can set p using command line

#_export:
#  p: default

+dag:
  +step:
    !include : 'workflows/echo.dig'

+subdag:
  +echo:
    #p: 10
    echo>: "hello ${p}"

digdag start sample sample -p p=hoge --session now

It outputs hello hoge.

[FredericoCoelhoNunes]

Got it, thanks! We'll go for an alternative solution that doesn't involve parameterization.

Thank you!