Closed danielnorberg closed 7 years ago
danielnorberg
commented
8 years ago This was inspired by seeing the programmatic looping here in the digdag-analytics poc: https://github.com/treasure-data/digdag-analytics/blob/master/se_poc_emails/tasks/poc_workflow.rb#L15
It seems desirable to be able to use digdag to perform these queries and leverage the digdag parallelism, retry and dependency tracking etc facilities instead.
frsyuki
commented
8 years ago & and *). This works now as you mentioned.key: @foo, key: @{foo}, key: @${foo}, key: $@foo, key=: foo, key: !param foo etc.foo if a value is ${foo} without any characters around it.
If foo is an integer, prefix_${foo} is string, ${foo}_suffix is string, but ${foo} is integer.
I think this is problematic when users want to convert integer to string. In YAML syntax, you can use "${foo}" but there're no ways to distinguish it from ${foo}. So, users need to use ${foo.toString()}. I think this is not intuitive.int foo = conf.get("foo", int.class) or List<String> foo = conf.getList("foo", String.class) API to get a config value. This get or getList methods have a chance to convert value type.
Currently, conf.getList("foo", String.class) fails if actual value of foo is not a list. But we can change the behavior so that it tries to parse the string as JSON if the actual value is string.
This works because ${foo} converts foo to a JSON string if foo is a list or map.
But this doesn't work if operator is Ruby or Python (dynamically-typed). params.get("foo") is string even if the operator expects list.raw(...) to preserve type. So, ${foo} is string, but ${raw(foo)} is list. prefix_${raw(foo)} is string, though.In a very old version, syntax was key=: foo (so, idea 2).
danielnorberg
commented
8 years ago I like the key: @foos syntax but @ is reserved and cannot be used to start a token in yaml =/
http://yaml.org/spec/current.html#c-directive
On the other hand, less ways to parameterize things means less surprise and mistakes for our users, so allowing key: ${foos} (idea 4) might be more desirable.
danielnorberg
commented
8 years ago It would also be useful to be able to do things like:
run: +main
+main:
for_each>:
foo:
- bar: a
baz: b
- bar: c
baz: d
_do:
sh>: "echo ${foo.bar}: ${foo.baz}"Where the tasks executed for the above would be:
echo a b
echo c dCurrently the above produces:
error:
* +main:
io.digdag.core.repository.ModelValidationException: Validating workflow task failed
name can't contain character "+for-foo={bar=a, baz=b}"
for_each>parameters can be parameterized like below by utilizing YAML anchors:But it might be useful to allow for parameterizing
for_each>using actual digdag parameters. E.g.And using parameters explicitly set by e.g. a
py>task:Attempting to parameterize
for_each>like this currently fails with the below error:I'm uncertain whether the best approach would be to make
${...}expansion more intelligent about types of parameters and not coerce everything to a string, or if it would make more sense to introduce another syntax.