pollination / queenbee

👑 Queenbee is a workflow language for creating workflows
https://pollination.github.io/queenbee
MIT License
18 stars 5 forks source link

Function artifact inputs as command parameters #289

Open AntoineDao opened 3 years ago

AntoineDao commented 3 years ago

I have noticed that @chriswmackey has used an input artifact as a parameter inside of a function's command. Technically this should not be allowed by Queenbee and we could write a test for it to make sure it doesn't break. Here is a shortened example of the command Function written by Chris:

Should be Invalid

type: Function
inputs:
- type: FunctionFileInput
  name: condition-csv
  description: A CSV file of thermal conditions output by a thermal mapping function.
  required: true
  path: condition.csv
name: tcp
command: ladybug-comfort map tcp {{inputs.condition-csv}}

Valid equivalent

type: Function
inputs:
- type: FunctionFileInput
  name: condition-csv
  description: A CSV file of thermal conditions output by a thermal mapping function.
  required: true
  path: condition.csv
name: tcp
command: ladybug-comfort map tcp condition.csv

Bug or Feature?

I think this issue could be turned into something more interesting than a simple bug fix. My thinking is that we already duplicate the condition.csv path in the example above.

What do we think of allowing this sort of parametrization of the command where an artifact input can be templated in the command such that executors (ie: queenbee-argo and queenbee-luigi then replace the templated value with the path key in the input artifact object?

This would have the following benefits:

I'm going to tag the people that seem most relevant to add to this discussion here: @chriswmackey, @mostaphaRoudsari and @tymokvo

AntoineDao commented 3 years ago

^ I have deployed the newer version of queenbee-argo which supports this proposed feature to staging so we can test it live while making a decision on whether or not to accept using artifact inputs and command template parameters.

tymokvo commented 3 years ago

I am in favor of increased flexibility by allowing this. Clearly one of the users already thinks it should be valid :smirk_cat: . I'm not totally sure what the trade off is though. Is there some risk or complexity involved in enabling it?