argoproj-labs / hera

Hera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo. ⭐️ Remember to star!
https://hera.rtfd.io
Apache License 2.0
550 stars 105 forks source link

Add ability to subclass from both Input and Output allowing for passthrough IO #1109

Closed elliotgunton closed 2 days ago

elliotgunton commented 2 months ago

Pull Request Checklist

Description of PR This PR makes it possible to subclass Input and Output for the new decorators in HEP0001, and then pass an output from a script template call directly into the next step or task. i.e. instead of

@w.dag()
def my_dag():
    output_task = give_output()
    take_input(
        PassthroughIO(
            my_str=output_task.my_str,
            my_int=output_task.my_int,
            my_artifact=output_task.my_artifact,
        )
    )

you can do

@w.dag()
def my_dag():
    output_task = give_output()
    take_input(output_task)

Note this functionality is limited to script-decorated functions only for now, as other template types would need an even more involved approach where this code had already become complex enough, as we are reconstructing an Output from a step or task object.

elliotgunton commented 2 months ago

I think this will cause a lot of confusion if it only works for script templates

Fair point, I wanted to see if it was possible for script templates first. As for stubs it should work fine as long as they are decorated as script templates - we only need the signature, which is why it's fairly easy to do for script templates as we can access template.source. Haven't looked into other template types extensively

elliotgunton commented 2 days ago

Closing this as stale and it's only a POC which isn't high priority for now