flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.74k stars 651 forks source link

[Core feature] Flytekit: Add support for ReferenceEntities in FlyteRemode.execute() #3627

Open tomasz-sodzawiczny opened 1 year ago

tomasz-sodzawiczny commented 1 year ago

Motivation: Why do you think this is important?

If I have a ReferenceEntity, e.g.

@reference_task(
    project="flytesnacks",
    domain="development",
    name="recipes.aaa.simple.join_strings",
    version="553018f39e519bdb2597b652639c30ce16b99c79",
)
def ref_t1(a: typing.List[str]) -> str:
    ...

I would like to be able to call it with FlyteRemote.execute:

flyte_remote.execute(ref_t1, dict(a=["foo", "bar"]))

Currently this code runs (because RemoteTask extends PythonTask and is treated as such), but ignores the version (and probably domain and project) - runs with the latest registered version of that task.

Goal: What should the final outcome look like, ideally?

Ideally I'd want to run any ReferenceEntity with

flyte_remote.execute(ref_entity, inputs=...)

And execute would respect all set values.

For consistency with the other types, methods like FlyteRemote.execute_reference_X() would have to be introduced too.

Describe alternatives you've considered

This is a working work-around:

flyte_remote.execute(ref_entity, version=ref_entity.id.version, inputs=...)

(Not considering project/domain, but same thing should work).

Propose: Link/Inline OR Additional context

Note: I'm happy to create a PR to flytekit if this change is reasonable.

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

github-actions[bot] commented 9 months ago

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏