[Core feature] pyflyte run hybrid mode

[kumare3]

Motivation: Why do you think this is important?

There are times when it would be nice to execute some tasks remotely, while keeping other tasks local. With Flyte Agents we have already enabled this. Flyte agents have the capability of calling external services as long as "secrets" are configured correctly. I opine that this should be available for all task types. With Single task execution and Flyte remote this would be possible. This issue also proposes a UX for the same.

Goal: What should the final outcome look like, ideally?

A user could use pyflyte run --hybrid module.py my_wf and it should send certain tasks to the Flyte remote instance. If --hybrid is not specified then all tasks will be run locally if --remote is specified, then all tasks with workflow will be run remotely.

This will also lend well into @eager workflows as they become more rampant. This relates to this issue https://github.com/flyteorg/flytekit/pull/1833

IMO the solution proposed here --hybrid could be applied to eager as well! cc @cosmicBboy

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

Recommendation introduce a new decorator @remote which can be added to any task

@task 
@remote 
def foo1(i: int) -> int:
   ...

@task 
def foo2(i: int) -> int:
   ...

@workflow
def wf(i: int):
   j = foo1(i=i)
   foo2(i=j)

Now if we invoke pyflyte run wf.py wf -> no deviation from current behavior pyflyte run --remote wf.py wf -> no deviation from current behavior pyflyte run --hybrid wf.py wf -> only task marked with @remote decorator is invoked remotely. in the above example foo1

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[kumare3]

I don't think this is hacktoberfest- this one maybe too complicated?

[cosmicBboy]

let's discuss the design for this feature, it can easily become confusing between this and eager workflows what the "right way" of doing things is, might even be worthy of an RFC. The basic idea of this issue is supported with @eager(..., local_entrypoint=True), see here.

IMO @hybrid would be less confusing naming here... or whatever name we choose, it should match the pyflyte run flag that we decide on.

[kumare3]

this could be called mixed-mode too