search

PrefectHQ / prefect-ray

Prefect integrations with Ray
https://prefecthq.github.io/prefect-ray/
Apache License 2.0
63 stars 5 forks source link

Handle dependent tasks concurrently in the `RayTaskRunner` #37

Closed desertaxle closed 1 year ago

desertaxle commented 2 years ago

When updating the RayTaskRunner #35, wrapping of Prefect futures with Ray task futures was introduced. For tasks with upstream dependencies, the upstream tasks need to synchronously unwrap the Ray future to provide the output of the upstream task to the downstream task. This may result in degraded performance for flows using the RayTaskRunner with many dependent tasks. The aim is to discover a way to concurrently handle Ray futures to enable concurrent execution for flows with dependent tasks.

@anticorrelator @zangell44 feel free to add any additional details or correct any inaccuracies in my description of the issue!

zanieb commented 2 years ago

I believe instead of submitting the task run function directly it could submit its own helper which would resolve any ray references in the input then call the task run function. Seems kind of straightforward maybe? :)