project-codeflare / codeflare

Simplifying the definition and execution, scaling and deployment of pipelines on the cloud.
https://codeflare.dev
Apache License 2.0
218 stars 35 forks source link

Future references #18

Closed raghukiran1224 closed 3 years ago

raghukiran1224 commented 3 years ago

Overview

For uniformity of input output to pipeline stages, we need to have a list of Future references to Xy objects. Since Ray remote calls always return a future reference and we return a holder class with reference to X and y, which are compute tasks that can be in flight. A list of references allows for the parallelization of operations.

Uniformity of data exchange is enabled by choosing a list of future references as the way to exchange data.

Acceptance Criteria

Questions

Assumptions

Reference

raghukiran1224 commented 3 years ago

moving to external, addressed