Design a path for work items to report progress

[pabloem]

The Beam API implements several RPCs for processing of data.

The main RPC is the ProcessBundle RPC, which we've implemented.

During the execution of a ProcessBundle rpc, the Beam worker consumes data to process, and outputs data. While this request is being executed, the runner can request progress reports from the worker. These progress reports are requested via the ProcessBundleProgress rpc.

This work item consists on:

• Adding a thread/task that performs periodic ProcessBundleProgress rpcs for the SDK worker
• Passing the results of these progress reports back to the coordinator thread somehow

Some ideas:

• We could spawn a thread in ray_execute_bundle that polls the worker
• This thread could then push updates to an actor that is shared globally and the coordinator thread can consult?

This is just one idea, but we can do things in different ways.

[pabloem]

@valiantljk this is something that you could consider taking a stab with? : )

you'd have to add some smart code so that ongoing ray_execute_bundle tasks can report progress - a bit of Beam code to assemble the ProcessBundleProgressRequest protos, and some Ray code to create actors/threads to report results

[pabloem]

lmk if that helps

[iasoon]

For how to do the ProcessBundleProgressRequests, you can check the original FnApiRunner code, I think there's a basic implementation there.