emews / EQ-SQL

1 stars 0 forks source link

Batch Retrieve in as_complete #42

Closed ncollier closed 6 months ago

ncollier commented 6 months ago

as_complete can be slow over remote queues - each individual result is returned from the remote. A batch argument would improve performance by allowing the remote to retrieve a batch of batch_size and return that amount. This is particularly useful when the ME is synchronous and waiting for all results, or asynchronous but working with a large number of results.

ncollier commented 6 months ago

local implementation can accumulate values in list and yield from that list. Remote function should accumulate and return that list to the queue implementation which then yields from that list.

ncollier commented 6 months ago

Implemented here: d90e5d0