grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
965 stars 52 forks source link

The work stealer should prefer expanding existing allocs to creating new ones #9

Closed mariusae closed 5 years ago

mariusae commented 7 years ago

The work stealer currently always creates new allocs, but this can create unnecessary data transfer (from the primary alloc's repository). Instead, the work stealer should attempt to expand an existing alloc before creating a new one.

mariusae commented 5 years ago

Stealer is deprecated in favor of the new -sched mode.