votingworks / arlo-cvr-encryption

e2e encryption and verification features to assist RLA audits
GNU Affero General Public License v3.0
7 stars 3 forks source link

Replaced ray_reduce with ray_map_reduce #38

Closed danwallach closed 3 years ago

danwallach commented 3 years ago

The old code did the map in batches, manually, then called out to our own reduce library, which used ray.wait. This was a mess.

The new code has a general-purpose solution, where you define map, reduce, and zero methods on a "context" class, and all the rest is handled under the hood. In particular, rather than batching, we extend the original reduction strategy, of using ray.wait to pay attention to the number of pending tasks, and when it gets below the desired number, more map tasks are launched.

Thus, no batching, and we're always keeping as many tasks in flight, up to the desired ceiling, as possible.

All the tests pass, and it runs fine on localhost. Still need to try this on a big cluster. Also need a second pair of eyes to make sure I didn't do something stupid.

CLAassistant commented 3 years ago

CLA assistant check
All committers have signed the CLA.