tf-encrypted / moose

Secure distributed dataflow framework for encrypted machine learning and data processing
Apache License 2.0
57 stars 15 forks source link

Cleaning up LocalMooseRuntime API #1099

Closed jvmncs closed 2 years ago

jvmncs commented 2 years ago

Closes #1093 Closes #1094

Removes role_assignments from LocalMooseRuntime.evaluate_computation.

The standard way to construct LocalMooseRuntime is now:

identities=["alice", "bob", "carole"]
runtime = pm.LocalMooseRuntime(identities)

For adding storage to the simulated hosts:

identities=["alice", "bob", "carole"]
storage = {"alice": {"alice_array": np.ones((3,3))}}
runtime = pm.LocalMooseRuntime(identities, storage_mapping=storage)

Also removes role_assignment from AsyncTestRuntime, since it's purpose is nearly identical as the python version. Note that AsyncSession still requires a role_assignment: HashMap<Role, Identity>, since this reaches deeper into the Moose code than I wanted this change to go.

Rewrote a bit of the correlation notebook example to match this (and also fixed a tiny bug)

jvmncs commented 2 years ago

One concern I had: what happens when I feed a computation to the LocalMooseRuntime with placements that don't have corresponding identities in the runtime? Previously, we would error out, but now I think this kind of error might reach all the way into the AsyncTestRuntime and just result in hanging processes. Not totally sure about that yet though, will check it out tomorrow.

mortendahl commented 2 years ago

but now I think this kind of error might reach all the way into the AsyncTestRuntime and just result in hanging processes

Yes I think you might be right. At least if it makes it to AsyncExecutor::run_computation, which will silently filter out operations and causing a hang.