Open mortendahl opened 5 years ago
Spoke with @mortendahl briefly and we agreed that much of the code beneath this API can be written as a custom tf.distribute.Strategy. Roughly speaking, the aggregate
function above could be written in the language of these Strategies. This should simplify how this interface works with the Keras and Estimator APIs.
Note to self: find a way where the federated protocol simply wraps an existing model on the model owner?
Design and discussion has moved into a RFC: https://github.com/tf-encrypted/rfcs/pull/1
@yanndupis @jvmancuso the more I look into this the more I think we should simply stick with the FL example we have, but upgrade it to use TF Keras and TF 2.0.
I'll keep working on the details in integrating with TF Federated and distribution strategies but worried the time will be short and not worth rushing.
WDYT?
@mortendahl - happy to explore other alternatives. In general, I think it's important that's easy to use and we can support arbitrary Keras models. So maybe we just need to improve the abstraction and automate some of the stuff in the example. For example [here], we would like to avoid defining manually the model weights.
It would be nice to kick the training process with model.fit(data_sources, epochs=10)
and see the training process. Also I think we would like to have some utils to distribute the data among several data owners.
@yanndupis @jvmancuso the more I look into this the more I think we should simply stick with the FL example we have, but upgrade it to use TF Keras and TF 2.0.
Yeah, I agree. Let's stick with the current abstraction and add some syntactic sugar, like what @yanndupis referred to.
It would be nice to kick the training process with model.fit(data_sources, epochs=10) and see the training process.
@yanndupis how do you think we could accomplish the above? Would we create a keras model wrapper? Or be able to make use of tfe.keras somehow?
Also I think we would like to have some utils to distribute the data among several data owners.
Can you provide some more details about this also? Not 100% sure what you mean.
Can you provide some more details about this also? Not 100% sure what you mean.
In tf federated, they have this concept of tff.simulation.datasets.emnist.load_data. It constructs a tf.data.Dataset
object for each data owner. Currently here, the number of data owner and tfrecord file is hard coded. If we want be able to easily experiment, it would be nice to have a method which takes data as a numpy format (or even tfrecord) and number of players. Then the data would be distributed evenly among these players.
For model.fit
, I will put more thoughts into it tomorrow. It could be just wrapping up this section in a .fit
. I also like the API above suggested by @mortendahl, we just need to assess if it's too much work for our timeline.
For example [here], we would like to avoid defining manually the model weights.
Yes, fully agree.
It would be nice to kick the training process with
model.fit(data_sources, epochs=10)
and see the training process.
Following the current abstractions in the example, it may make more sense to take the data owners as input instead of the data sources.
it would be nice to have a method which takes data as a numpy format (or even tfrecord) and number of players. Then the data would be distributed evenly among these players.
This seems useful indeed, but let's make sure to clearly mark it as something that's for simulation use only.
This is being worked on here. Feedback welcome!
In https://github.com/tf-encrypted/tf-encrypted/pull/695 we added the flag to support simulating the splitting the data into the parties. It currently only works with local computations we should see if we can get this to work with remote computations with either tf.device
or some other way. See this thread: https://github.com/tf-encrypted/tf-encrypted/pull/695#discussion_r335050263
Goals
Dense
andReLU
to sometimes let the how depend on the whatDeliverables:
Background
TF Distribute strategies
From the docs, distribute strategies are about
state & compute distribution policy on a list of devices
.From the guide:
TF Federated
From the docs:
On TF distribution strategies vs Federated Core, from the docs:
From the tutorial on text:
Note that
state
can used to update a local clone of the model for evaluation after each iteration:Terminology
Functionality and Protocol: the former is basically used as a function from and to local tensors, while the latter as a more general means to specify how functionalities are to be computed using e.g. cryptographic techniques. As such, protocols are the only one of the two that are used as context handlers. Roughly follows UC terminology although functionalities are not intended to be used as sub-protocols.
Suggested API