experiment.py: Graph: local trained model vs. global model accuracy

georgymh / decentralized-ml

Interoperable and decentralized machine learning.

Apache License 2.0

9 stars 5 forks source link

experiment.py: Graph: local trained model vs. global model accuracy #19

Open setsometso opened 6 years ago

setsometso commented 6 years ago

Hi Georgy,

Currently I am trying to understand your federated learning implementation (your paper and the experiment.py). I can reproduce your results. The following graph would be interesting: A graph, which shows the local accuracy of client1, client 2..., client n (e.g. after his first local training) and the accuracy of the server after he aggregates and distributes the models over n communication rounds (the updated master model)

How can i implement that kind of graph? A help would be great.

kiddyboots216 commented 6 years ago

@setsometso Hi, Ashwinee here (coauthor) here is some cursory advice, I will try to help more in detail at a later date if you're still stuck.

We currently have a Flask listener doing the federated averaging. You can use repurpose this, or make your own, it's not too hard. Add a POST endpoint to the Flask listener that will accept a float (accuracy) and saves that float.
When the individual clients do their SGD step and send the weights to the server, simply have them POST their accuracy as well.
Have the federated averaging server which is the Flask listener also save its accuracy to the list of saved accuracies.
At the end of a training "experiment" have your server save the accuracies to a file (or print them out, and copy-paste these into a file)
Create a graph from this (I don't think this is the hard part for you, but let me know if it is and I will follow up!) using Seaborn or Matplotlib

setsometso commented 6 years ago

Hi @kiddyboots216 thanks for your Advice, i am using the code out of your repository (experiment.py mainly). For my understanding: Is it correct, that if i number of clients = 3, and fraction = 1, that in reality I actually have 4 clients? The 3 worker clients and if I see right, a client who simply queries the aggregated model per communication round on the test record and then shows the accuracy per communication round?

Currently a client tracks only the loss (via tensorboard), I could not just add the accuracy (on the test record) and then pick this?

kiddyboots216 commented 6 years ago

@setsometso To your first question: Yes I believe that's correct - @georgymh feel free to chime in if that's not the case as this is mostly your code. You can certainly do that (track the accuracy). Whatever method works out for you, please let us know and feel free to file a PR with some interesting graphs or something!