FedDF

[jafermarq]

FedDF

• Title: Ensemble Distillation for Robust Model Fusion in Federated Learning
• Venue: NeurIPS 2020
• Link to paper: https://proceedings.neurips.cc/paper/2020/hash/18df51b97ccd68128e994804f3eccc87-Abstract.html

Do you want to work on this baseline?

🌻 Check everything about the Summer of Reproducibility on flower.dev/summer

All available baselines are listed in the Summer of Reproducibility Dashboard and also in the GitHub Issues with the summer-of-reproducibility label. The content is the same.

📝 It is advised to complete these steps before your start working on your code. But if you can't wait to implement your baseline with Flower (we totally understand it 😄), please ensure you follow the steps on how to contribute a new baseline.

What follows are the steps 1 & 2 in the Summer of Reproducibility instructions.

1. Join the Summer of Reproducibility program

• [x] Join the Flower Slack and say "hi! 👋" in the channel #summer-of-reproducibility.
• [x] Pick a baseline from our curated list <---------------------------------------- [you are doing this now]

  2. Define the scope of your contribution

• [x] What are you going to reproduce? Add a comment to your issue and tell us about your plan regarding this baseline: what experiments from the paper are you reproducing?, for which datasets ? the more details you provide us with the better !

• [x] Check if you are eligible for a reward.

  As we have to comply with US/EU regulations, we have checked that individual contributors based on these countries or territories are eligible: Australia, Austria, Belgium, Bulgaria, Canada, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Gibraltar, Greece, Hong Kong SAR China, Hungary, India, Ireland, Italy, Japan, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Romania, Singapore, Slovakia, Slovenia, South Korea, Spain, Sweden, Switzerland, Thailand, United Arab emirates, United Kingdom United States.

  If where you are based is not on the list, please send us an email (summer@flower.dev) letting us know a bit about yourself (where are you currently based?, are you a university student? do you work at a public institution?). Please tell us the baselines you are interested in implementing (i.e. tell us your GitHub issue if you have crated one). We will reach back to you.

• [x] We will discuss with you about your contribution plan, if it sounds like a substantial enough contribution according to the Summer of Reproducibility rules (check our website flower.dev/summer), you'll get the OK from us to start working on your baseline!

What happens next?

• [x] This item will be moved to the In Progress stage by a member of the Flower Team.

• [ ] Follow the instructions for creating a new baseline which will guide you through the process step-by-step.

Is something wrong or not clear ?

• Ask a question directly in your issue.
• Reach out to us via the Flower Slack and ask your question in the #summer-of-reproducibility channel
• Check all the details (including FAQ) in the Summer of Reproducibility website: flower.dev/summer

[divyanshugit]

Hi @jafermarq , I would love to implement FedDF framework in flower.

Contributions + Result Reproduction:

• My primary focus will be on implementing the ensemble distillation framework for robust model infusion. Specifically, I aim to reproduce the results from paper using FedDF in CV by employing ResNet and VGG architectures on the CIFAR10 dataset. Additionally, in NLP I plan to utilize DistillBERT to replicate the outcomes on the AG News and SST2 datasets.

• Furthermore, I will create example notebooks that incorporate various architectures and experiment on other datasets. The specific architectures and datasets to be included will be determined as we progress.

Background:

Currently, I work as a AI Research Engineer at EnkryptAI which focuses on model security. Previously, I was working as a Research Associate at IISc, where I worked on MT for low-resource languages and Privacy Preserving Generative Models. I've done my bachelor's Electronics & Communication Engineering and I'm based in India(One of the eligble country in the list).

I would to discuss more on this and get started as soon as it possible. I go by @Divyanshu on the slack.

[jafermarq]

Hey @divyanshugit, it's great to see you are keen in implementing FedDF! 💯

From the architectures-datasets pairs you listed, could you point me to which experiments you want to reproduce exactly (i.e. what figure/table in the paper). Maybe Table 1, Table 2 (CIFAR-10 part), Table 3 and Figure 3? If you think you can take this then go for it! Your plan sound quite ambitious! Just think if you have the bandwidth for all these experiments before we move on to Stage 3.

Also, please indicate which baselines that appear in these tables are you also going to implement for the purpose of having a point of reference in the tables your baseline will reproduce (it could be FedAvg, FedProx or FedAvgM).

[divyanshugit]

For starters, I will focus on replicating the results from Table 2 and Figure 3 with FedAvg. Yeah, like my focus is going to be to finish replicating results, and if time permits then creating example notebooks as well.

[jafermarq]

@divyanshugit Great! thanks for the clarification. I've ✅ all the points above for Steps 1 & 2 and add you as the assignee to this issue. I also moved this FedDF baseline to "In Progress". You can find first steps on how to start working on your baseline via the link in the What happens Next ? section in the issue description.

Looking forward to see FedDF in action!