Add Flower Baseline: FedHe

Paper

FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

Link

Maybe give motivations about why the paper should be implemented as a baseline.

FL typically assumes that all edge devices share the same machine learning model during training, such as an identical neural network architecture. However, in reality, the computational and storage capabilities of various devices may differ. Furthermore, minimizing communication overhead is a challenging task but crucial for enhancing training efficiency in FL. In their research, the authors introduce a novel FL technique known as FedHe, which draws inspiration from knowledge distillation. This approach enables the training of diverse models and facilitates asynchronous training processes while significantly reducing communication overhead. The analysis and experimental results presented in their paper indicate that their method outperforms state-of-the-art algorithms in terms of both communication overhead and model accuracy.

Key Points:

Supports heterogenous models (model architectures can stay private and vary depending on compute power of device)
Massive reduction in communication cost (reduced load on network)
Improved accuracy over previous state-of-the-art algorithms

Is there something else you want to add?

No response

Implementation

To implement this baseline, it is recommended to do the following items in that order:

For first time contributors

[x] Read the first contribution doc
[x] Complete the Flower tutorial
[ ] Read the Flower Baselines docs to get an overview:
- [ ] https://flower.dev/docs/using-baselines.html
- [ ] https://flower.dev/docs/contributing-baselines.html

Prepare - understand the scope

[ ] Read the paper linked above
[ ] Create the directory structure in Flower Baselines (just the __init__.py files and a README.md)
[ ] Before starting to write code, write down all of the specs of this experiment in a README (dataset, partitioning, model, number of clients, all hyperparameters, …)
[ ] Open a draft PR

Implement - make it work

[ ] Implement some form of dataset loading and partitioning in a separate dataset.py (doesn’t have to match the paper exactly)
[ ] Implement the model in PyTorch
[ ] Write a test that shows that the model has the number of parameters mentioned in the paper
[ ] Implement the federated learning setup outlined in the paper, maybe starting with fewer clients
[ ] Plot accuracy and loss
[ ] Run it and check if the model starts to converge

Align - make it converge

[ ] Implement the exact data partitioning outlined in the paper
[ ] Use the exact hyperparameters outlined in the paper
[ ] Make it converge to roughly the same accuracy that the paper states
[ ] Commit the final hyperparameters and plots
[ ] Mark the PR as ready

adap / flower