Power-of-Choice

[jafermarq]

Power-of-Choice

• Title: Towards Understanding Biased Client Selection in Federated Learning
• Venue: AISTATS 2022
• Link to paper: https://proceedings.mlr.press/v151/jee-cho22a.html

Do you want to work on this baseline?

🌻 Check everything about the Summer of Reproducibility on flower.dev/summer

All available baselines are listed in the Summer of Reproducibility Dashboard and also in the GitHub Issues with the summer-of-reproducibility label. The content is the same.

📝 It is advised to complete these steps before your start working on your code. But if you can't wait to implement your baseline with Flower (we totally understand it 😄), please ensure you follow the steps on how to contribute a new baseline.

What follows are the steps 1 & 2 in the Summer of Reproducibility instructions.

1. Join the Summer of Reproducibility program

• [x] Join the Flower Slack and say "hi! 👋" in the channel #summer-of-reproducibility.
• [x] Pick a baseline from our curated list <---------------------------------------- [you are doing this now]

  2. Define the scope of your contribution

• [x] What are you going to reproduce? Add a comment to your issue and tell us about your plan regarding this baseline: what experiments from the paper are you reproducing?, for which datasets ? the more details you provide us with the better !

• [x] Check if you are eligible for a reward.

  As we have to comply with US/EU regulations, we have checked that individual contributors based on these countries or territories are eligible: Australia, Austria, Belgium, Bulgaria, Canada, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Gibraltar, Greece, Hong Kong SAR China, Hungary, India, Ireland, Italy, Japan, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Romania, Singapore, Slovakia, Slovenia, South Korea, Spain, Sweden, Switzerland, Thailand, United Arab emirates, United Kingdom United States.

  If where you are based is not on the list, please send us an email (summer@flower.dev) letting us know a bit about yourself (where are you currently based?, are you a university student? do you work at a public institution?). Please tell us the baselines you are interested in implementing (i.e. tell us your GitHub issue if you have crated one). We will reach back to you.

• [x] We will discuss with you about your contribution plan, if it sounds like a substantial enough contribution according to the Summer of Reproducibility rules (check our website flower.dev/summer), you'll get the OK from us to start working on your baseline!

What happens next?

• [x] This item will be moved to the In Progress stage by a member of the Flower Team.

• [ ] Follow the instructions for creating a new baseline which will guide you through the process step-by-step.

Is something wrong or not clear ?

• Ask a question directly in your issue.
• Reach out to us via the Flower Slack and ask your question in the #summer-of-reproducibility channel
• Check all the details (including FAQ) in the Summer of Reproducibility website: flower.dev/summer

[imtiaz051]

i want to implement figure 2 in the paper

[jafermarq]

Hi @imtiaz051, nice choice of baseline! Figure 2 shows the results of power-of-choice for quadratic optimisation but wouldn't it be more reasonable to reproduce the experiments in Figure 4 (FEMNIST) and Figure 6 (CIFAR-10)? What do you think?

Also, could provide a bit more info about how do you plan to carryout this work ? Maybe you can use as reference the comment provided by other contributors: FedDF(#2012), FedNova(#2011)

[imtiaz051]

Yes OfCourse Implementing the experiments on the FEMNIST dataset will add valuable insights to our research. FEMNIST is a commonly used dataset for federated learning tasks, particularly for character recognition. By conducting experiments on FEMNIST, we can evaluate the performance of the Power-of-Choice framework in the context of this specific dataset.

To carry out the experiments on the FEMNIST dataset using the Power-of-Choice framework, the first step is to acquire the dataset and preprocess it for federated learning. This involves resizing the images, normalizing pixel values, and partitioning the dataset into client nodes while maintaining the original distribution. Next, a suitable model architecture for character recognition is selected, which can be an existing model such as a CNN or RNN, or a custom-designed model specifically tailored to FEMNIST. The Power-of-Choice framework is then implemented, taking into account the details provided in the paper. The client selection process is modified to bias the selection towards clients with higher local loss values. Additionally, the communication and computation strategies are adapted to suit the experimental setup. With the experimental setup in place, the training and evaluation phase begins. The model is trained using the Power-of-Choice framework on the FEMNIST dataset, and the convergence speed is monitored by tracking the loss function or other relevant metrics. The model's performance is evaluated on a separate validation or test set. The experimental results are analyzed to compare the performance of the Power-of-Choice framework with the baseline random selection. Key factors such as convergence speed, solution bias, and test accuracy are assessed to determine the effectiveness of Power-of-Choice. The findings are discussed in terms of the implications of biased client selection in federated learning and the trade-off between convergence speed and solution bias.

[imtiaz051]

My Background I am currently a PhD Electrical Engineering student at sir Syed CASE institute of technology Islamabad Pakistan. It is one of the leading institutes in AI research. I am working on a privacy preserving federated learning

Would love to discuss further and get started.

[andrearestelli]

Hi @jafermarq, is this baseline still available for implementation?

My contribution plan would be:

• Implement power-of-choice client selection strategy, as proposed in the paper. In particular I would start by implementing the standard version, then implement the two variants cpow and rpow useful to reproduce all the experiments.
• Reproduce the experiments in Figure 4 (FMNIST) and Figure 6 (CIFAR-10), as suggested above, by applying the standard strategy to a MLP on FMNIST dataset and the two variants using a CNN on CIFAR-10 dataset.

My background is: I'm a final year Computer Science MSc student at Politecnico di Milano, Italy. I would like to use the results of the implementation for my Master thesis focusing on Federated Learning.

Let me know, in case this baseline has been already assigned, I would choose another one.

Thanks

[imtiaz051]

yes, you can choose this baseline. i have no issue

On Mon, Jul 24, 2023 at 2:20 AM Andrea Restelli @.***> wrote:

Hi @jafermarq https://github.com/jafermarq, is this baseline still available for implementation?

My contribution plan would be:

• Implement power-of-choice client selection strategy, as proposed in the paper. In particular I would start by implementing the standard version, then implement the two variants cpow and rpow useful to reproduce all the experiments.
• Reproduce the experiments in Figure 4 (FMNIST) and Figure 6 (CIFAR-10), as suggested above, by applying the standard strategy to a MLP on FMNIST dataset and the two variants using a CNN on CIFAR-10 dataset.

My background is: I'm a final year Computer Science MSc student at Politecnico di Milano, Italy https://www.polimi.it/en. I would like to use the results of the implementation for my Master thesis focusing on Federated Learning.

Let me know, in case this baseline has been already assigned, I would choose another one.

Thanks

— Reply to this email directly, view it on GitHub https://github.com/adap/flower/issues/2020#issuecomment-1647535596, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRQIL5BVD7LCU47TX7PTZDXRY45LANCNFSM6AAAAAA2ATTZII . You are receiving this because you were mentioned.Message ID: @.***>

[jafermarq]

hi @andrearestelli, yes power-of-choice is available and your contribution plan makes sense! 🙌 With the details you have provided, everything is set and ready for you to move on to Step 3. I have now ✅ all the points in Step 1&2, made you the assignee to this issue and moved it to in progress status. You will find the steps on how to start with the code by following the link in the What happens next? section above in the issue description. If you encounter any issues or if you have questions please reach out to me and others also participating in the Summer of Reproducibility via our Flower Slack.

Looking forward to seeing your power-of-choice in action!