FedHarmony

[jafermarq]

FedHarmony

• Title: FedHarmony: Unlearning Scanner Bias with Distributed Data
• Venue: MICCAI 2022
• Link to paper: https://conferences.miccai.org/2022/papers/216-Paper1639.html

Do you want to work on this baseline?

🌻 Check everything about the Summer of Reproducibility on flower.dev/summer

All available baselines are listed in the Summer of Reproducibility Dashboard and also in the GitHub Issues with the summer-of-reproducibility label. The content is the same.

📝 It is advised to complete these steps before your start working on your code. But if you can't wait to implement your baseline with Flower (we totally understand it 😄), please ensure you follow the steps on how to contribute a new baseline.

What follows are the steps 1 & 2 in the Summer of Reproducibility instructions.

1. Join the Summer of Reproducibility program

• [x] Join the Flower Slack and say "hi! 👋" in the channel #summer-of-reproducibility.
• [x] Pick a baseline from our curated list <---------------------------------------- [you are doing this now]

  2. Define the scope of your contribution

• [x] What are you going to reproduce? Add a comment to your issue and tell us about your plan regarding this baseline: what experiments from the paper are you reproducing?, for which datasets ? the more details you provide us with the better !

• [x] Check if you are eligible for a reward.

  As we have to comply with US/EU regulations, we have checked that individual contributors based on these countries or territories are eligible: Australia, Austria, Belgium, Bulgaria, Canada, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Gibraltar, Greece, Hong Kong SAR China, Hungary, India, Ireland, Italy, Japan, Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Romania, Singapore, Slovakia, Slovenia, South Korea, Spain, Sweden, Switzerland, Thailand, United Arab emirates, United Kingdom United States.

  If where you are based is not on the list, please send us an email (summer@flower.dev) letting us know a bit about yourself (where are you currently based?, are you a university student? do you work at a public institution?). Please tell us the baselines you are interested in implementing (i.e. tell us your GitHub issue if you have crated one). We will reach back to you.

• [x] We will discuss with you about your contribution plan, if it sounds like a substantial enough contribution according to the Summer of Reproducibility rules (check our website flower.dev/summer), you'll get the OK from us to start working on your baseline!

What happens next?

• [x] This item will be moved to the In Progress stage by a member of the Flower Team.

• [ ] Follow the instructions for creating a new baseline which will guide you through the process step-by-step.

Is something wrong or not clear ?

• Ask a question directly in your issue.
• Reach out to us via the Flower Slack and ask your question in the #summer-of-reproducibility channel
• Check all the details (including FAQ) in the Summer of Reproducibility website: flower.dev/summer

[Nisarg-P-Patel]

Hi @jafermarq !!

I would like to implement the FedHarmony baseline in the flower framework

This is the contribution plan I'd like to propose:

--> I would like to implement the whole Federated harmonization Procedure -- FedHarmony from steps 1 to steps 5 --> I would like to follow this procedure for Fully Supervised System (with subset of data due to limited computation powers) --> I would try to replicate the Table 1 for (50% of all all site data) possibly for all different methods --> If I can successfully replicate the results of FedHarmony with three different loss then I will definitely try to replicate the results for semi-supervised method for 1-site and 3-site. --> Also, If time permits, I can also replicates results with Ablation A,B, and C.

Let me know your thoughts and give your feedback on what can be done in the given time frame.

Thanks, Nisarg Patel

[Nisarg-P-Patel]

Hello @jafermarq

Any updates on this!!??

[jafermarq]

Hi @Nisarg143, it's great to hear you want to implement FedHarmony! For the Summer of Reproducibility we aim to collaboratively construct a catalog of high quality baselines that reproduce the results in the paper. You mention that you would reproduce results "with subset of data due to limited computation powers". Unfortunately this conflicts with one of the core aims of Summer of Reproducibility. You can take a look at the rules and FAQ on our website: flower.dev/summer

If you currently don't have access to the compute resources needed to replicate this baseline, I'd encourage you to look into the other available baselines and choose one that still is interesting to you (there are some great paper available still!) and requires less computational power to reproduce. I hope this makes sense! Let me know your thoughts

[Nisarg-P-Patel]

Hi @jafermarq,

I agree with your your suggestions and looked into another to-do proposed baselines. I have consider another project #2055 (SSFL +KWS) for the contribution in the SoR 2023.

Thank you for your response.

Regards, Nisarg Patel

[jafermarq]

Ok, thanks for getting back to me about this.

This baseline then is still available for anyone interested.

[18bce133]

Hi @jafermarq I wish to incorporate the FedHarmony baseline into the flower framework.

Here's the contribution plan I'd like to propose: --> My goal is to implement the entire Federated harmonization Procedure, known as FedHarmony, encompassing steps 1 to 5. --> Specifically, I plan to apply FedHarmony to a Fully Supervised System. --> As part of my implementation, I will attempt to reproduce the results presented in Table 1 for various methods. --> Once I successfully replicate the results of FedHarmony using three different loss functions, I will proceed to replicate the results for the semi-supervised method.

[18bce133]

Hi @jafermarq , Any feedback on the proposal ?

[jafermarq]

Hi @18bce133 , good to see you are interested to implement FedHarmony. About your contribution plan: are you going to reproduce the entire Table 1? Do you have access to compute resources comparable to those used in the paper? (in particular note they use a 16GB GPU card, the inputs used for the VGG network are much larger to the typical image sizes in other FL papers). If you have your own desktop machine or maybe a server with GPUs as part of your university/employer then that would do. But other alternatives such as Google Colab might not be suitable for this baseline (and in general for baselines in the Summer of Reproducibility).

Finally, could you please confirm you are based in an eligible territory/country (see the second point in Step 2 above) -- for the up to date list of countries you could check the FAQ in the Summer of Reproducibility website: flower.dev/summer

[18bce133]

Hi @jafermarq, Thank you for your response. Answering your first question regarding Computing power, Yes I have access to Server of my research lab in University, It has Nvidia A10 24Gb graphics card. For the eligibility part I am based in USA. Regarding implementation part, I will replicate Table 1 , including Ablation C so that there is clear cut comparison between Ablation C and FedHarmony MAE. I will also implement FedAvg, FedProx and FedEqual and FedHarmony. If time permits I will also implement, Ablation A and Ablation B. Rest, I will implement whole table 1.

[jafermarq]

Hi @18bce133 , thanks for the additional info. The plan you propose for Table 1 looks good to me. Then everything is set. FedHarmony is yours to implement now 🙌. I have now ✅ all the points in Step 1 & 2 above, added you as assignee to this GitHub issue and moved this baseline to the In Progress stage.

Just as a quick reminder, the Flower Summer of Reproducibility runs until the end of September and by then, contributors should have completed all the stages outlined in our website (flower.dev/summer). Therefore, I encourage you to start as soon as you can. You'll find a link in the What happens Next? section above, that will point you to a guide explaining how to start with the code. Feel free to reach out to me and other contributors in the Flower Slack channel.

Looking forward to seeing FedHarmony in action !!

[jafermarq]

Hi @18bce133 !

This is just a gentle reminder that the Flower Summer of Reproducibility is ending at the end of the month. With just a little more than 3 weeks to go, we are excited to see quite a few baselines well ahead in the process with their respective PRs close to ready. If your PR is already on the list, great !! Please make sure the PR is linked to this issue (you just need to copy the URL of this issue somewhere in the main message of your PR). Ping me when you'd like me to take a look.

Also, make sure you keep an eye:eyes: on the #summer-of-reproducibility channel in the Flower Slack. I’ll announce very soon a new (the third!) round of 1:1 ask-me-anything sessions to help Summer of Reproducibility contributors like yourself to meet the deadline. Please consider booking a time slot if you want to chat with me about your baseline, potential issues you have making your code run, how to open a PR, doubts about what to include in your readme, how to use Hydra configs more effective, etc … all questions are welcome!!