Extension of flexibility in federated learning scenarios

[yanghaojin]

Background Information: My team and I (from Hasso Plattner Institute) will make a new MOOC lecture on the https://ki-campus.org/ platform. I want to use Sedna to design two practical tasks in my lecture: federated learning and cloud-edge collaborative inference. Given our current practice environment, we hope to get extended support from Sedna.

Why is this needed: Based on our current scenario settings, it is impossible to provide a suitable environment completely on the cloud for thousands of students. Therefore, it is hoped that Sedna can support the independent operation of the lib library and support the dynamic access of federated learning workers.

Value for customer: Germany and the EU have extremely strict protection of data privacy, and Federal Learning can be used for developing machine learning applications under the premise of protecting privacy and can improve the overall accuracy for all parties.

Value for Sedna: MOOC online lecture usually has a large number of participants. Including not only students but also a large number of practitioners from the industry. Therefore, Sedna can be well promoted through the course. Enhance its industry influence. In addition, the dynamic joining of edge workers is a very promising feature for other scenarios as well, which can be designed as a general capability for Sedna federation learning.

[JoeyHwong-gk]

GM/LC shield lots of things that algorithm developers don't need to pay much attention to. It is feasible to run the sedna lib library independently. However, problems such as network engineering (for example, you need to open ports) and log monitoring (for locating exceptions) may be introduced, and I'm not sure it's up to the community to block.

What my suggestion is, Sedna can provide an image with development status (like a notebook) that integrates relevant components and environments. Students only need to pull and run the image to connect to the sedna service (like aggregate worker) that teachers have deployed on the cloud in advance.

[yanghaojin]

Thank you for the comments! I want to explain further. Due to the scale of the course (up to 10k participants) and the control of the learning quality (maybe most people do not have enough computing power for DL model training at home), we will ask students to use Kaggle notebooks to complete practical assignments. Kaggle notebook can provide all participants with a free GPU quota to complete tasks. However, the Kaggle notebook cannot be used with the current Sedna. We hope to use Sedna to do the federal learning task with up to 10k clients in this course, and we also hope to promote Sedna in Germany.

[yanghaojin]

Another limitation of the current Sedna is as mentioned in my first post. We cannot ask up to 10k clients to start the federated learning task at roughly the same time, due to its large number and the heterogeneous distribution regarding time and geography. So dynamic and asynchronous federated learning interactions are also needed.