swarm-learning-hpe

Swarm learning based on HPE platform, experiments performed based on HPE Swarm Learning version number 2.2.0

This repository contains:

1. SWARM Learning For Histopathology Image Analysis and Radiology Image Analysis
2. Work flow to help keep track of what's under process.
3. Issue section where people can dump ideas and raise questions encountered when using this repo.
4. Working version of marugoto_mri for Attention MIL based model, originally suitable for histopathology images but Marta has modified it to work with MRI images.
5. Working version of odelia-breast-mri for 3D-CNN model by @Gustav.

Table of Contents

• Background
• Install
• Usage
• Maintainers
• Milestone
• NotionPage
• Contributing
• License

Background

Brief description about HPE platform

Course of Swarm Leaning explained in a generally understandable way: https://learn.software.hpe.com/swarm-learning-essentials

HPE Swarm Learning extends the concept of federated learning to decentralized learning by adding functionality that obviates the need for a central leader. It combines the use of AI, edge computing, and blockchain.

HPE Swarm Learning is a decentralized, privacy-preserving Machine Learning (ML) framework. Swarm Learning framework uses the computing power at, or near, the distributed data sources to run the ML algorithms that train the models. It uses the security of a blockchain platform to share learning with peers safely and securely. In Swarm Learning, training of the model occurs at the edge, where data is most recent, and where prompt, data-driven decisions are mostly necessary. In this decentralized architecture, only the insights learned are shared with the collaborating ML peers, not the raw data. This tremendously enhances data security and privacy.

The following image provides an overview of the Swarm Learning framework. Only the model parameters (learnings) are exchanged between the various edge nodes and not the raw data. This ensures that the privacy of data is preserved.

This is the Swarm Learning framework:

Install

Prerequisites

Hardware recommendations

• 64 GB of RAM (32 GB is the absolute minimum)
• 16 CPU cores (8 is the absolute minimum)
• an NVIDIA GPU with 48 GB of RAM (24 is the minimum)
• 8 TB of Storage (4 TB is the absolute minimum)
• We deliberately want to show that we can work with lightweight hardware like this. Here are three quotes for systems like this for less than 10k EUR (Lambda, Dell Precision, and Dell Alienware)
• Typical installation time can take 30 minutes to build up the necessary dependencies and another 1 hour to build up the environment for running demo experiments

Operating System

• Ubuntu 20.04 LTS
  • We have tested the Swarm Learning Environment on [Ubuntu 20.04 LTS, Ubuntu 22.04.2 LTS, Ubuntu 20.04.5 LTS] and they work fine.
  • Any experimental release of Ubuntu greater than LTS 20.04 MAY result in unsuccessful swop node running.
  • It also works on WSL2(Ubuntu 20.04.2 LTS) on Windows systems. WSL1 may have some issues with the docker service.

Upgrade the Swarm Learning Environment from Older Version

1. Run the following command to upgrade the Swarm Learning Environment from 1.x.x to 2.x.x

   sh workspace/automate_scripts/server_setup/cleanup_old_sl.sh

   Then proceed to 1. Prerequisite in Setting up the Swarm Learning Environment

Setting up the user and repository

1. Create a user named "swarm" and add it to the sudoers group. Login with user "swarm".

   sudo adduser swarm
   sudo usermod -aG sudo swarm
   sudo su - swarm

2. Add the Docker user to the sudoers group

   sudo usermod -aG docker swarm

   After running this command, you will need to log out and log back in for the changes to take effect, or you can use the newgrp command like so:

   newgrp docker

3. Run the following commands to set up the repository:

cd / && sudo mkdir opt/hpe && cd opt/hpe && sudo chmod 777 -R /opt/hpe
git clone https://github.com/KatherLab/swarm-learning-hpe.git && cd swarm-learning-hpe

4. Install cuda environment and nvidia drivers, as soon as you could see correct outputs of the following command you may proceed.

   nvidia-smi

   Please disable secure boot. On some systems, Secure Boot might prevent unsigned kernel modules (like NVIDIA's) from loading. Check Loaded Kernel Modules:

   • To see if the NVIDIA kernel module is loaded:

     lsmod | grep nvidia

     Review System Logs:

   • Sometimes, system logs can provide insights into any issues with the GPU or driver:

     dmesg | grep -i nvidia

     Manually Load the NVIDIA Module:

   • You can try manually loading the NVIDIA kernel module using the modprobe command:

     sudo modprobe nvidia

     Requirements and dependencies will be automatically installed by the script mentioned in the following section.

Setting up the Swarm Learning Environment

PLEASE REPLACE THE <PLACEHOLDER> WITH THE CORRESPONDING VALUE!

<sentinel_ip> = 172.24.4.67 currently it's the IP assigned by VPN server for TUD host.

<host_index> = Your institute's name. For ODELIA project should be chosen from TUD Ribera VHIO Radboud UKA Utrecht Mitera Cambridge Zurich

<workspace_name> = The name of the workspace you want to work on. You can find available modules under workspace/ folder. Each module corresonds to a radiology model. Currently we suggest to use odelia-breast-mri or marogoto_mri here.

Please only proceed to the next step by observing "... is done successfully" from the log

0. Optional: download preprocessed datasets. Please refer to the Data Preparation section for more details.

1. Prerequisite: Runs scripts that check for required software and open/exposed ports.

   sh workspace/automate_scripts/automate.sh -a

2. Server setup: Runs scripts that set up the swarm learning environment on a server.

   sh workspace/automate_scripts/automate.sh -b -s <sentinel_ip> -d <host_index>

3. Final setup: Runs scripts that finalize the setup of the swarm learning environment. Only <> is required. The [-n num_peers] and [-e num_epochs] flags are optional.

   sh workspace/automate_scripts/automate.sh -c -w <workspace_name> -s <sentinel_ip> -d <host_index> [-n num_peers] [-e num_epochs]

Optional 5. Reconnect to VPN

sh /workspace/automate_scripts/server_setup/setup_vpntunnel.sh

In case your machine got restarted or lost the vpn connection. Here is the guide to reconnect: VPN connect guide The file.ovpn is the config file that TUD assigned to you.

If a problem is encountered, please observe this README.md file for step-by-step setup. Specific instructions are given about how to run the commands. All the processes are automated, so you can just run the above command and wait for the process to finish.

If any problem occurs, please first try to figure out which step is going wrong, try to google for solutions and find solution in Troubleshooting.md. Then contact the maintainer of the Swarm Learning Environment and document the error in the Troubleshooting.md file.

Usage

Ensuring Dataset Structure

To ensure proper organization of your dataset, please follow the steps outlined below:

1. Directory Location

   Place your dataset under the specified path:

/workspace/odelia-breast-mri/user/data-and-scratch/data

Within this path, create a folder named multi_ext. Your directory structure should then resemble: /opt/hpe/swarm-learning-hpe/workspace/odelia-breast-mri/user/data-and-scratch/data └── multi_ext ├── datasheet.csv # Your clinical tabular data ├── test # External validation dataset ├── train_val # Your own site training data └── segmentation_metadata_unilateral.csv # External validation table

2. Data Organization

Inside the train_val or test directories, place folders that directly contain NIfTI files. The folders should be named according to the following convention: