Support for installing client sites on HPC systems

[dirkpetersen]

Is your feature request related to a problem? Please describe.

Some organizations have all their GPUs allocated in HPC systems and find it difficult to allocate dedicated GPU servers to NVFlare. Currently the use on HPC systems is undocumented.

Describe the solution you'd like In an ideal world, a client would be installed on a virtual machine which then submits jobs to an HPC system to prevent that the GPU is allocated for long periods of time without being used.

Describe alternatives you've considered I currently use this workaround and describe some of the issues with running on HPC systems (Slurm in this case) https://github.com/dirkpetersen/nvflare-cancer#install-a-client-on-hpc

[YuanTingHsieh]

Hi @dirkpetersen thanks for bringing this topic up!

One workaround as you said is to run a NVFlare client process (client_train.py) directly in your SLURM cluster node. This is the client monitoring process (CP) and it usually just waiting for jobs to come and spawn a job process (CJ) to handle this job. As you observed, this CP will be waiting for jobs and it is not using the GPU. It is the CJ that might be using the GPU, so ideally, we want to run CP outside of the GPU node, and only starts the process to be run in the GPU node when we need to.

We do have several ways to achieve that in HPC cluster.

1. Use 2.4.1:

• Run a NVFlare client process (client_train.py) at the HPC login node (the node that can submit job)
• Prepare a training script (read in from input and writes out result to a file) to be run in slurm GPU node
• Write a custom "SlurmExecutor", extending Executor (https://github.com/NVIDIA/NVFlare/blob/main/nvflare/apis/executor.py) implement "execute" that does the following: (1) writes out the input weights out to file system (2) submit a slurm job "sbatch XXX" with the path to the input weights (3) waits for slurm job to writes out result (4) read in the result and return the result
• You will need to handle Shareable -> model weights -> Shareable serialization
• We do have a bunch of executors already implemented that you can take a look at
• Configure to use SlurmExecutor in the NVFlare job config (config_fed_client)

2. Use main branch:

• Run a NVFlare client process (client_train.py) at the HPC login node (the node that can submit job)
• Prepare a training script to import and use IPCAgent (https://github.com/NVIDIA/NVFlare/tree/main/nvflare/client/ipc)
• Configure to use IPCExchanger (https://github.com/NVIDIA/NVFlare/blob/main/nvflare/app_common/executors/ipc_exchanger.py) in the NVFlare job config (config_fed_client)

As we are seeing more interests, we might write out a whole tutorial/reference implementation on this later.

[dirkpetersen]

Awesome, this is very helpful and I will try that !

[dirkpetersen]

@YuanTingHsieh, it seems that both options require that client_train.py will run on the HPC login node, I think that is a reasonable assumption at least for many life sciences HPC systems that tend to have beefy login nodes and tolerant HPC admins. There are other disciplines where login nodes are guarded more strictly and it may not be allowed to run an agent. For those it would be better to have client_train.py run on a system adjacent to HPC and then submit the job via ssh and sbatch (for example using paramiko) . Perhaps a lower priority right now as I understand that FL use cases are focusing on life sciences right now ?

[YuanTingHsieh]

@dirkpetersen thanks for the discussion!

Yes, as you said, if you have a mechanism to submit the job via ssh and sbatch from machine A to your HPC system. Then you can run the NVFlare client (client_train.py) on that machine A.

Note that, you can also start NVFlare client using the "start.sh" in our startup kits, as you can see we add some restarting mechanisms inside that script as well.