ssh-less kube-openmpi - Githubissues

everpeace commented 6 years ago

I confirmed -mca plm_rsh_agent can realize ssh-less kube-openmpi environment. We can use kubectl exec instead of default ssh.

root@everpeace1-master:/# cat kube-plm-rsh-agent
#! /bin/bash

pod=$1
shift

kubectl exec -i $pod -c mpi-ctr -- $@

root@everpeace1-master:/# cat hostfile
everpeace1-worker-0

root@everpeace1-master:/# mpirun --allow-run-as-root \
  -mca plm_rsh_agent /kube-plm-rsh-agent \
  --display-map -n 1 -npernode 1 --hostfile /hostfile \
  -- hostname
 Data for JOB [32177,1] offset 0

 ========================   JOB MAP   ========================

 Data for node: everpeace-worker-0 Num slots: 2    Max slots: 0    Num procs: 1
        Process OMPI jobid: [32177,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0]]:[B/.]

 =============================================================
everpeace1-worker-0

However, kubectl exec traffic through kube-apiserver during its execution. So, we have to consider performance impact for open-mpi execution.

EdwardZhang88 commented 5 years ago

Hi @everpeace if I use kubectl exec to initialise the MPI processes, will it(kubectl exec) be also used for worker-to-worker communication, say, synced SGD training among multiple GPUs across multiple nodes? If that's the case, then it might have some negative impact on some large model training in a large cluster whose apiServer is supposedly busy all the time. Is my understanding correct?

everpeace commented 5 years ago

I suspected so first. However, plm_rsh_agent seems to be used in initialization. Consequently, worker to worker communication seems to happen pod to pod directly. So, probably, plm_rsh_agent won't cause a performance impact.

ssh less mpi on k8s is used in kubeflow/mpi-operator, kubeflow/chainer-operator

EdwardZhang88 commented 5 years ago

@everpeace Thanks for the reply. After a few days' debugging, now I am pretty sure that kubectl exec as a RSH agent is only used during initialization and it's a good replacement for password-less SSH.

everpeace commented 5 years ago

@EdwardZhang88 by the way, would you often use kube-openmpi to execute your mpi jobs??

Actually, I haven't been active on this project so far, but, PR is always welcome 👍

everpeace / kube-openmpi

ssh-less kube-openmpi #5