NVIDIA / deepops

Tools for building GPU clusters
BSD 3-Clause "New" or "Revised" License
1.25k stars 326 forks source link

Is ssh into the Enroot container supposed to be passwordless? #1259

Closed stephandooper closed 1 year ago

stephandooper commented 1 year ago

Hi, I've been trying to ssh into a running enroot container on our SLURM cluster via Pyxis/Enroot. I'd like to be able to access the running container, which is running on our cluster, from my local machine, and more importantly, access it using an IDE.

After submitting the job using

srun --ntasks=1 --gpus-per-task=1 --gpus=1 --container-mounts=/data/pathology:/data/pathology --no-container-remap-root --container-image=${CONTAINER_IMAGE} --pty bash

I was able to start the sshd service in the container using /usr/sbin/sshd -p 2222

However, trying to access the enroot container through my local machine on port 2222 yielded:

ssh -p 2222 stephandooper@dlc-togepi
stephandooper@dlc-togepi's password:
Permission denied, please try again.

Looking within the container, I found that the /etc/passwd and /etc/shadow files were marked with an 'x' for the user that submitted the job in the first place, so I can't login using a password.

stephandooper:x:10011:10011:stephandoooper:/home/stephandooper:/bin/bash  (/etc/passwd)
stephandooper:x:19422:0:99999:7:::  (/etc/shadow)

My question is: Is this by design? I was expecting to be able to login with the same username/password as for the controller node.

Eventually, I was able to access the running container by adding ssh keys to the shared nfs folder, but I was wondering whether it was possible to access the containers through password as well.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 60 days with no activity. Please update the issue or it will be closed in 7 days.