Open ThomasA opened 1 year ago
I've looked around on the current cluster and indeed, there seem to be some differences on their respective sssd.conf
files - these files are managed by another ansible project, so I'm not sure how to fix this down the road - but for now we will simply have to change these deployed files by hand
We should keep this issue open for a long-term solution
It works for me now. Maybe we should document what the fix is, so it can eventually be included in the Ansible project that is responsible for that part of the set-up.
I believe it is considered "normal" Slurm behaviour to let users log into compute nodes via SSH while they have a job running on the node in question. This should work:
Yet, it only works for me on a256-t4-[02-03].srv.aau.dk and nv-ai-04.srv.aau.dk. All the other compute nodes cut me off when I try this:
Something in the configuration on the latter nodes must differ in a way that makes this not work.
We need to figure out what the problem is and fix it. SSH to compute nodes during running jobs is necessary for example to inspect GPU utilisation via
nvidia-smi
.