[NERSC] ssh doesn't work for me in `desc-stack-weekly`

[wmwv]

Description A clear and concise description of what the issue is.

ssh doesn't work for me in the desc-stack-weekly environment loaded from command line.

If I use start-kernel-cli.py to load desc-stack-weekly, and then try to ssh to another node (to, e.g., monitor load, GPU usage...), I get

[login25 ~] start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
ssh lUnable to find an acceptable version of lsst_sims
[login25 ~] ssh login28
ssh: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5.so.3)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libk5crypto.so.3)
ssh: /lib64/libselinux.so.1: no version information available (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libcrypto.so.1.1)

Choose all applicable topics by placing an 'X' between the [ ]:

• [ ] jupyter
• [ ] jupyter terminal
• [x] Cori interactive command line
• [ ] Batch jobs
• [ ] python
• [ ] CSCRATCH
• [ ] Community File System
• [ ] HPSS (tape)
• [ ] Data transfer and Globus
• [ ] New User Account or account access problems

To Reproduce

[login25 ~] start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
ssh lUnable to find an acceptable version of lsst_sims
[login25 ~] ssh login28
ssh: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5.so.3)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libk5crypto.so.3)
ssh: /lib64/libselinux.so.1: no version information available (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libcrypto.so.1.1)

[heather999]

Hi @wmwv desc-stack-weekly is a shifter container without ssh. The primary purpose of the containers has been to provide a runtime environment rather than a full featured virtual machine. I'm not sure we need ssh from within the containers themselves (but could be coaxed into considering it). Can you just ssh to a NERSC login node from another terminal? I usually use NX when I connect to NERSC, where it's not a big deal to just open another terminal. Take care, Heather

[wmwv]

Sure. Won't fix is a fine answer in practice.

I think it was confusing that it triggered a GLIBC version mismatch. I thought loading the image with shifter would completely override existing paths.

[wmwv]

FWIW, the image does have ssh.

[login22 ~] start-kernel-cli.py desc-stack-weekly
[login22 ~] whereis ssh
ssh: /usr/bin/ssh /etc/ssh /opt/udiImage/bin/ssh

So /usr/bin/ssh remains in the path, but the GLIBC is no longer for correct for it. Here's the tail of the path (after all of the /opt/lsst) entries:

/opt/udiImage/modules/gpu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/udiImage/bin

If you explicitly run /opt/udiImage/bin/ssh, that works fine.

[github-actions[bot]]

This issue is stale because it has been 90 days since last activity. If no further activities take place, this issue will be closed in 14 days.
Add label keep to keep this issue.