SSH "Bad owner or permissions on /home/jovyan/.ssh/config" when starting up JupyterLab

[zkromerUCD]

Whenever I initially log in to JupyterLab, I am unable to use ssh, getting the error: "Bad owner or permissions on /home/jovyan/.ssh/config". I am able to fix this by changing the permissions on the config file (chmod 600 config).

[zonca]

Thanks for reporting this, it's not critical, I'll investigate in the next weeks.

[pibion]

Thanks @zonca . Of the SuperCDMS deployment issues, I'd put this near the top since it adds a barrier to people pushing and pulling their git repositories.

[zonca]

I cannot reproduce this, I wiped my volume and had it recreated, there is no .ssh/config file:

when this happens again can you please give me more details? have you created .ssh/config yourself? Send me the outputs of:

whoami
pwd
ls -l .ssh
df -h

[pibion]

@zonca yes, we do create the .ssh/config ourselves.

@zkromerUCD , could you provide the outputs @zonca has requested?

[zonca]

Is it possible you just create it with the wrong permissions? Do you have docs that explain how to create it somewhere?

[pibion]

@zonca it definitely is possible - but even if I fix the permissions, they're reset when I log out and log back in.

So a working set of credentials looks like:

bash-4.2$ ls -l .ssh
total 16
-rw-r--r-- 1 jovyan users  372 Jul  7 23:47 config
-rw------- 1 jovyan users 1679 Jul  7 23:44 id_rsa
-rw-rw-r-- 1 jovyan users  403 Jul  7 23:44 id_rsa.pub
-rw-rw-r-- 1 jovyan users  838 Jul 13 22:23 known_hosts

And this is what things look like when I re-login:

bash-4.2$ whoami
jovyan
bash-4.2$ pwd
/home/jovyan
bash-4.2$ ls -l .ssh
total 16
-rw-rw-r-- 1 jovyan users  372 Jul  7 23:47 config
-rw-rw---- 1 jovyan users 1679 Jul  7 23:44 id_rsa
-rw-rw-r-- 1 jovyan users  403 Jul  7 23:44 id_rsa.pub
-rw-rw-r-- 1 jovyan users  838 Jul 13 22:23 known_hosts
bash-4.2$ df -h
Filesystem                              Size  Used Avail Use% Mounted on
overlay                                  58G   15G   44G  26% /
tmpfs                                    64M     0   64M   0% /dev
tmpfs                                   7.9G     0  7.9G   0% /sys/fs/cgroup
10.233.46.63:/                          976M  2.0M  958M   1% /cvmfs
/dev/sdf                                9.8G  192M  9.6G   2% /home/jovyan
/dev/sda1                                58G   15G   44G  26% /etc/hosts
shm                                      64M     0   64M   0% /dev/shm
tmpfs                                   7.9G     0  7.9G   0% /proc/acpi
tmpfs                                   7.9G     0  7.9G   0% /proc/scsi
tmpfs                                   7.9G     0  7.9G   0% /sys/firmware
10.233.46.63:/cdms.opensciencegrid.org   20G   15G  5.5G  73% /cvmfs/cdms.opensciencegrid.org
10.233.46.63:/sft.cern.ch                20G   15G  5.5G  73% /cvmfs/sft.cern.ch
10.233.46.63:/geant4.cern.ch             20G   15G  5.5G  73% /cvmfs/geant4.cern.ch

[zonca]

ok, thanks, now I understand the issue and I can reproduce it:

I create files with restricted permissions:

bash-4.2$ ls -l
total 16
-rw------- 1 jovyan users     0 Jul 16 17:55 id_rsa
-rw-r----- 1 jovyan users     0 Jul 16 17:55 id_rsa_2
-rw-r-x--- 1 jovyan users     0 Jul 16 17:56 id_rsa_3

then I kill the container and restart it:

bash-4.2$ ls -l
total 16
-rw-rw---- 1 jovyan users     0 Jul 16 17:55 id_rsa
-rw-rw---- 1 jovyan users     0 Jul 16 17:55 id_rsa_2
-rw-rwx--- 1 jovyan users     0 Jul 16 17:56 id_rsa_3

I have no idea how this can happen, I'm investigating.

In the meantime as a workaround, you can add to your .bashrc:

chmod 600 .ssh/id_rsa .ssh/config

so that every time the shell is initialized the permissions are fixed.

[zonca]

I tested in Openstack, detaching and reattaching volume to instance and checking permissions, that works fine. It must be Kubernetes doing something strange when mounting the volume.

[pibion]

@zonca would it be helpful to have @glass-ships take a look at this now that we have a "develop" instance? I can ask if he's interested.

[zonca]

yes, please, I have no idea how to debug this, it must be some Kubernetes internals. Out of desperation I asked on stack overflow: https://stackoverflow.com/questions/62941935/kubernetes-changing-permissions-of-mounted-volumes If anyone else you know has knowledge of Kubernetes and wants to take a look at that question it would be useful.

[zonca]

I spent some hours debugging this, I think it is not a Kubernetes issue, but it is the JupyterHub image which is applying some permission fixes, however I could not identify exactly where this is happening.

But I have found a better workaround, now each time a user container starts, I automatically run:

chmod 700 .ssh
chmod 600 .ssh/*

So now permissions are fine even for users that haven't implemented any fix in their bashrc. So I consider this issue fixed.

If the same problem still appears or you would like modification to this fix, please reopen the issue.

[zonca]

See https://github.com/det-lab/jupyterhub-deploy-kubernetes-jetstream/commit/b717099038a147c38351feb51fcfb87c24e2f198

[zonca]

@pibion @glass-ships, understood what is the issue with the .ssh folder, see https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/38#issuecomment-665811462