search

det-lab / jupyterhub-deploy-kubernetes-jetstream

CDMS JupyterHub deployment on XSEDE Jetstream
0 stars 1 forks source link

JupyterHub boot up timeout on Tiny server configuration #52

Closed zkromerUCD closed 3 years ago

zkromerUCD commented 3 years ago

I received a timeout error while booting up JupyterHub on the Tiny server configuration. I had previously received the same timeout error when booting up on the Default server configuration, and then decided to retry with a Tiny server configuration.

Here is the event log for the Tiny configuration:

Server requested
2021-02-25 00:15:24+00:00 [Normal] Successfully assigned jhub/jupyter-zkromer to zonca-k8s-master-1
2021-02-25 00:15:27+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r6a6a768faa154a8583abc7c98ca0be7b.scope mount.nfs: Operation not permitted
2021-02-25 00:15:30+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r724eb5e9a9794a3994f3564ef0e04832.scope mount.nfs: Operation not permitted
2021-02-25 00:15:33+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r7f54813ea3f249bd8e4d1e4d12eca05b.scope mount.nfs: Operation not permitted
2021-02-25 00:15:36+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r1b65550e5a7649f89ca73052fa5fd8a1.scope mount.nfs: Operation not permitted
2021-02-25 00:15:41+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-rfc770abe56ee49df86fb77fe7106cfb4.scope mount.nfs: Operation not permitted
2021-02-25 00:15:51+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r084be4dee5284884b974cb8a0de309db.scope mount.nfs: Operation not permitted
2021-02-25 00:16:01+00:00 [Normal] AttachVolume.Attach succeeded for volume "pvc-666b461d-e7d0-43f7-8747-44483d7b19a8"
2021-02-25 00:16:10+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r308a0c45b6df4c4fa77e087b71e1df91.scope mount.nfs: Operation not permitted
2021-02-25 00:16:43+00:00 [Warning] MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r0c099fa65d754c779938401dbc1fa3e2.scope mount.nfs: Operation not permitted
2021-02-25 00:17:27+00:00 [Warning] Unable to attach or mount volumes: unmounted volumes=[cvmfs-nfs-volume], unattached volumes=[volume-zkromer cvmfs-nfs-volume]: timed out waiting for the condition
2021-02-25 00:17:49+00:00 [Warning] (combined from similar events): MountVolume.SetUp failed for volume "cvmfs-nfs-volume" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume --scope -- mount -t nfs 10.233.46.63:/ /var/lib/kubelet/pods/bb9c71a5-6a02-4f6f-b83f-7015be5549b4/volumes/kubernetes.io~nfs/cvmfs-nfs-volume Output: Running scope as unit: run-r35c2a8d4c7bb40b1bdf8d111076efd07.scope mount.nfs: Operation not permitted
Spawn failed: pod/jupyter-zkromer did not start in 600 seconds!
zonca commented 3 years ago

ok, I'm working on it, there was 1 volume that was stuck in detaching state, I reported the error to the Jetstream admins. Now I am trying to put CVMFS back up.

zonca commented 3 years ago

ok, it's back, please reopen the issue if still doesn't work.

zkromerUCD commented 3 years ago

Sorry for the late response, but I am still getting a timeout error while trying to start a Tiny server.

Here is the event log:


2021-03-04T00:05:34.154658Z [Normal] Successfully assigned jhub/jupyter-zkromer to kubejetstream-k8s-node-nf-1
2021-03-04T00:05:34Z [Warning] AttachVolume.Attach failed for volume "pvc-666b461d-e7d0-43f7-8747-44483d7b19a8" : failed to find object
2021-03-04T00:07:37Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-zkromer], unattached volumes=[volume-zkromer cvmfs-nfs-volume]: timed out waiting for the condition
2021-03-04T00:12:09Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-zkromer], unattached volumes=[cvmfs-nfs-volume volume-zkromer]: timed out waiting for the condition
Spawn failed: Timeout
zonca commented 3 years ago

can you try again now? I don't see errors on my side

zkromerUCD commented 3 years ago

I still seem to be getting an error. Loading stalls starting with the message 2021-03-04T00:32:19Z [Warning] AttachVolume.Attach failed for volume "pvc-666b461d-e7d0-43f7-8747-44483d7b19a8" : failed to find object

I will try another sized configuration to see if that works, however.

zonca commented 3 years ago

sorry, your volume disappeared from openstack, so your files are lost. my bad I must have messed up a step in the transfer, some volumes are lost, I'll investigate further.

So you should logout and then log back in

zonca commented 3 years ago

ok, it should work now, but all your data are lost. apologies.

zkromerUCD commented 3 years ago

No problem! Thanks for fixing it!