Spawn Failed: Timeout when trying to create a Default sized instance

[zkromerUCD]

When trying to launch a default sized instance of jupyter hub, I get a timeout error. The event log is included below.

Event log Server requested 2022-01-27T17:02:46.434473Z [Normal] Successfully assigned jhub/jupyter-zkromer to kubejetstream-k8s-node-nf-1 2022-01-27T17:03:37Z [Warning] AttachVolume.Attach failed for volume "pvc-53704015-6aff-43ee-b05c-5ef3bd90e954" : Volume "1ef72334-350a-4e4e-8f8f-9166c479aac9" is reserved, can't finish within the alloted time 2022-01-27T17:04:49Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-zkromer], unattached volumes=[cvmfs-nfs-volume volume-zkromer]: timed out waiting for the condition 2022-01-27T17:09:23Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-zkromer], unattached volumes=[volume-zkromer cvmfs-nfs-volume]: timed out waiting for the condition Spawn failed: Timeout

[zkromerUCD]

It seems that any size instance of JupyterHub is not working. I have currently received timeout errors for bot the Tiny and the Default versions, and will check to see if the Full Node size works

[zkromerUCD]

Got this error when booting up a Full Node, the "0/2 nodes are avalible" errors were because there were too many nodes running at the server at the time. When Amy closed one of the running nodes, the "0/2 nodes are avalible" errors resolved themselves

Server requested 2022-01-27T17:42:32.745703Z [Warning] 0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient memory, 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. 2022-01-27T17:42:32.957425Z [Warning] 0/2 nodes are available: 1 Insufficient cpu, 1 Insufficient memory, 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. 2022-01-27T17:42:46Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-zkromer cvmfs-nfs-volume], unattached volumes=[volume-zkromer cvmfs-nfs-volume]: timed out waiting for the condition 2022-01-27T17:50:43.606650Z [Normal] Successfully assigned jhub/jupyter-zkromer to kubejetstream-k8s-node-nf-1 2022-01-27T17:51:38Z [Warning] AttachVolume.Attach failed for volume "pvc-53704015-6aff-43ee-b05c-5ef3bd90e954" : Volume "1ef72334-350a-4e4e-8f8f-9166c479aac9" is reserved, can't finish within the alloted time Spawn failed: pod/jupyter-zkromer did not start in 600 seconds!

[pibion]

@zonca any thoughts on this? I'm not sure what to make of the unmounted volumes?

[zonca]

It's probably the usual issue with reserved volumes. I will reply here once I have time to look into it.

On Thu, Jan 27, 2022, 14:59 pibion @.***> wrote:

@zonca https://github.com/zonca any thoughts on this? I'm not sure what to make of the unmounted volumes?

— Reply to this email directly, view it on GitHub https://github.com/det-lab/jupyterhub-deploy-kubernetes-jetstream/issues/60#issuecomment-1023718637, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5Q4XURHB6VUZQGT3QJD3UYHE57ANCNFSM5M6SDJOQ . You are receiving this because you were mentioned.Message ID: <det-lab/jupyterhub-deploy-kubernetes-jetstream/issues/60/1023718637@ github.com>

[zonca]

ok @zkromerUCD, I fixed it, unfortunately it is an issue we have never been able to solve, so when it happens, I need to fix it via openstack.

/ref https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/40