det-lab / jupyterhub-deploy-kubernetes-jetstream

CDMS JupyterHub deployment on XSEDE Jetstream
0 stars 1 forks source link

Error when logging in for first time #86

Closed JJAmidei closed 1 year ago

JJAmidei commented 1 year ago

2023-03-22T17:20:30Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-jamesjamidei], unattached volumes=[volume-jamesjamidei cvmfs-nfs-volume]: timed out waiting for the condition

JJAmidei commented 1 year ago

2023-03-22T17:30:27Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-jamesjamidei], unattached volumes=[volume-jamesjamidei cvmfs-nfs-volume]: timed out waiting for the condition

This is what happened the second time I tried logging in.

zonca commented 1 year ago

thanks @JJAmidei, I get the same error. It seems like there is an authorization error between Kubernetes and Openstack:

  Warning  FailedAttachVolume  98s   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6e9eb56a-82dc-4960-8bfe-8ee66db83521" : rpc error: code = Internal desc = [ControllerPublishVolume] get volume failed with error Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://js2.jetstream-cloud.org:8776/v3/37193e7566414ba2b209e8e249242f34/volumes/886261f2-fb84-4ce5-93bb-a4d32fbda76b], but got 401 instead
{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}

I need to investigate, I'll update the issue when the deployment is usable again.

zonca commented 1 year ago

the credentials which are in Kubernetes to interface with Openstack have expired. I asked the Jetstream support if they can extend them. Otherwise I will try to modify the external-openstack-cloud-config secret in the kube-system namespace and add newly created credentials.

zonca commented 1 year ago

ok @JJAmidei it should be fixed now, please reopen if you still have issues

zonca commented 1 year ago

wrote the procedure here, in case it is useful to someone else: https://www.zonca.dev/posts/2023-03-23-update-openstack-credentials-kubernetes

@julienchastang