search

det-lab / jupyterhub-deploy-kubernetes-jetstream

CDMS JupyterHub deployment on XSEDE Jetstream
0 stars 1 forks source link

Hub down - connection timeout #49

Closed zonca closed 3 years ago

zonca commented 3 years ago

Got a notification at 6am PT that the hub is down, investigating now @pibion

zonca commented 3 years ago

openstack commands hang, web interface of Horizon is unreachable as well, I believe it is an issue on Jetstream

zonca commented 3 years ago

there is a power outage at Indiana University, I subscribed to the Jetstream users mailing list (for some reason I wasn't...), @pibion I suggest you do the same: https://portal.xsede.org/group/xup/user-news/-/news/manage-subscription

They will announce when it is back up.

zonca commented 3 years ago

power came back up over the weekend: https://portal.xsede.org/user-news/-/news/item/12698?controlPanelCategory=portlet_usernews_WAR_usernewsportlet, checking the status of the deployment

zonca commented 3 years ago

the instances are still in openstack but are in SHUTOFF state, booting and checking if Kubernetes comes back up

zonca commented 3 years ago

ok, Kuberentes is up, the hub is up, there are errors mounting the NFS volume, so user sessions won't start

zonca commented 3 years ago

ok, I restarted the cvmfs pod, now the service is operative again @pibion please open another issue if anyone has troubles. I'll also ask Jetstream is next time they can restart our instances.