User unable to access their R Studio/network timeouts

ministryofjustice / analytics-platform

Parent repository for the MOJ Analytics Platform

MIT License

14 stars 1 forks source link

User unable to access their R Studio/network timeouts #60

Closed RobinL closed 6 years ago

RobinL commented 6 years ago

User ianaelliott is unable to access his R Studio, even after restarting the pod.

His pod is running, but attempts to access it at http://ianaelliott-rstudio.tools.alpha.mojanalytics.xyz/ timeout

This does not seem related to the network he's using (Quantum), because his R Studio is also inaccessible from MoJ Digital Wifi on a macbook

RobinL commented 6 years ago

This issue appears to be affecting a lot more users - at least another 4 have reported the issue

davidread commented 6 years ago

This has been resolved. There was a problem unidling/restarting R Studio.

We were unable to schedule pods, due to a k8s master node being unhealthy. We fixed it by deleting the node (EC2 instance). The issue went away, and the auto-scaling group recreated the master node without intervention.

xoen commented 6 years ago

We think this may have been caused by one of the kubernetes master stopping working. We terminated that faulty instance and replaces with a new one and things should be back to operational now.

davidread commented 6 years ago

We have fixed the broken master node again and R Studios are working fine. However it points to something underlying issue causing this to happen twice in 24hrs, so I've opened a separate ticket for that: https://github.com/ministryofjustice/analytics-platform/issues/64