Distinguish between ISPyB being available and the requests not succeeding

[alanbchristie]

The Fragalysis Stack determines whether ISPyB is available by simply creating an SSH tunnel. If this works ISPyB is considered "available". Unfortunately the presence of a Tunnel is not sufficient to determine wether the ISPyB database is responding to requests. To do this we actually need to (continually) make requests and time the response.

This service could run as a separate container/Pod that could offer a REST endpoint that the stack could query to get the following status: -

• OFFLINE (no connection)
• ONLINE (meaning online and request responses are 'fast')
• DEGRADED (messing requests are slow)

[mwinokan]

At the moment each user's frontend is polling the for ISPyB status causing a lot of queries, @kaliif is moving the queries to the backend to reduce traffic

@kaliif has this figured out, just needs to port existing services (Celery). All the queries are there, just need rewriting.

[alanbchristie]

We have introduced metrics to monitor SSH Tunnels and ISPyB (MySQL) connection separately, including the cache of collected proposals (hits and misses). A dashboard for this has been deployed and has been stored in the new b/e grafana...

So we can now see cache performance and SSH tunnel and MySQL failure behaviour.

These charts show no failures.

[mwinokan]

@alanbchristie also please confirm if this has made its way into production