[Elektra] Dashboard is down if any glance pod is in pending or crashloop

sapcc / elektra

An opinionated openstack Web UI for consumer self service and operations.

Apache License 2.0

72 stars 28 forks source link

[Elektra] Dashboard is down if any glance pod is in pending or crashloop #1330

Open rajivmucheli opened 3 months ago

rajivmucheli commented 3 months ago

Hi Team,

Due to recent cpu resource crunch in lab regions, i noticed the below error on the Elektra dashboard :

`Elektron::Errors::Request Failed to open TCP connection to glance.monsoon3.svc.kubernetes.qa-de-3.cloud.sap:9292 (Connection refused - connect(2) for "glance.monsoon3.svc.kubernetes.qa-de-3.cloud.sap" port 9292)

There are no further details available. Sorry!`

rajivmucheli commented 3 months ago

FYI:

Screenshot 2024-03-20 at 6 56 36 PM

hgw77 commented 3 months ago

As you said there was a problem that not enough resource where available in K8s. This is a very seldom edge case and I am not sure what we should do here?

rajivmucheli commented 3 months ago

there were instances in the past even when the glance-api pods were in crashloop due to maintenance controller. Was wondering what could support do if this issue occurs during off work hours.

Do we know how or why elektra fetches this message from ? this is generated on the home page (meaning none can access the dashboard or their services from UI) not when clicking server snapshots or images.