lens-metrics breaks if original node of prometheus-0 pod is shut down

[cerebrate]

The lens-metrics/prometheus-0 pod cannot be scheduled on any node other than that which it was first scheduled on. This appears to be because of the Persistent Volume Claim for the data volume.

Steps to reproduce the behavior:

1. Cordon the node on which the prometheus-0 pod is running.
2. Drain the node on which the prometheus-0 pod is running.
3. Observe events of the form:

0/5 nodes are available: 1 Insufficient memory, 1 node(s) were unschedulable, 3 node(s) had volume node affinity conflict.

And that the pod remains in a perpetual pending state.

Expected behavior

The prometheus-0 pod to be rescheduled and run successfully on another node.

Environment (please complete the following information):

• Lens Version: 3.6.7
• OS: Windows 10, build 20236; cluster is running on Raspbian Buster.

Kubeconfig: Quite often the problems are caused by malformed kubeconfig which the application tries to load. Please share your kubeconfig, remember to remove any secret and sensitive information.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [REDACTED]
    server: https://twilight-sparkle.arkane-systems.lan:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: [REDACTED]
    client-key-data: [REDACTED]

[Nokel81]

I assume this is still a problem in 4.1.4?

[cerebrate]

Presumably - unfortunately, since opening this issue, my cluster's been upgraded to use prometheus-operator based metrics, so I can't confirm that at this instant. I'll update this when I can find time and hardware to put a suitable test setup together.

[cerebrate]

Found the time; yes, it is. Looks like the issue is the local PV it uses, to me.