Prometheus metrics are empty with prometheus-scrape-config rev 58

[theoctober19th]

When the bundle is deployed on top of GKE (Google Kubernetes Engine) and EKS (Elastic Kubernetes Service) cluster, all the UATs pass except one -- test_spark_metrics_in_prometheus.

The test_spark_metrics_in_prometheus fails because somehow the metrics data is not populated to Prometheus. The list returned by the query to http://127.0.0.1:19090/api/v1/query?query=push_time_seconds%5B20m%5D is always empty. Need to investigate this further.

Steps to replicate:

1. Provision a GKE cluster, a VM and bootstrap Juju controller in it. (Terraform script available here )
2. Deploy Charmed Spark bundle on top of it.
3. Run the UAT tests on top of the bundle.

   UUID=$(uuidgen)
   tox run -e integration-sparkjob --  --backend terraform --storage-backend s3 --keep-models --model spark --uuid $UUID

[syncronize-issues-to-jira[bot]]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6930.

This message was autogenerated

[Batalex]

We found a temp fix by pinning the prometheus-scrape-config charm to rev 51. Rev 58 was promoted to stable mid integration pulse, which is why I got the UATs green on AKS but we could not on EKS and GKE