Not able to retrieve prometheus-credentials-PROJECT secret

[jetzlstorfer]

The prometheus-sli-service is not able to fetch the prometheus-credentials-PROJECT secret that is used for an external Prometheus installation as described here: https://keptn.sh/docs/0.7.x/monitoring/prometheus/sli-provider/

The problem seems that the prometheus-sli-service does not have the sufficient rights to query the secret. In the logs below, I've already enhanced the logging to find the root cause:

{"timestamp":"2020-12-14T08:39:01.303323159Z","logLevel":"INFO","message":"Retrieving Prometheus metrics"}
{"timestamp":"2020-12-14T08:39:01.307146226Z","logLevel":"INFO","message":"Checking if external prometheus instance has been defined for project demo"}

{"timestamp":"2020-12-14T08:39:01.31712322Z","logLevel":"INFO","message":"secrets \"prometheus-credentials-demo\" is forbidden: User \"system:serviceaccount:keptn:default\" cannot get resource \"secrets\" in API group \"\" in the namespace \"keptn\""}

{"timestamp":"2020-12-14T08:39:01.317172905Z","logLevel":"INFO","message":"No external prometheus instance defined for project demo. Using default: http://prometheus-service.monitoring.svc.cluster.local:8080"}
{"timestamp":"2020-12-14T08:39:01.317257518Z","logLevel":"INFO","message":"Checking for custom SLI queries"}
{"timestamp":"2020-12-14T08:39:01.370170559Z","logLevel":"ERROR","message":"Failed to get custom queries for project demo"}
{"timestamp":"2020-12-14T08:39:01.37030915Z","logLevel":"ERROR","message":"yaml: line 2: did not find expected key"}
{"level":"warn","ts":1607935141.3703835,"logger":"fallback","caller":"http/transport.go:502","msg":"got an error from receiver fn","error":"yaml: line 2: did not find expected key"}
{"level":"warn","ts":1607935141.3705163,"logger":"fallback","caller":"http/transport.go:594","msg":"error returned from invokeReceiver","error":"yaml: line 2: did not find expected key"}

Potential fixes:

• move the secret from the source code to the deployment.yaml file and make it optional there. That would ease the use of this as no direct access to the secret is needed anymore in the source code.
• add the needed permissions to the serviceAccount that is used for the prometheus-sli-service or create a new one.