No app metrics when using Eirini

[troytop]

Describe the bug Applications deployed to KubeCF with Eirini enabled do not report their CPU and memory usage.

$ cf app 12factor
Showing health and status for app 12factor in org suse / space dev as admin...

name:                12factor
requested state:     started
isolation segment:   placeholder
routes:              12factor.cloud-native.app
last uploaded:       Wed 29 Apr 20:04:35 PDT 2020
stack:               cflinuxfs3
buildpacks:          ruby

type:           web
instances:      2/2
memory usage:   64M
     state     since                  cpu    memory     disk        details
#0   running   2020-04-30T03:57:50Z   0.0%   0 of 64M   0 of 256M   
#1   running   2020-04-30T04:04:39Z   0.0%   0 of 64M   0 of 256M

To Reproduce Deploy KubeCF with Eirini, deploy any app, and observe the metrics with cf app or stratos console.

Expected behavior Expect metrics to appear as they do with Diego.

Environment

• CF Version KubeCF 2.1 (api_version 2.146.0)
• Component/Plugin Name Eirini

[qu1queee]

I saw this behaviour before, if I recalled correctly in the eirini logs we saw issues around the eirini SA not being able to get metrics from the nodes. This is related to the eirini SA not having cluster access via the clusterrole definition

[mook-as]

Current state:

• Need to install the metrics-server addon in minikube.
• The eirini SA seems to be able to get metrics for the nodes (at least, it's not reporting errors). I recall fixing something along these lines before.
• My metrics server is responding with non-zero things:
  pod metrics with CPU & memory

curl --cacert ~/.minikube/ca.crt --cert ~/.minikube/profiles/minikube/client.crt --key ~/.minikube/profiles/minikube/client.key "$(kubectl config view --minify --output=jsonpath='{.clusters[].cluster.server}')/apis/metrics.k8s.io/v1beta1/namespaces/eirini/pods/go-env-test-space-a82380fc86-0"

{
"kind": "PodMetrics",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"name": "go-env-test-space-a82380fc86-0",
"namespace": "eirini",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/eirini/pods/go-env-test-space-a82380fc86-0",
"creationTimestamp": "2020-05-07T19:47:53Z"
},
"timestamp": "2020-05-07T19:47:00Z",
"window": "1m0s",
"containers": [
{
"name": "opi",
"usage": {
"cpu": "1011m",
"memory": "10008Ki"
}
}
]
}

[troytop]

To be clear, I'm able to get to the app metrics via Kubernetes, just not via cf app <appname> or the Applications view in Stratos (via stratos-metrics).