Reconciliation for install of Application Monitoring

[david-martin]

Verify steps

oc new-project middleware-monitoring
oc create -f ./deploy/roles/
oc create -f ./deploy/crds
make code/run

The middleware-monitoring namespace should eventually reconcile with below pods:

oc get po
NAME                                    READY     STATUS    RESTARTS   AGE
alertmanager-application-monitoring-0   2/2       Running   0          32m
grafana-operator-fb565d8cc-n7fwv        1/1       Running   7          28m
prometheus-application-monitoring-0     3/3       Running   1          17m
prometheus-operator-6c7b4fb6c4-xkwv7    1/1       Running   0          32m

NOTE There should also be a grafana pod, but there is an error in the grafana operator logs at present (see comment below for more info)

TODO

• [x] Add cluster roles & rolebindings required by dependent operators
• [x] Add crds for dependent operators
• [x] Install Grafana Operator & CR
• [x] Install Prometheus Operator & CRs
• (in a follow up PR) Configure Grafana, Alertmanager & Prometheus for monitoring other namepsaces

[david-martin]

@pb82 for thoughts on this error in the grafana operator. The operator is using the image quay.io/integreatly/grafana-operator:0.0.1

{"level":"error","ts":1548454449.445631,"logger":"kubebuilder.controller","caller":"<autogenerated>:1","msg":"Reconciler error","Controller":"grafana-controller","Request":"middleware-monitoring/application-monitoring","error":"open ./templates/grafana-serviceaccount.yaml: no such file or directory","stacktrace":"github.com/integr8ly/grafana-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/integr8ly/grafana-operator/vendor/sigs.k8s.io/controller-runtime/pkg/runtime/log.(*DelegatingLogger).Error\n\t<autogenerated>:1\ngithub.com/integr8ly/grafana-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/integr8ly/grafana-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/peter/work/go/src/github.com/integr8ly/grafana-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Looks like files are in the right place in the image though

/ $ ls -la /usr/local/bin/templates/
total 32
drwxr-xr-x    2 root     root           246 Jan 25 10:44 .
drwxr-xr-x    1 root     root            30 Jan 25 10:44 ..
-rw-r--r--    1 root     root          1170 Jan 25 10:43 grafana-config.yaml
-rw-r--r--    1 root     root           123 Jan 25 10:43 grafana-dashboards.yaml
-rw-r--r--    1 root     root           562 Jan 25 10:43 grafana-datasources.yaml
-rw-r--r--    1 root     root          2399 Jan 25 10:43 grafana-deployment.yaml
-rw-r--r--    1 root     root           299 Jan 25 10:43 grafana-providers.yaml
-rw-r--r--    1 root     root           267 Jan 25 10:43 grafana-route.yaml
-rw-r--r--    1 root     root           271 Jan 25 10:43 grafana-service.yaml
-rw-r--r--    1 root     root           132 Jan 25 10:43 grafana-serviceaccount.yaml

[pb82]

@david-martin Yes, i've seen this before. It could be that the workdir of the container is not the one where the binary is located. I've updated the operator deployment to set an explicit template path: https://github.com/integr8ly/grafana-operator/blob/master/deploy/operator.yaml#L34. That should fix it.

[david-martin]

superceded by #3