Configure Prometheus to auto discover pods as scraping targets

aliok commented 6 years ago

How to verify:

Provision this PR of the APB
Clone the test app https://github.com/aliok/node-app-with-prometheus-metrics.git
Switch to project running the metrics apb: oc project foo
Deploy the test app: oc new-app -f openshift-template.json
This will deploy the test app with annotations on the service.
Delete the annotation org.aerogear.metrics/plain_endpoint: /metrics from the test app service
Add the same annotation to the test app pod
Check the pod logs: you should see requests coming at /metrics
Scale the test app up (need to add the annotation to new pods manually) and make sure you see log entries as above

psturc commented 6 years ago

👀

psturc commented 6 years ago

@aliok is apb test working for you locally? Prometheus pod is crashing on my testing cluster and I see level=error ts=2018-08-08T11:06:43.349761951Z caller=main.go:582 err="Error loading config couldn't load configuration (--config.file=/etc/prometheus/prometheus.yml): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 85: mapping values are not allowed in this context" in Prometheus pod's log

aliok commented 6 years ago

@psturc I haven't run apb test but I was able to provision the APB on my local cluster many times. let me try apb test

aliok commented 6 years ago

@psturc can you get that file's (prometheus.yml) content into a gist?

aliok commented 6 years ago

@psturc apb test worked for me locally..

UPDATE: NO IT DIDN'T!

psturc commented 6 years ago

@aliok https://gist.github.com/psturc/276fe32a74a200f35888a19caca5235b

aliok commented 6 years ago

@psturc this is the latest output from apb test from my local. I have no prev exp with it.

[aliok@localhost metrics-apb]$ apb test
Running APB image: docker.io/ansibleplaybookbundle/apb-tools:latest
Targetting minishift host: tcp://192.168.42.210:2376
Finished writing dockerfile.
Building APB using tag: [172.30.1.1:5000/openshift/metrics2-apb]
Successfully built APB image: 172.30.1.1:5000/openshift/metrics2-apb
Found image: 172.30.1.1:5000/openshift/metrics2-apb
Successfully deleted sha256:9bc44916d27d3f42e9e8348f3c33c30386c7bec8ff9ccc3153e4b48433dfa4d3
Pushing the image, this could take a minute...
Successfully pushed image: 172.30.1.1:5000/openshift/metrics2-apb
Creating project apb-test-metrics2-apb-u287y
Created project
Creating service account in apb-test-metrics2-apb-u287y
Created service account
Creating role binding for apb-test-metrics2-apb-u287y in apb-test-metrics2-apb-u287y
Created Role Binding
Creating pod with image 172.30.1.1:5000/openshift/metrics2-apb in apb-test-metrics2-apb-u287y
Created Pod
Unable to retrieve test result.
Deleting project apb-test-metrics2-apb-u287y
Project deleted

Does this seem good?

psturc commented 6 years ago

@aliok there was a problem with openshift router on testing slave. It's fixed now

psturc commented 6 years ago

@aliok verified that metrics are showing for each pod

screen shot 2018-08-08 at 13 57 51

aerogearcatalog / metrics-apb

Configure Prometheus to auto discover pods as scraping targets #77