Metrics and Monitoring related question

[rakesh561]

What is the best way to monitor/get awx metrics and status deployed on kubernetes and send information to Prometheus ?. Any help would be much appreciated i have been trying on this but not getting much information. I dont see any information in README.

[fosterseth]

you can follow the setup guide here https://docs.ansible.com/automation-controller/4.2.0/html/administration/metrics.html

are there any specific metric you are looking to monitor?

[rakesh561]

I'm looking to monitor more around the postgres database health and also some thing like usage of persistent volume created for postgres because especially if file system gets filled up then its going to cause AWX to have issues like jobs going into pending/spinning state.

[mateuszdrab]

@fosterseth Am I right in thinking there is no support for a ServiceMonitor with the AWX operator?

[erlisb]

Hi @fosterseth,

I think you need to create one manually:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: awx-dev
  name: awx-dev-servicemonitor
  namespace: automate-dev
spec:
  endpoints:
  - interval: 30s
    path: /api/v2/metrics/
    port: http
    scheme: http
    scrapeTimeout: 25s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenSecret:
      key: "token"
      name: "awx-servicemonitor-secret"
  namespaceSelector:
    matchNames:
    - automate-dev
  selector:
    matchLabels:
      app.kubernetes.io/name: awx-dev

[r0b1ndot]

in https://github.com/ansible/awx/pull/12832 there was also anonymous scraping introduced maybe this is also something worth to try.

[mateuszdrab]

in ansible/awx#12832 there was also anonymous scraping introduced maybe this is also something worth to try.

Interesting, except that I can't enable this option ALLOW_METRICS_FOR_ANONYMOUS_USERS in the UI as it doesn't show when I hit edit (it shows in the normal view).

I'll try to see if I can enable it via env variable

[r0b1ndot]

@mateuszdrab i noticed the same, starting helm-chart 1.4.0 you can use the extra_settings to configure the variable. i did this with this helm chart and it worked.

[mateuszdrab]

@reyerdam can you let me know which image version you're using? (image: quay.io/ansible/awx:21.14.0)

I applied the env via the AWX resource as I don't use the helm chart but it's still returning unauthorized on the metrics endpoint.

  web_extra_env: |
    - name: ALLOW_METRICS_FOR_ANONYMOUS_USERS 
      value: "true"

We're talking about metrics of AWX instance, not the operator, right? I notice I have a service deployed for the AWX operator metrics, but no servicemonitor for that either.

Edit: this is what is needed in the AWX resource:

  extra_settings:
    - setting: ALLOW_METRICS_FOR_ANONYMOUS_USERS
      value: "True"

[r0b1ndot]

@mateuszdrab sorry for the late reply. the container image which is used there is:

quay.io/ansible/awx:21.14.0

and as you said with this param you can enable the anonymous mode. and of course this will be just for the metrics api of awx directly. i also wish that there is more prometheus integration natively from the chart. but also as you stated there is no servicemonitor the only prometheus related scrape config is there:

https://github.com/ansible/awx-operator/blob/devel/config/prometheus/monitor.yaml

[mateuszdrab]

@mateuszdrab sorry for the late reply. the container image which is used there is:

quay.io/ansible/awx:21.14.0

and as you said with this param you can enable the anonymous mode. and of course this will be just for the metrics api of awx directly. i also wish that there is more prometheus integration natively from the chart. but also as you stated there is no servicemonitor the only prometheus related scrape config is there:

https://github.com/ansible/awx-operator/blob/devel/config/prometheus/monitor.yaml

No worries, I got it to work but I did realize this was actually also publicly accessible so I got my ingress controller to rewrite requests for the metrics endpoint to 'break' it. I need to figure out how to get authenticated access to work.

Regarding the above servicemonitor, it's for the operator but strangely the helm chart did not deploy it and I don't see options to turn it on. I'll try to manually deploy the manifest and see if I can get some metrics from the operator though it's not something I need to be fair.

[mateuszdrab]

It worked for the operator, though I don't see any dashboards around for it. I'll just keep the metrics in case they're needed. Must admit it would be really nice if the awx operator helm chart could install the servicemonitor and same with the instance

[r0b1ndot]

how can i apply the annotations for the scraping only for the awx-web pod?

  service_annotations: |
    prometheus.io/scrape: "true"
    prometheus.io/path: "/api/v2/metrics/"
    prometheus.io/port: "80"

this won't work since it would also try to scrape the other services which are not offering there metrics via /api/v2/metrics

[dchittibala]

This is my ServiceMonitor definition, that I use for my Prometheus operator to scrape metrics. For anyone still trying to collect metrics for AWX

kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: awx-web
  name: awx-web-monitor
  namespace: awx
spec:
  jobLabel: awx-web-monitor
  endpoints:
    - path: /api/v2/metrics
      port: http
      scheme: http
      authorization:
        type: Bearer
        credentials: 
          key: token
          name: awx-admin-pat
  selector:
    matchLabels:
      app.kubernetes.io/component: awx
      app.kubernetes.io/managed-by: awx-operator

[kevrrnet]

I am using the awx operator 2.19.1, using this version, a metrics endpoint is available in the API: /api/v2/metrics/ that surfaces instantaneous metrics about Tower, which can be consumed by system monitoring software like the open source project Prometheus.

I use the following config, service annotations, config prometheus awx target on the existing prometheus server (assume you have setup the prometheus already on same k8s cluster)

  service_annotations: |
    "prometheus.io/scrape": "true"
    "prometheus.io/path": "/api/v2/metrics"
    "environment": "dev"