Support for creation of AWX instance servicemonitor

[David-Igou]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX Operator is open source software provided for free and that I might not receive a timely response.

Feature Summary

Current state:

Many users deploy AWX on Kubernetes clusters that are also running the Prometheus Operator. Currently, it's pretty straight forward to create a servicemonitor for the awx-operator metrics but not the metrics for the instance of awx itself (https://awx.domain/api/v2/metrics/)

Currently, the most straight forward way is to set ALLOW_METRICS_FOR_ANONYMOUS_USERS to True, and creating the following servicemonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: awx-servicemonitor
  namespace: awx
spec:
  endpoints:
  - interval: 30s
    path: /api/v2/metrics/
    port: http
    scheme: http
    scrapeTimeout: 25s
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - awx
  selector:
    matchLabels:
      app.kubernetes.io/name: awx

This works, but now your metrics are available to anybody that can hit your AWX web endpoint. This isn't practical for many users. It can by danced around by configuring your ingress object to rewrite or "break" metrics requests (per https://github.com/ansible/awx-operator/issues/938#issuecomment-1488150387)

So the next option requires creating an oauth2 token, placing it into a secret, and then making a servicemonitor that references the secret so it can authenticate. This works, but is a post-deployment lift, and also might not be applicable in some organizations that have to have token expirations?

Proposed solution:

It would be nice if the operator managed the servicemonitor through configuration of the crd, ie:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  servicemonitor: true

Based on my research, I'm aware it isn't this straight forward. But I think it's valuable to have something this simple as the goal, as it's how many operators and helm charts work.

If it were possible for unauthenticated awx metrics to have their own dedicated kubernetes service, this would be the easiest solution.

[smrowley]

Another option would be to have a variable for configuring the metrics endpoint port. This would allow for metrics to be exposed separately from the main API for security. It wouldn't require elevated privileges, and could easily be blocked from external access.

[goldyfruit]

Agree that it should be handled by the operator. Here is how I'm able to make it works with a Bearer token created from the AWX UI.

echo -n "PcKs9RLQw5kEnoZm5YcryEnmqcW4zR" | base64

apiVersion: v1
kind: Secret
metadata:
  name: awx-prometheus
  namespace: awx
type: opaque
data:
  secret: UGNLczlSTFF3NWtFbm9abTVZY3J5RW5tcWNXNHpS

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: awx
  namespace: awx
spec:
  endpoints:
  - interval: 30s
    path: /api/v2/metrics
    port: http
    scheme: http
    scrapeTimeout: 25s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenSecret:
      name: awx-prometheus
      key: secret 
  namespaceSelector:
    matchNames:
    - awx
  selector:
    matchLabels:
      app.kubernetes.io/part-of: awx

I hope it helps.

[Karif]

Is it possible to create the bearer token on the AWX end as code also?