Support ServiceMonitor for deployed applications

[arthurdm]

We need a key under service (perhaps service.enableMonitoring) where under the covers our operator creates the ServiceMonitor that the Prometheus Operator is watching.

[navidsh]

With the following resource definitions, Prometheus operator was able to scrape metrics from Liberty through HTTPS with basic user registry enabled. Liberty used self-signed certificates.

ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: myapp-monitor
  name: myapp-monitor
  namespace: appsody-app
spec:
  endpoints:
    - basicAuth:
        password:
          key: password
          name: appsody-app
        username:
          key: username
          name: appsody-app
      interval: 5s
      path: /metrics
      scheme: https
      targetPort: 9443
      tlsConfig:
        insecureSkipVerify: true
  namespaceSelector:
    matchNames:
      - appsody-app
  selector:
    matchLabels:
      app.kubernetes.io/name: metrics

AppsodyApplication:

apiVersion: appsody.dev/v1beta1
kind: AppsodyApplication
metadata:
  name: metrics
  namespace: appsody-app
spec:
  applicationImage: 'navidsh/metrics:latest'
  expose: true
  livenessProbe:
    failureThreshold: 30
    httpGet:
      path: /health/live
      port: 9443
      scheme: HTTPS
    initialDelaySeconds: 30
    periodSeconds: 5
  pullPolicy: Always
  readinessProbe:
    failureThreshold: 30
    httpGet:
      path: /health/ready
      port: 9443
      scheme: HTTPS
    initialDelaySeconds: 30
    periodSeconds: 5
  resourceConstraints:
    requests:
      memory: 512Mi
  service:
    annotations:
      prometheus.io/scrape: 'true'
    port: 9443
    type: NodePort
  stack: java-microprofile

Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: appsody-monitoring
  labels:
    prometheus: prometheus
spec:
  enableAdminAPI: false
  resources:
    requests:
      memory: 400Mi
  serviceAccountName: prometheus
  serviceMonitorNamespaceSelector:
    matchLabels:
      prometheus: monitoring
  serviceMonitorSelector:
    matchExpressions:
      - key: k8s-app
        operator: Exists

• appsody-app namespace was labelled with prometheus: monitoring
• Prometheus namespace's network should be global: oc adm pod-network make-projects-global appsody-monitoring

[navidsh]

I think we should consider the following when designing this:

• bearerTokenFile vs basicAuth
• Certificate validation through TLSConfig when not using a self-signed cert
• Labeling ServiceMonitor

[arthurdm]

Great work @navidsh - thanks. So we still need the prometheus.io/scrape annotation, even with ServiceMonitor?

[arturdzm]

Nope no need for annotation, we will need to figure out the route, when 9443 is used the expose won't work correctly, as Edge termination policy assumes HTTP connection to pod

[navidsh]

prometheus.io/scrape is added by the Appsody stack. Once we implement this issue, we can open an issue against stacks to remove this.

[arturdzm]

We will need a design for how to provide all this info, eg basic auth, prometheus labels, etc to SM.

[arthurdm]

Need to confirm the following:

1. for MicroProfile / Java, SpringBoot / Java, and Node.js (express & loopback), what is the default metrics endpoint and does it require auth and/or https.

2. Based on that, specify min config for operator CR.

3. Finally, agree on how to configure this (top annotations, vs new fields, vs new field + annotations)

[arturdzm]

• Node.JS - Express, HTTP 3000 - /metrics
• Loopback 4 - no metrics
• SpringBoot 2 - HTTP 8080 - /actuator/prometheus
• MicroProfile on Liberty - HTTP - 9080 -> HTTPS 9443 - /metrics - Basic Auth (admin/adminpwd)

[seabaylea]

prometheus.io/scrape is added by the Appsody stack. Once we implement this issue, we can open an issue against stacks to remove this.

We should keep the existing annotations as well unless there's a good reason to remove them - anyone that's deploying Prometheus using Helm (which is most users) will still require the annotation.