ServiceMonitor cannot access service port

[r2ronoha]

I'm trying to set up a service monitor on the authentication service, and it does not seem to be able to access the port.

Checks:

• app is up and running properly
• port is reachable from the cluster and other apps
• I've deployed a prometheus test container on the deployment + service, and pointed the servicemonitor to it, and it works as expected

Is it possible to monitor the port? If so, is there any additional configuration required to do so?

• Standard config

  apiVersion: apps/v1
  kind: Deployment
  metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
  app: traefik-forward-auth
  chart: traefik-forward-auth-0.0.1
  release: wandera
  heritage: Tiller
  spec:
  replicas: 1
  selector:
  matchLabels:
    app: traefik-forward-auth
  strategy:
  type: Recreate
  template:
  metadata:
    labels:
      app: traefik-forward-auth
  spec:
    terminationGracePeriodSeconds: 60
    containers:          
    - image: mesosphere/traefik-forward-auth:3.1.0
      name: traefik-forward-auth
      ports:
      - containerPort: 4181
        protocol: TCP
      env:
      - name: LOG_LEVEL
        value: debug
      - name: DOMAIN
        value: my-domain
      - name: INSECURE_COOKIE
        value: "false"
      - name: URL_PATH
        value: /oauth2/callback
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: secret
      - name: PROVIDER_URI
        value: https://my-okta-domain/oauth2/default
      - name: CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: client-id
      - name: CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: client-secret
      - name: SCOPE
        value: email profile openid
  ---
  apiVersion: v1
  kind: Service
  metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
  app: traefik-forward-auth
  chart: traefik-forward-auth-0.0.1
  release: wandera
  heritage: Tiller
  spec:
  type: ClusterIP
  selector:
  app: traefik-forward-auth
  ports:
  - name: auth-http
  port: 4181
  targetPort: 4181
  ---
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
  app: traefik-forward-auth
  chart: traefik-forward-auth-0.0.1
  heritage: Tiller
  release: wandera
  spec:
  jobLabel: app
  selector:
  matchLabels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
  namespaceSelector:
  matchNames:
  - ingress-traefik
  endpoints:
  - port: auth-http
  interval: 30s

• Config with additional container/service

  
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
  app: traefik-forward-auth
  chart: traefik-forward-auth-0.0.1
  release: wandera
  heritage: Tiller
  spec:
  replicas: 1
  selector:
  matchLabels:
    app: traefik-forward-auth
  strategy:
  type: Recreate
  template:
  metadata:
    labels:
      app: traefik-forward-auth
  spec:
    terminationGracePeriodSeconds: 60
    containers:
    - name: prom-test
      image: supergiantkir/prometheus-test-app
      ports:
      - name: prom-test
        containerPort: 8081
      resources:
        limits:
          cpu: 100m
          memory: 20Mi
        requests:
          cpu: 10m
          memory: 20Mi
  
    - image: mesosphere/traefik-forward-auth:3.1.0
      name: traefik-forward-auth
      ports:
      - containerPort: 4181
        protocol: TCP
      env:
      - name: LOG_LEVEL
        value: debug
      - name: DOMAIN
        value: my-domain
      - name: INSECURE_COOKIE
        value: "false"
      - name: URL_PATH
        value: /oauth2/callback
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: secret
      - name: PROVIDER_URI
        value: https://my-okta-domain/oauth2/default
      - name: CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: client-id
      - name: CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: traefik-forward-auth
            key: client-secret
      - name: SCOPE
        value: email profile openid

---

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: traefik-forward-auth namespace: ingress-traefik labels: app: traefik-forward-auth chart: traefik-forward-auth-0.0.1 heritage: Tiller release: wandera spec: jobLabel: app selector: matchLabels: app: traefik-forward-auth chart: traefik-forward-auth-0.0.1 namespaceSelector: matchNames:

• ingress-traefik endpoints:
  • port: prom-test interval: 30s

---

apiVersion: v1 kind: Service metadata: name: traefik-forward-auth namespace: ingress-traefik labels: app: traefik-forward-auth chart: traefik-forward-auth-0.0.1 release: wandera heritage: Tiller spec: type: ClusterIP selector: app: traefik-forward-auth ports:

• name: auth-http port: 4181 targetPort: 4181

  apiVersion: v1 kind: Service metadata: name: traefik-forward-auth-monitoring namespace: ingress-traefik labels: app: traefik-forward-auth chart: traefik-forward-auth-0.0.1 release: wandera heritage: Tiller spec: selector: app: traefik-forward-auth ports:

• name: prom-test port: 8081 targetPort: 8081