Emissary Ingress is not returning any data in Grafana

[imoisharma]

Hi Alex,

I am using Emissary Ingress 3.6 and I enable the service monitor inside Helm values file and created a service monitor configuration

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8slens-edit-resource-version: v1
    release: prometheus
  name: ambassador-monitor
  namespace: monitoring
spec:
  endpoints:
    - port: emissary-poc-emissary-ingress-admin
      # path: /metrics
  namespaceSelector:
    matchNames:
      - emissary-poc
  selector:
    matchLabels:
      service: emissary-poc-emissary-ingress-admin

but it's not working. Is there any way to fix this?

[yalattas]

same here, metrics seems to be scrapped correctly via VictoriaMetrics. Yet, dashboard doesn't show anything

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    app: ambassador
    argocd.argoproj.io/instance: emissary-ingress
  name: ambassador
  namespace: ambassador
  resourceVersion: "23845"
spec:
  endpoints:
  - interval: 30s
    path: /metrics
    port: ambassador-admin
    scrapeTimeout: 30s
  namespaceSelector:
    matchNames:
    - ambassador
  selector:
    matchLabels:
      service: ambassador-admin

I noticed the following in Ambassador pod

apiVersion: v1
kind: Pod
metadata:
  name: ambassador-6954fd4445-b4nbx
  namespace: ambassador
spec:
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    - containerPort: 8443
      name: https
      protocol: TCP
    - containerPort: 8877
      name: admin
      protocol: TCP

Possibly is mis-configuration from Emissary side, I noticed that port name in the pod is admin, Yet, ServiceMonitor looking for ambassador-admin which is different

In addition, this dashboard expects the following

Displays statistics exported from Ambassador Edge Stack, Emissary-ingress and Envoy scraped on the :8877/metrics endpoint.

Not sure if its related or not

[yalattas]

Just discovered the issue, VMServiceScrape is not watching ServiceMonitor. I got the dashboard working by applying the following, which means ServiceMonitor is not managed by VM

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: ambassador-scrape
  namespace: ambassador
spec:
  endpoints:
    - port: ambassador-admin  # Name or number of the port to scrape metrics from
      interval: 15s  # Scrape interval for fetching metrics (adjust as needed)
      path: /metrics  # Path where metrics are exposed (optional, defaults to '/metrics')
      scheme: http  # Protocol scheme ('http' or 'https' depending on your setup)