No data shown in Grafana Dashboards

[Loahrs]

I noticed that I don't see any data in my Grafana Dashboards. I hoped for this problem to be fixed after updating to the latest version of the chart (3.0.0. -> 3.3.0) but it persists ever since. All dashboards are showing no data.

I checked Grafanas settings and see that a Prometheus datasource is configured (http://pulsar-kube-prometheus-sta-prometheus.default:9090). If I click on "Test" to test the connection, I receive "Succesfully queried the Prometheus API".

After that I opened the Prometheus UI and checked the configuration under http://prometheus-address:9090/config. In it I see a bunch of jobs related to Pulsar:

job_name: podMonitor/default/pulsar-zookeeper/0
job_name: podMonitor/default/pulsar-proxy/0
job_name: podMonitor/default/pulsar-broker/0
job_name: podMonitor/default/pulsar-bookie/0

Looking up the Metrics Explorer I can't see any Pulsar related metrics.

I'll post here my values.yaml:

clusterName: cluster-a
namespace: pulsar
namespaceCreate: false
initialize: false

auth:
  authentication:
    enabled: true
    jwt:
      usingSecretKey: false
    provider: jwt
  authorization:
    enabled: true
  superUsers:
    broker: broker-admin
    client: admin
    proxy: proxy-admin

broker:
  configData:
    proxyRoles: proxy-admin

certs:
  internal_issuer:
    enabled: true
    type: selfsigning

components:
  pulsar_manager: false

tls:
  broker:
    enabled: true
  enabled: true
  proxy:
    enabled: true
  zookeeper:
    enabled: true

I installed the Pulsar helm chart into a namespace "pulsar" and noticed that all Grafana-Stack related components were installed into the "default" namespace.

Could this be an issue? I also enabled authentication/authorization, maybe the issue has to do with that?

[lhotari]

This might be caused by the configured authentication. I guess metrics requires a token currently.

[lhotari]

for the broker authenticateMetricsEndpoint defaults to false, so it might be something else.