Can't get Kafka metrics on Prometheus server with listen cmd

[tomasrojas26]

Hello! I'm running gNMIc as a server (Listen) with this command line:

gnmic listen --config /gnmic-config.yaml --prometheus-address :9805

This is the Kafka Output configuration:

  kafka:
    type: kafka
    address: (my bootstrap servers)
    topic: (my topic)
    enable-metrics: true

According to the documentation, the kafka output metrics are exposed when a prometheus server is enabled, but it's not my case. By the way, my prometheus endpoint works fine, exposing the grpc metrics, memory and other useful data, but I can't get Kafka Output metrics. I read this issue #175 and the only difference is that they expose the metrics through 7890 port (API Rest), which I can't get running for some reason, I think the API can't be configured in a server mode.

Thank you in advance for your help.

[karimra]

Hi,

The outputs metrics are not available when running the listen command (That's btw a gNMI dial-out method specific to Nokia SROS routers). They are available when using the subscribe command. If you cannot switch to using dial-in (or dial-out using grpc-tunnel), I can make the outputs metrics available on the listen command prometheus server.

[tomasrojas26]

Hi, thank you for your answer.

I'm sure that I can't switch to dial-in method because we're using it with Nokia SROS routers, but let me take a look to the documentation to see how to config dial-out using grpc-tunnel. We have a large infrastructure working with the current configuration and I want to make sure this change won't affect the service.

I let you know as soon as possible, but if you have some recommendation, it's welcome.

Thanks!

[karimra]

Have a look here for configuring gRPC tunnels in SROS. And here for running gNMIc as a tunnel server.

Note that dial-out using gRPC tunnel works like dial-in (once the tunnel is established), meaning that unlike SROS dial-out the subscription(s) are defined on the collector side. If it's too much of a change to switch, I will add those metrics to the listen command prom server.

BTW: What's the blocking point in switching from the SROS dial-out to dial-in? The same paths should be supported in dial-in and dial-out mode.

[tomasrojas26]

Ok thank you for your guide. I understand the architecture and possible impact of changing to gRPC tunnel.

The main reason for staying on dial-out mode is because of a self-governance policy. We have multiple device owners, so we want to give them the freedom to configure the paths and parameters that better suits their capabilities.

It would be very helpful if you can add the kafka metrics to the listen command.

Please let me know if you need any other info related to my platform.

[karimra]

@tomasrojas26 Can you give v0.35.0 a try ? The output metrics are now initialized with the listen command.

[karimra]

feel free to reopen if this doesn't work as intended

[tomasrojas26]

Hi @karimra, thank you so much for the feature, it works perfect as I expected.

PD: Sorry for my delay, I was reasigned temporarily to another project and I hadn't time to work on this. Thank you again for your help!