JMX exporter empty metrics

PPACI commented 4 years ago

Hello Confluentinc people

Context

I want to monitor the confluentic platform using the included JMX exporter and prometheus

Was was expected

The prometheus JMX exporter should have exported something like the following in its /metrics endpoint

# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP cp_kafka_controller_kafkacontroller_activecontrollercount Attribute exposed for management (kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_activecontrollercount untyped
cp_kafka_controller_kafkacontroller_activecontrollercount 0.0
# HELP cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=TopicUncleanLeaderElectionEnableRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems untyped
cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems 0.0
# HELP cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=PartitionReassignmentRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems untyped
cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec untyped
cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec 0.0
# HELP cp_kafka_controller_controllerstats_logdirchangerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=LogDirChangeRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_logdirchangerateandtimems untyped
cp_kafka_controller_controllerstats_logdirchangerateandtimems 0.0
...

What I got

The JMX exporter only exported the basic JMX data

# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0

I had nothing related to cp_kafka at all.

This was all the same for every sub chart (kafka, zookeeper, etc.)

How I solved the issue

I manually edited this configmap https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/templates/jmx-configmap.yaml#L17 to remove the whitelistObjectNames section. Now it's scraping everything. For reference, the "upstream" repository for the JMX exporter provide an example without the whitelistObjectNames to scrape all https://github.com/solsson/dockerfiles/blob/master/prometheus-jmx-exporter/collect-all-slow.yml

So now I have two (linked) questions:

Is it a problem to leave my config map like this? Beside the fact that it will, of course, generate more data..
Could we remove the whitelistObjectNames section in all configmap in this repository directly?

flxs commented 4 years ago

I've been seeing the same behavior, but from Kafka itself. Removing the whitelistObjectNames section made everything show up, before I've had only the same basic data as you. In this case, this issue started when we updated to the latest version of the chart (0.5.0)

azhang commented 4 years ago

It's working for me, without removing whitelistObjectNames. 0.5.0 as well.

olysyuk commented 3 years ago

I can confirm that with 0.4.1 helm chart version and imageTag: 5.4.1 it shows metrics. Curl https://localhost:5556/metrics gives

# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP cp_kafka_controller_kafkacontroller_activecontrollercount Attribute exposed for management (kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_activecontrollercount untyped
cp_kafka_controller_kafkacontroller_activecontrollercount 0.0
# HELP cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=TopicUncleanLeaderElectionEnableRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems untyped
cp_kafka_controller_controllerstats_topicuncleanleaderelectionenablerateandtimems 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_produce_version_8 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=Produce, version=8><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_produce_version_8 untyped
cp_kafka_network_requestmetrics_requestspersec_produce_version_8 0.4292851664759036
# HELP cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=PartitionReassignmentRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems untyped
cp_kafka_controller_controllerstats_partitionreassignmentrateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec untyped
cp_kafka_server_brokertopicmetrics_failedfetchrequestspersec 0.0
# HELP cp_kafka_controller_controllerstats_logdirchangerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=LogDirChangeRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_logdirchangerateandtimems untyped
cp_kafka_controller_controllerstats_logdirchangerateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_status Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=confluent-kafka-cp-kafka-connect-status><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_status untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_status 15.677319653824064
# HELP cp_kafka_controller_controllerstats_isrchangerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=IsrChangeRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_isrchangerateandtimems untyped
cp_kafka_controller_controllerstats_isrchangerateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_invalidmagicnumberrecordspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=InvalidMagicNumberRecordsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_invalidmagicnumberrecordspersec untyped
cp_kafka_server_brokertopicmetrics_invalidmagicnumberrecordspersec 0.0
# HELP cp_kafka_controller_controllerstats_leaderelectionrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=LeaderElectionRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_leaderelectionrateandtimems untyped
cp_kafka_controller_controllerstats_leaderelectionrateandtimems 0.0
# HELP cp_kafka_server_replicamanager_isrshrinkspersec Attribute exposed for management (kafka.server<type=ReplicaManager, name=IsrShrinksPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_replicamanager_isrshrinkspersec untyped
cp_kafka_server_replicamanager_isrshrinkspersec 0.0
# HELP cp_kafka_controller_controllerstats_uncleanleaderelectionenablerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=UncleanLeaderElectionEnableRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_uncleanleaderelectionenablerateandtimems untyped
cp_kafka_controller_controllerstats_uncleanleaderelectionenablerateandtimems 0.0
# HELP cp_kafka_controller_controllerstats_topicdeletionrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=TopicDeletionRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_topicdeletionrateandtimems untyped
cp_kafka_controller_controllerstats_topicdeletionrateandtimems 0.0
# HELP cp_kafka_network_socketserver_networkprocessoravgidlepercent Attribute exposed for management (kafka.network<type=SocketServer, name=NetworkProcessorAvgIdlePercent><>Value)
# TYPE cp_kafka_network_socketserver_networkprocessoravgidlepercent untyped
cp_kafka_network_socketserver_networkprocessoravgidlepercent 0.9995840001533102
# HELP cp_kafka_network_requestmetrics_requestspersec_listoffsets_version_5 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=ListOffsets, version=5><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_listoffsets_version_5 untyped
cp_kafka_network_requestmetrics_requestspersec_listoffsets_version_5 3.359766110636556E-16
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec 424.24848772360207
# HELP cp_kafka_network_requestmetrics_requestspersec_fetchfollower_version_11 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=FetchFollower, version=11><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_fetchfollower_version_11 untyped
cp_kafka_network_requestmetrics_requestspersec_fetchfollower_version_11 6.443429515381385
# HELP cp_kafka_network_requestmetrics_requestspersec_metadata_version_9 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=Metadata, version=9><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_metadata_version_9 untyped
cp_kafka_network_requestmetrics_requestspersec_metadata_version_9 0.04388066293022111
# HELP cp_kafka_controller_kafkacontroller_topicstodeletecount Attribute exposed for management (kafka.controller<type=KafkaController, name=TopicsToDeleteCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_topicstodeletecount untyped
cp_kafka_controller_kafkacontroller_topicstodeletecount 0.0
# HELP cp_kafka_server_brokertopicmetrics_invalidmessagecrcrecordspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=InvalidMessageCrcRecordsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_invalidmessagecrcrecordspersec untyped
cp_kafka_server_brokertopicmetrics_invalidmessagecrcrecordspersec 0.0
# HELP cp_kafka_controller_kafkacontroller_replicastodeletecount Attribute exposed for management (kafka.controller<type=KafkaController, name=ReplicasToDeleteCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_replicastodeletecount untyped
cp_kafka_controller_kafkacontroller_replicastodeletecount 0.0
# HELP cp_kafka_controller_kafkacontroller_controllerstate Attribute exposed for management (kafka.controller<type=KafkaController, name=ControllerState><>Value)
# TYPE cp_kafka_controller_kafkacontroller_controllerstate untyped
cp_kafka_controller_kafkacontroller_controllerstate 0.0
# HELP cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_support_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=__confluent.support.metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_support_metrics untyped
cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_support_metrics 7.247759409772295E-13
# HELP cp_kafka_network_requestmetrics_requestspersec_updatemetadata_version_6 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=UpdateMetadata, version=6><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_updatemetadata_version_6 untyped
cp_kafka_network_requestmetrics_requestspersec_updatemetadata_version_6 1.2248456745834972E-15
# HELP cp_kafka_controller_controllerstats_controlledshutdownrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=ControlledShutdownRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_controlledshutdownrateandtimems untyped
cp_kafka_controller_controllerstats_controlledshutdownrateandtimems 0.0
# HELP cp_kafka_server_sessionexpirelistener_zookeeperreadonlyconnectspersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperReadOnlyConnectsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeeperreadonlyconnectspersec untyped
cp_kafka_server_sessionexpirelistener_zookeeperreadonlyconnectspersec 0.0
# HELP cp_kafka_server_replicamanager_offlinereplicacount Attribute exposed for management (kafka.server<type=ReplicaManager, name=OfflineReplicaCount><>Value)
# TYPE cp_kafka_server_replicamanager_offlinereplicacount untyped
cp_kafka_server_replicamanager_offlinereplicacount 0.0
# HELP cp_kafka_controller_controllerstats_uncleanleaderelectionspersec Attribute exposed for management (kafka.controller<type=ControllerStats, name=UncleanLeaderElectionsPerSec><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_uncleanleaderelectionspersec untyped
cp_kafka_controller_controllerstats_uncleanleaderelectionspersec 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_syncgroup_version_4 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=SyncGroup, version=4><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_syncgroup_version_4 untyped
cp_kafka_network_requestmetrics_requestspersec_syncgroup_version_4 3.1537437592527264E-16
# HELP cp_kafka_network_requestmetrics_requestspersec_metadata_version_4 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=Metadata, version=4><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_metadata_version_4 untyped
cp_kafka_network_requestmetrics_requestspersec_metadata_version_4 0.0026650089411346238
# HELP cp_kafka_controller_controllerstats_controllershutdownrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=ControllerShutdownRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_controllershutdownrateandtimems untyped
cp_kafka_controller_controllerstats_controllershutdownrateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_fetchmessageconversionspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=FetchMessageConversionsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_fetchmessageconversionspersec untyped
cp_kafka_server_brokertopicmetrics_fetchmessageconversionspersec 0.0
# HELP cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_schemas Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=_schemas><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_schemas untyped
cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_schemas 3.9918259648137384E-14
# HELP cp_kafka_server_replicafetchermanager_maxlag_replica Attribute exposed for management (kafka.server<type=ReplicaFetcherManager, name=MaxLag, clientId=Replica><>Value)
# TYPE cp_kafka_server_replicafetchermanager_maxlag_replica untyped
cp_kafka_server_replicafetchermanager_maxlag_replica 0.0
# HELP cp_kafka_controller_kafkacontroller_globalpartitioncount Attribute exposed for management (kafka.controller<type=KafkaController, name=GlobalPartitionCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_globalpartitioncount untyped
cp_kafka_controller_kafkacontroller_globalpartitioncount 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_fetch_version_11 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=Fetch, version=11><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_fetch_version_11 untyped
cp_kafka_network_requestmetrics_requestspersec_fetch_version_11 12.422623458233604
# HELP cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec, topic=_confluent-metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_metrics untyped
cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_metrics 0.6094519953730376
# HELP cp_kafka_network_requestmetrics_requestspersec_joingroup_version_6 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=JoinGroup, version=6><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_joingroup_version_6 untyped
cp_kafka_network_requestmetrics_requestspersec_joingroup_version_6 6.307487518505453E-16
# HELP cp_kafka_server_brokertopicmetrics_totalproducerequestspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalproducerequestspersec untyped
cp_kafka_server_brokertopicmetrics_totalproducerequestspersec 0.5898305017032198
# HELP cp_kafka_network_requestmetrics_requestspersec_apiversions_version_3 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=ApiVersions, version=3><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_apiversions_version_3 untyped
cp_kafka_network_requestmetrics_requestspersec_apiversions_version_3 9.022756608146356E-15
# HELP cp_kafka_server_replicamanager_isrexpandspersec Attribute exposed for management (kafka.server<type=ReplicaManager, name=IsrExpandsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_replicamanager_isrexpandspersec untyped
cp_kafka_server_replicamanager_isrexpandspersec 0.0
# HELP cp_kafka_controller_controllerstats_leaderandisrresponsereceivedrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=LeaderAndIsrResponseReceivedRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_leaderandisrresponsereceivedrateandtimems untyped
cp_kafka_controller_controllerstats_leaderandisrresponsereceivedrateandtimems 0.0
# HELP cp_kafka_controller_controllerstats_manualleaderbalancerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=ManualLeaderBalanceRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_manualleaderbalancerateandtimems untyped
cp_kafka_controller_controllerstats_manualleaderbalancerateandtimems 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_leaderandisr_version_4 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=LeaderAndIsr, version=4><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_leaderandisr_version_4 untyped
cp_kafka_network_requestmetrics_requestspersec_leaderandisr_version_4 9.154745190752675E-16
# HELP cp_kafka_controller_kafkacontroller_topicsineligibletodeletecount Attribute exposed for management (kafka.controller<type=KafkaController, name=TopicsIneligibleToDeleteCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_topicsineligibletodeletecount untyped
cp_kafka_controller_kafkacontroller_topicsineligibletodeletecount 0.0
# HELP cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_support_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec, topic=__confluent.support.metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_support_metrics untyped
cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_confluent_support_metrics 6.237228070021466E-16
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_support_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=__confluent.support.metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_support_metrics untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_support_metrics 5.8156082889399165
# HELP cp_kafka_controller_kafkacontroller_globaltopiccount Attribute exposed for management (kafka.controller<type=KafkaController, name=GlobalTopicCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_globaltopiccount untyped
cp_kafka_controller_kafkacontroller_globaltopiccount 0.0
# HELP cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_schemas Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=_schemas><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_schemas untyped
cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_schemas 4.519512298062532E-16
# HELP cp_kafka_kafkarequesthandlerpool_requesthandleravgidlepercent Attribute exposed for management (kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate)
# TYPE cp_kafka_kafkarequesthandlerpool_requesthandleravgidlepercent untyped
cp_kafka_kafkarequesthandlerpool_requesthandleravgidlepercent 0.999787748510939
# HELP cp_kafka_server_brokertopicmetrics_replicationbytesinpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=ReplicationBytesInPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_replicationbytesinpersec untyped
cp_kafka_server_brokertopicmetrics_replicationbytesinpersec 3518.4092161348262
# HELP cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_schemas Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec, topic=_schemas><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_schemas untyped
cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_schemas 4.519512298062532E-16
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=_confluent-metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_metrics untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_metrics 46.01798058700829
# HELP cp_kafka_controller_kafkacontroller_replicasineligibletodeletecount Attribute exposed for management (kafka.controller<type=KafkaController, name=ReplicasIneligibleToDeleteCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_replicasineligibletodeletecount untyped
cp_kafka_controller_kafkacontroller_replicasineligibletodeletecount 0.0
# HELP cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_consumer_offsets Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec, topic=__consumer_offsets><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_consumer_offsets untyped
cp_kafka_server_brokertopicmetrics_totalproducerequestspersec_topic_consumer_offsets 3.1537437592527264E-16
# HELP cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_schemas Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=_schemas><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_schemas untyped
cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_schemas 3.9918259648137384E-14
# HELP cp_kafka_server_brokertopicmetrics_bytesrejectedpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesRejectedPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesrejectedpersec untyped
cp_kafka_server_brokertopicmetrics_bytesrejectedpersec 0.0
# HELP cp_kafka_server_brokertopicmetrics_nokeycompactedtopicrecordspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=NoKeyCompactedTopicRecordsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_nokeycompactedtopicrecordspersec untyped
cp_kafka_server_brokertopicmetrics_nokeycompactedtopicrecordspersec 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_heartbeat_version_4 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=Heartbeat, version=4><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_heartbeat_version_4 untyped
cp_kafka_network_requestmetrics_requestspersec_heartbeat_version_4 0.3388054417561008
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_offset Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=confluent-kafka-cp-kafka-connect-offset><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_offset untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_confluent_kafka_cp_kafka_connect_offset 138.59310752443412
# HELP cp_kafka_server_brokertopicmetrics_failedproducerequestspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_failedproducerequestspersec untyped
cp_kafka_server_brokertopicmetrics_failedproducerequestspersec 0.0
# HELP cp_kafka_server_brokertopicmetrics_messagesinpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_messagesinpersec untyped
cp_kafka_server_brokertopicmetrics_messagesinpersec 0.7262622466368989
# HELP cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_offset Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=confluent-kafka-cp-kafka-connect-offset><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_offset untyped
cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_offset 0.0
# HELP cp_kafka_controller_kafkacontroller_offlinepartitionscount Attribute exposed for management (kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_offlinepartitionscount untyped
cp_kafka_controller_kafkacontroller_offlinepartitionscount 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_fetchconsumer_version_11 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=FetchConsumer, version=11><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_fetchconsumer_version_11 untyped
cp_kafka_network_requestmetrics_requestspersec_fetchconsumer_version_11 5.992614135140193
# HELP cp_kafka_server_replicamanager_leadercount Attribute exposed for management (kafka.server<type=ReplicaManager, name=LeaderCount><>Value)
# TYPE cp_kafka_server_replicamanager_leadercount untyped
cp_kafka_server_replicamanager_leadercount 34.0
# HELP cp_kafka_server_brokertopicmetrics_replicationbytesoutpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=ReplicationBytesOutPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_replicationbytesoutpersec untyped
cp_kafka_server_brokertopicmetrics_replicationbytesoutpersec 4161.754570593261
# HELP cp_kafka_controller_controllerstats_topicchangerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=TopicChangeRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_topicchangerateandtimems untyped
cp_kafka_controller_controllerstats_topicchangerateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=_confluent-metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_metrics untyped
cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_metrics 0.7475774451773249
# HELP cp_kafka_server_replicamanager_underminisrpartitioncount Attribute exposed for management (kafka.server<type=ReplicaManager, name=UnderMinIsrPartitionCount><>Value)
# TYPE cp_kafka_server_replicamanager_underminisrpartitioncount untyped
cp_kafka_server_replicamanager_underminisrpartitioncount 0.0
# HELP cp_kafka_server_brokertopicmetrics_bytesinpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesinpersec untyped
cp_kafka_server_brokertopicmetrics_bytesinpersec 2261.7139479456496
# HELP cp_kafka_server_brokertopicmetrics_producemessageconversionspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=ProduceMessageConversionsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_producemessageconversionspersec untyped
cp_kafka_server_brokertopicmetrics_producemessageconversionspersec 0.0
# HELP cp_kafka_server_replicamanager_partitioncount Attribute exposed for management (kafka.server<type=ReplicaManager, name=PartitionCount><>Value)
# TYPE cp_kafka_server_replicamanager_partitioncount untyped
cp_kafka_server_replicamanager_partitioncount 96.0
# HELP cp_kafka_server_sessionexpirelistener_zookeeperauthfailurespersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperAuthFailuresPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeeperauthfailurespersec untyped
cp_kafka_server_sessionexpirelistener_zookeeperauthfailurespersec 0.0
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_consumer_offsets Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=__consumer_offsets><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_consumer_offsets untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_consumer_offsets 196.1770640786753
# HELP cp_kafka_server_replicamanager_underreplicatedpartitions Attribute exposed for management (kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value)
# TYPE cp_kafka_server_replicamanager_underreplicatedpartitions untyped
cp_kafka_server_replicamanager_underreplicatedpartitions 0.0
# HELP cp_kafka_network_requestmetrics_requestspersec_findcoordinator_version_3 Attribute exposed for management (kafka.network<type=RequestMetrics, name=RequestsPerSec, request=FindCoordinator, version=3><>OneMinuteRate)
# TYPE cp_kafka_network_requestmetrics_requestspersec_findcoordinator_version_3 untyped
cp_kafka_network_requestmetrics_requestspersec_findcoordinator_version_3 1.0395380116702435E-15
# HELP cp_kafka_controller_controllerstats_controllerchangerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=ControllerChangeRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_controllerchangerateandtimems untyped
cp_kafka_controller_controllerstats_controllerchangerateandtimems 2.079076023340485E-16
# HELP cp_kafka_server_sessionexpirelistener_zookeeperdisconnectspersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperDisconnectsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeeperdisconnectspersec untyped
cp_kafka_server_sessionexpirelistener_zookeeperdisconnectspersec 0.0
# HELP cp_kafka_controller_controllerstats_autoleaderbalancerateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=AutoLeaderBalanceRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_autoleaderbalancerateandtimems untyped
cp_kafka_controller_controllerstats_autoleaderbalancerateandtimems 0.0
# HELP cp_kafka_server_sessionexpirelistener_zookeepersyncconnectspersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperSyncConnectsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeepersyncconnectspersec untyped
cp_kafka_server_sessionexpirelistener_zookeepersyncconnectspersec 1.9128422828641576E-16
# HELP cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_schemas Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec, topic=_schemas><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_schemas untyped
cp_kafka_server_brokertopicmetrics_totalfetchrequestspersec_topic_schemas 15.695485239161133
# HELP cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_consumer_offsets Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=__consumer_offsets><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_consumer_offsets untyped
cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_consumer_offsets 1.2930349412936186E-13
# HELP cp_kafka_server_brokertopicmetrics_invalidoffsetorsequencerecordspersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=InvalidOffsetOrSequenceRecordsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_invalidoffsetorsequencerecordspersec untyped
cp_kafka_server_brokertopicmetrics_invalidoffsetorsequencerecordspersec 0.0
# HELP cp_kafka_server_sessionexpirelistener_zookeepersaslauthenticationspersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperSaslAuthenticationsPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeepersaslauthenticationspersec untyped
cp_kafka_server_sessionexpirelistener_zookeepersaslauthenticationspersec 0.0
# HELP cp_kafka_controller_controllerstats_listpartitionreassignmentrateandtimems Attribute exposed for management (kafka.controller<type=ControllerStats, name=ListPartitionReassignmentRateAndTimeMs><>OneMinuteRate)
# TYPE cp_kafka_controller_controllerstats_listpartitionreassignmentrateandtimems untyped
cp_kafka_controller_controllerstats_listpartitionreassignmentrateandtimems 0.0
# HELP cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_consumer_offsets Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=__consumer_offsets><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_consumer_offsets untyped
cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_consumer_offsets 3.1537437592527264E-16
# HELP cp_kafka_server_replicamanager_atminisrpartitioncount Attribute exposed for management (kafka.server<type=ReplicaManager, name=AtMinIsrPartitionCount><>Value)
# TYPE cp_kafka_server_replicamanager_atminisrpartitioncount untyped
cp_kafka_server_replicamanager_atminisrpartitioncount 1.0
# HELP cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=_confluent-metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_metrics untyped
cp_kafka_server_brokertopicmetrics_bytesinpersec_topic_confluent_metrics 2159.7990771759664
# HELP cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_support_metrics Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=__confluent.support.metrics><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_support_metrics untyped
cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_confluent_support_metrics 6.237228070021466E-16
# HELP cp_kafka_server_replicamanager_failedisrupdatespersec Attribute exposed for management (kafka.server<type=ReplicaManager, name=FailedIsrUpdatesPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_replicamanager_failedisrupdatespersec untyped
cp_kafka_server_replicamanager_failedisrupdatespersec 0.0
# HELP cp_kafka_controller_kafkacontroller_preferredreplicaimbalancecount Attribute exposed for management (kafka.controller<type=KafkaController, name=PreferredReplicaImbalanceCount><>Value)
# TYPE cp_kafka_controller_kafkacontroller_preferredreplicaimbalancecount untyped
cp_kafka_controller_kafkacontroller_preferredreplicaimbalancecount 0.0
# HELP cp_kafka_server_brokertopicmetrics_bytesoutpersec Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesoutpersec untyped
cp_kafka_server_brokertopicmetrics_bytesoutpersec 3.469058413262994E-15
# HELP cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_status Attribute exposed for management (kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=confluent-kafka-cp-kafka-connect-status><>OneMinuteRate)
# TYPE cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_status untyped
cp_kafka_server_brokertopicmetrics_bytesoutpersec_topic_confluent_kafka_cp_kafka_connect_status 0.0
# HELP cp_kafka_server_sessionexpirelistener_zookeeperexpirespersec Attribute exposed for management (kafka.server<type=SessionExpireListener, name=ZooKeeperExpiresPerSec><>OneMinuteRate)
# TYPE cp_kafka_server_sessionexpirelistener_zookeeperexpirespersec untyped
cp_kafka_server_sessionexpirelistener_zookeeperexpirespersec 0.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 5.430316518
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0

In 0.5.0 helm chart version 5.5.0 image has missing Curl https://localhost:5556/metrics gives

root@confluent-kafka-cp-kafka-0:/# curl http://localhost:5556/metrics
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.00543444
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0

Most likely the problem is with prometheus-jmx-exporter whitelist added in this commit 15e3b73

SoerenHenning commented 3 years ago

This problem is reported here and in #464. In particular, that means that even with a default installation, the Grafana dashboard does not show all metrics. This issue was definitely introduces with #428 as when commenting out the whitelistObjectNames everything is working fine again.

I would like to support in solving this issue, but I'm neither experienced in Prometheus nor in Helm. What would be a good way to solve this issue? I'm currently thinking of three options:

Revert the ConfigMaps and remove the whitelistObjectNames part as suggested by @PPACI. This, however, has the disadvantages reported by @srolija in #428.
Explicitly add all metrics required for the Grafana dashboard. Does someone know which additional object names should be whitelisted? And also, what would be the impact on performance?
Make the ConfigMaps configurable via Helm. This seems to be quite a clean solution. However, question would be how reasonable defaults could look like. And, as I mentioned, I not sure how to adjust the Helm chart in that case.

In the end, I think if Grafana dashboards are provided with this repo/Helm chart, they should work. Otherwise, I would suggest to remove them (which would be the worst solution of all). @gAmUssA do you have any thoughts on this?

srolija commented 3 years ago

Since I made that PR, let me jump in. Those whitelists were based on the dashboard that was current at that time and since we are using the Grafana dashboard as well, we tested with that one. It should work since we are running the exact same version on our production.

Do note that initial PR had an formatting issue which I fixed a week latter but didn't end up in 5.5.0, rather only in v6.0.1: https://github.com/confluentinc/cp-helm-charts/commit/596dd559c1085faf76a03a66178834578203c395

If you are keen on staying on 5.X I would recommend applying the patch from the linked commit, otherwise, from 6.0.1 it should work as expected.

Let me know if that helps and sorry for inconvenience.

SoerenHenning commented 3 years ago

Thank you for your comment. Based on it, I started to dig a bit deeper and found out that there seem to be two independent issues (at least in my setting).

Indeed, the provided dashboard seems to work almost as expected. For me, the dashboard does not show anything, but that is because the $Release variable cannot be resolved. When removing the $Release query filters, the plots do show something. (Could be an RBAC thing and is not a problem for me.)

However, for topic-specific metrics such as cp_kafka_server_brokertopicmetrics_messagesinpersec_topic_..., I am only able to switch between some "default" topics as shown below. For the topics I created by myself, no metrics are available. This is not the case when removing the whitelistObjectNames section. Do you @srolija have any idea what could cause this issue?

Bildschirmfoto von 2021-02-25 18-34-29

srolija commented 3 years ago

It could be that additional whitelist is needed; I didn't see that one in the default dashboard. Can you try adding the following for whitelist below the line: https://github.com/confluentinc/cp-helm-charts/blob/596dd559c1085faf76a03a66178834578203c395/charts/cp-kafka/templates/jmx-configmap.yaml#L20

- kafka.server:type=BrokerTopicMetrics,name=*,topic=*

SoerenHenning commented 3 years ago

So first, sorry for my initial misunderstanding. With the default whitelist and the default dashboard everything seems to work as expected. So there is nothing "wrong" with the current version of the chart.

I replaced your suggested whitelist rule and, indeed, now I'm getting topic metrics again. Thanks!

Question would now be how we could support this in the upstream chart? I could open a pull request for your suggested change, but the cleaner solution would be to allow configuration via Helm values.yaml, right?

srolija commented 3 years ago

That is likely question for someone else within the team that is maintaining the chart. :)

I can tell you what we did for our use-case. We are using the https://github.com/danielqsj/kafka_exporter for those metrics instead. We liked that it used labels instead of metric names and if I recall correctly, it exposes some additional statistics. It also didn't timeout unlike the JMX exporter for which I made a PR here. While that works for us, I would note that it doesn't look like it was updated for a while so keep that in mind if you are just starting out.

Also not sure what are you planning on using those metrics for; if you need them to alert for the lagging consumers, I would much more recommend using https://github.com/linkedin/Burrow, it is way smarter that it tracks the processing and lag instead of just looking at whether some limit was passed and latest version ships with exporter built in.

SoerenHenning commented 3 years ago

Thanks again for your elaborate thoughts. I opened a pull request for including the broker topic metrics (#499), but would also be fine with some other solution.

Currently the only reason we need these metrics for is displaying the rate of incoming messages per topic in a Grafana dashboard. We're performing heavy load testing on stream processing engines by sending huge amounts of data to Kafka and have to ensure that the desired data generation rate is actually received at Kafka.

For monitoring the consumer lag, we're using Lightbend's Kafka Lag Exporter, which works fine for us. I think it's sort of a Burrow light. Monitoring the consumer lag is actually the central task in our Theodolite framework. :smiley:

sshah90 commented 3 years ago

I can see other metrics but not able to see cp_kafka_network_requestmetrics metrics at all.

If some can help me here.

paul-bormans commented 2 years ago

@srolija @SoerenHenning I also experienced the topic specific metrics (in BrokerTopicMetrics) are missing when running CP 6.1.0 using chart version 0.6.1. And i also can confirm i do get them when i extend the cm with:

    - kafka.server:type=BrokerTopicMetrics,name=*,topic=*

I'm wondering however why there isn't a default rule setup to extract the topic name from the metric and publish it as a label?

There is even a confluent blog with code example on this how to do it.:

But to be honest i'm a bit puzzled; that's why i respond here on the topic. There are many kafka dashboards on grafana (like Kafka Metrics (https://grafana.com/grafana/dashboards/11962) which seems to be expecting topics as a label. Yet confluent uses a minimal rules setup and so as a reuslt most dashboards won't work out of the box.

Is there an upstream example / best practice to which i can match my modified jmx-exporter rules?

Paul

SoerenHenning commented 2 years ago

@paul-bormans To be honest, I don't think that this Helm chart is maintained anymore. Last commit is 9 month old, pull requests are not merged/discussed and the original maintainer (@gAmUssA) is not working for Confluent anymore. If possible for you, I would suggest to either use a different Helm chart (e.g. the Bitnami Kafka chart) or what we'll do is using the Strimzi Kafka operator.

PPACI commented 2 years ago

I switched my company of Strimzi. It’s way more stable and maintained.

From: Sören Henning @.> Sent: Tuesday, January 4, 2022 2:15:07 PM To: confluentinc/cp-helm-charts @.> Cc: Pierre PACI @.>; Mention @.> Subject: Re: [confluentinc/cp-helm-charts] JMX exporter empty metrics (#461)

@paul-bormanshttps://github.com/paul-bormans To be honest, I don't think that this Helm chart is maintained anymore. Last commit is 9 month old, pull requests are not merged/discussed and the original maintainer @.***https://github.com/gAmUssA) is not working for Confluent anymore. If possible for you, I would suggest to either use a different Helm chart (e.g. the Bitnami Kafka charthttps://github.com/bitnami/charts/tree/master/bitnami/kafka) or what we'll do is using the Strimzi Kafka operatorhttps://strimzi.io/.

— Reply to this email directly, view it on GitHubhttps://github.com/confluentinc/cp-helm-charts/issues/461#issuecomment-1004800541, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABDRRTT3TQQNCPUUDABC5LDUULXFXANCNFSM4RXGVUEA. You are receiving this because you were mentioned.Message ID: @.***>

confluentinc / cp-helm-charts