Unable to get JMX metrics specific to Snowflake connector

[carao2020]

Unable to get JMX metrics specific to Snowflake connector. Could you please share the sample JMX configuration for Snowflake connector. I couldnt find this anywhere.

[sfc-gh-tzhang]

Would our online doc about JMX help here?

[carao2020]

I have tried but it's not working. Could you please share the sample JMX config file. Thanks

[sfc-gh-japatel]

Can you paste what you have tried? This is all you need

[sfc-gh-japatel]

Confluent: https://docs.confluent.io/platform/current/installation/docker/operations/monitoring.html

Reference articel: https://docs.datastax.com/en/kafka/doc/kafka/monitoring/kafkaEnableJmx.html

[carao2020]

I have tried with below pattern in my JMX config file. - pattern: 'snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+)><>(.+): (.+)' name: snowflake_kafka_connector_$3 labels: connector: "$1" pipe: "$2" category: "$3" "$4": "$5"

Additionally, PFB the config parameters in my dockerfile. ENV KAFKA_JMX_OPTS="-javaagent:/usr/share/java/kafka-connect/jmx_prometheus_javaagent-0.17.0.jar=0.0.0.0:9280:/opt/optum/jmx-config.yaml -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Still I couldnt find the JMX metrics mentioned as per snowflake documentation. Kindly help if there is any issue in the config.

[sfc-gh-japatel]

can you try this:

- pattern: snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+), name=(.+)><>(.+)
    name: snowflake_kc_$3_$4
    labels:
      connectorName: $1
      pipeName: $2
      category: $3
      metric: $4
    type: GAUGE

[sfc-gh-japatel]

Also do you use any ui based software like jconsole or visualvm to first identify if such mbeans are reporting the metrics?

[sfc-gh-japatel]

https://grafana.com/blog/2020/06/25/monitoring-java-applications-with-the-prometheus-jmx-exporter-and-grafana/

You should be able to go to localhost: for viewing logs if what was exported in jmx (mentioned in above mentioned link)

[carao2020]

Hi @sfc-gh-japatel ,

Sorry for late response on this thread. I have tried above config. However, still i am not getting any snowflake specific metrics.

• pattern: "snowflake.kafka.connector<>(.+)" name: snowflakekc$3_$4 labels: connector: $1 pipe: $2 category: $3 metric: $4 type: GAUGE

I dont see any metrics coming out of JMX with prefix snowflake.kafka.connector. However, I see the default metrics from Kafka connector. For example, I am getting below metrics.

kafka_connect_connector_metrics{connector_class="com.snowflake.kafka.connector.SnowflakeSinkConnector",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_version="1.8.1",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_type="sink",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{status="running",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_class="com.snowflake.kafka.connector.SnowflakeSinkConnector",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{connector_version="1.8.1",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{connector_type="sink",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{status="running",connector="gpt-dv-nwpr-raw-dev",} 1.0

These are the configs that i mentioned in my docker build file.

Connect JMX config

ENV JMX_PORT=9280 ENV KAFKA_JMX_HOSTNAME=127.0.0.1 ENV JMX_OPT=true

ENV KAFKA_JMX_OPTS="-javaagent:/usr/share/java/kafka-connect/jmx_prometheus_javaagent-0.17.2.jar=8085:/opt/optum/jmx-config.yaml -Djava.rmi.server.hostname=${KAFKA_JMX_HOSTNAME} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9280 -Dcom.sun.management.jmxremote.rmi.port=9280 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Kindly review and suggest.

[sfc-gh-japatel]

lowercaseOutputName: true
rules:
  # Example: snowflake.kafka.connector:connector=SnowflakeSinkConnector_SS_JP_Test,pipe=SNOWFLAKE_KAFKA_CONNECTOR_SnowflakeSinkConnector_SS_JP_Test_PIPE_JUNE_21_SS_LOAD_LOCAL_0,category=file-counts,name=file-count-on-ingestion
  - pattern: 'snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+), name=(.+)><>(.+): (.+)'
    name: snowflake_kc_$3_$4_$5
    value: $6
    labels: 
      connectorName: $1
      pipeName: $2
      category: $3
      metric: $5
    type: GAUGE

Note the space after comma in pattern line.

[carao2020]

Thanks for the response. I have tried with this pattern only in my JMX config file and i still couldnt get these metrics. Could you please help me with this issue.

[carao2020]

I am attaching my dockerfile

[carao2020]

This is the docker file that i am using and i am trying to listen to metrics on URL http://localhost:9403/metrics.

Still I am not getting any metrics related to snowflake. I am getting only JVM metrics.

HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.

TYPE jvm_gc_collection_seconds summary

jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 132.0 jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 2.088 jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0 jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0

HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.

TYPE jvm_memory_pool_allocated_bytes_total counter

jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 3.0817408E7 jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 2.4448564E8 jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.737490432E10 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1.1563136E7 jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 1.11149056E8 jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 2.0968E7 jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 1.48893824E8 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1506176.0

HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.

TYPE jmx_exporter_build_info gauge

jmx_exporter_build_info{version="0.17.2",name="jmx_prometheus_javaagent",} 1.0

HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.

TYPE jmx_config_reload_failure_total counter

jmx_config_reload_failure_total 0.0

HELP jvm_info VM version info

TYPE jvm_info gauge

jvm_info{runtime="OpenJDK Runtime Environment",vendor="Azul Systems, Inc.",version="11.0.16.1+1-LTS",} 1.0

HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.

TYPE process_cpu_seconds_total counter

process_cpu_seconds_total 230.66

HELP process_start_time_seconds Start time of the process since unix epoch in seconds.

TYPE process_start_time_seconds gauge

process_start_time_seconds 1.670582979265E9

HELP process_open_fds Number of open file descriptors.

TYPE process_open_fds gauge

process_open_fds 987.0

HELP process_max_fds Maximum number of open file descriptors.

TYPE process_max_fds gauge

process_max_fds 1048576.0

HELP process_virtual_memory_bytes Virtual memory size in bytes.

TYPE process_virtual_memory_bytes gauge

process_virtual_memory_bytes 9.687662592E9

HELP process_resident_memory_bytes Resident memory size in bytes.

TYPE process_resident_memory_bytes gauge

process_resident_memory_bytes 1.474605056E9

HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.

TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds 0.1118809

HELP jmx_scrape_error Non-zero if this scrape failed.

TYPE jmx_scrape_error gauge

jmx_scrape_error 0.0

HELP jmx_scrape_cached_beans Number of beans with their matching rule cached

TYPE jmx_scrape_cached_beans gauge

jmx_scrape_cached_beans 0.0

HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM

TYPE jvm_classes_currently_loaded gauge

jvm_classes_currently_loaded 32744.0

HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution

TYPE jvm_classes_loaded_total counter

jvm_classes_loaded_total 36965.0

HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution

TYPE jvm_classes_unloaded_total counter

jvm_classes_unloaded_total 4221.0

HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.

TYPE jmx_config_reload_success_total counter

jmx_config_reload_success_total 0.0

HELP jvm_threads_current Current thread count of a JVM

TYPE jvm_threads_current gauge

jvm_threads_current 197.0

HELP jvm_threads_daemon Daemon thread count of a JVM

TYPE jvm_threads_daemon gauge

jvm_threads_daemon 109.0

HELP jvm_threads_peak Peak thread count of a JVM

TYPE jvm_threads_peak gauge

jvm_threads_peak 198.0

HELP jvm_threads_started_total Started thread count of a JVM

TYPE jvm_threads_started_total counter

jvm_threads_started_total 298.0

HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers

TYPE jvm_threads_deadlocked gauge

jvm_threads_deadlocked 0.0

HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors

TYPE jvm_threads_deadlocked_monitor gauge

jvm_threads_deadlocked_monitor 0.0

HELP jvm_threads_state Current count of threads by state

TYPE jvm_threads_state gauge

jvm_threads_state{state="NEW",} 0.0 jvm_threads_state{state="TERMINATED",} 0.0 jvm_threads_state{state="RUNNABLE",} 48.0 jvm_threads_state{state="BLOCKED",} 95.0 jvm_threads_state{state="WAITING",} 41.0 jvm_threads_state{state="TIMED_WAITING",} 13.0 jvm_threads_state{state="UNKNOWN",} 0.0

HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.

TYPE jvm_memory_objects_pending_finalization gauge

jvm_memory_objects_pending_finalization 0.0

HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.

TYPE jvm_memory_bytes_used gauge

jvm_memory_bytes_used{area="heap",} 4.059264E8 jvm_memory_bytes_used{area="nonheap",} 2.0814068E8

HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.

TYPE jvm_memory_bytes_committed gauge

jvm_memory_bytes_committed{area="heap",} 2.43269632E9 jvm_memory_bytes_committed{area="nonheap",} 2.13962752E8

HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.

TYPE jvm_memory_bytes_max gauge

jvm_memory_bytes_max{area="heap",} 6.442450944E9 jvm_memory_bytes_max{area="nonheap",} -1.0

HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.

TYPE jvm_memory_bytes_init gauge

jvm_memory_bytes_init{area="heap",} 1.073741824E9 jvm_memory_bytes_init{area="nonheap",} 7667712.0

HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.

TYPE jvm_memory_pool_bytes_used gauge

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1467008.0 jvm_memory_pool_bytes_used{pool="Metaspace",} 1.45059368E8 jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 3.045696E7 jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 1.990704E7 jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 2.08666624E8 jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 1.9096832E8 jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 6291456.0 jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 1.1250304E7

HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.

TYPE jvm_memory_pool_bytes_committed gauge

jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0 jvm_memory_pool_bytes_committed{pool="Metaspace",} 1.47656704E8 jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 3.1522816E7 jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 2.0758528E7 jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 3.00941312E8 jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 2.125463552E9 jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 6291456.0 jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 1.14688E7

HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.

TYPE jvm_memory_pool_bytes_max gauge

jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0 jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0 jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8 jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9 jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0 jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 6.442450944E9 jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0 jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8

HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.

TYPE jvm_memory_pool_bytes_init gauge

jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0 jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0 jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0 jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0 jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 5.6623104E7 jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.01711872E9 jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0 jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0

HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.

TYPE jvm_memory_pool_collection_used_bytes gauge

jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0 jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 0.0 jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 6291456.0

HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.

TYPE jvm_memory_pool_collection_committed_bytes gauge

jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 3.00941312E8 jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 0.0 jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 6291456.0

HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.

TYPE jvm_memory_pool_collection_max_bytes gauge

jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0 jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 6.442450944E9 jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0

HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.

TYPE jvm_memory_pool_collection_init_bytes gauge

jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 5.6623104E7 jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.01711872E9 jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0

HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.

TYPE jvm_buffer_pool_used_bytes gauge

jvm_buffer_pool_used_bytes{pool="mapped",} 0.0 jvm_buffer_pool_used_bytes{pool="direct",} 1965074.0

HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.

TYPE jvm_buffer_pool_capacity_bytes gauge

jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0 jvm_buffer_pool_capacity_bytes{pool="direct",} 1965074.0

HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.

TYPE jvm_buffer_pool_used_buffers gauge

jvm_buffer_pool_used_buffers{pool="mapped",} 0.0 jvm_buffer_pool_used_buffers{pool="direct",} 131.0

HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.

TYPE jmx_config_reload_failure_created gauge

jmx_config_reload_failure_created 1.670582979894E9

HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.

TYPE jmx_config_reload_success_created gauge

jmx_config_reload_success_created 1.670582979893E9

HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.

TYPE jvm_memory_pool_allocated_bytes_created gauge

jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.670582981099E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.670582981106E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.670582981107E9

[carao2020]

Dockerfile.txt

[carao2020]

jmx-config-local.yaml.txt

[sfc-gh-japatel]

Umm, I dont see anything wrong with your config This is what I do it in tests:

KAFKA_HEAP_OPTS="-Xms512m -Xmx10g" KAFKA_OPTS="-Djava.io.tmpdir=/tmp/jdbclogs/ -javaagent:/Users/japatel/Desktop/Kafka-Connector/prometheus/jmx_prometheus_javaagent-0.16.0.jar=8085:/Users/japatel/Desktop/Kafka-Connector/prometheus/kafka-connect-no-pattern.yml" JMX_PORT=9876 $CONFLUENT_FOLDER_NAME/bin/connect-distributed $SNOWFLAKE_APACHE_CONFIG_PATH/$SNOWFLAKE_KAFKA_CONNECT_CONFIG > $APACHE_LOG_PATH/kc.log 2>&1 &

Ignore the env variables, but you use KAFKA_JMX_OPTS and I use KAFKA_OPTS before starting kafka connect. Not sure if this would help..

Another thing you can verify:

1. There is a logger page for all jmx metrics being emitted, you can search for that. It was http://localhost:8085/metrics on my local machine.
2. Make sure you start kafka connector and ingest some data. it needs at least one record from kafka to start emitting metrics.

[sfc-gh-japatel]

You can try this simple no pattern yml file too:

---
lowercaseOutputName: true
lowercaseOutputLabelNames: true

[sfc-gh-japatel]

for you port 9404/metrics

[sfc-gh-tzhang]

Closing, let us know if you need further help