Question - how to understand/debug verify issue

[runofthemill]

I was running into an issue where exporter_exporter couldn't verify the metrics being exported by https://github.com/prometheus/jmx_exporter (v 0.18.0 running the standalone http server) and was able to resolve it by setting verify: false in the exporter config in expexp.yaml.

Though disabling verify works, I'm hoping to understand the underlying issue; a sample log entry from the exporter_exporter service looks like this:

Apr 17 16:45:14 api0 exporter_exporter[677967]: time="2023-04-17T16:45:14-06:00" level=error msg="Verification for module 'core_api-jmx' failed: Failed to decode metrics from proxied server: text format parsing error in line 12: unknown metric
 type \"unknown\""

And the (truncated) output from the endpoint:

runofthemill@api0:~$ curl http://localhost:9999/proxy?module=core_api-jmx
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.18.0",name="jmx_prometheus_httpserver",} 1.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP java_lang_MemoryPool_UsageThresholdSupported java.lang:name=Metaspace,type=MemoryPool,attribute=UsageThresholdSupported
# TYPE java_lang_MemoryPool_UsageThresholdSupported untyped
java_lang_MemoryPool_UsageThresholdSupported{name="Metaspace",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="Code Cache",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="Compressed Class Space",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="G1 Eden Space",} 0.0
java_lang_MemoryPool_UsageThresholdSupported{name="G1 Old Gen",} 1.0
java_lang_MemoryPool_UsageThresholdSupported{name="G1 Survivor Space",} 0.0

Line 12 seems to be untyped from the preceding annotation, which (I think) should be okay? Any suggestions on how I can better understand why the verification is failing?

Thank you!

[tcolgate]

I've seen some mention of the JMX exporter and OpenMetrics (the prom expfmt parser actually doesn't support OpenMetrics, it's an outstanding feature request). One possible approach to debugging, though a bit rubbish, would be to put this text into a file, then use the recent support for file plugins.

[tcolgate]

FWIW, with the current build, if I put your data into a file called data.txt, and use this config

modules:
  data:
    method: file
    file:
      path: data.txt

I can then read the metrics with /proxy?module=data. It could be that there has been a recent fix for this?

[runofthemill]

I'll try that out! I'm using 0.5.0 but did have to build from source due to an issue w/ glibc that I couldn't quite figure out, so that might be a confounding variable.

Thanks for the suggestion :)