search

nickbabcock / OhmGraphite

Expose hardware sensor data to Graphite / InfluxDB / Prometheus / Postgres / Timescaledb
Other
426 stars 38 forks source link

Metrics exposed in invalid format: ReadObject: expect { or , or } or n, but found #, #449

Closed mpkonmbk closed 3 months ago

mpkonmbk commented 3 months ago

Here's the output I got from Grafana when trying to add the OhmGraphite Prometheus data source:

ReadObject: expect { or , or } or n, but found #, error found in #1 byte of ...|# HELP dotn|..., bigger context ...|# HELP dotnet_collection_count_total GC collection |... - There was an error returned querying the Prometheus API.

cURLing the endpoint, I can see some gibberish indeed, before finally the output shows some metrics:

/ # curl http://host.docker.internal:9090/metrics
# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="0"} 0
dotnet_collection_count_total{generation="1"} 0
dotnet_collection_count_total{generation="2"} 0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1720625680.7734044
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.453125
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2204141223936
# HELP process_working_set_bytes Process working set
# TYPE process_working_set_bytes gauge
process_working_set_bytes 101031936
# HELP process_private_memory_bytes Process private memory size
# TYPE process_private_memory_bytes gauge
process_private_memory_bytes 45342720
# HELP process_open_handles Number of open handles
# TYPE process_open_handles gauge
process_open_handles 553
# HELP process_num_threads Total number of threads
# TYPE process_num_threads gauge
process_num_threads 22
# HELP dotnet_total_memory_bytes Total known allocated memory
# TYPE dotnet_total_memory_bytes gauge
dotnet_total_memory_bytes 5714800
# HELP prometheus_net_metric_families Number of metric families currently registered.
# TYPE prometheus_net_metric_families gauge
prometheus_net_metric_families{metric_type="counter"} 3
prometheus_net_metric_families{metric_type="gauge"} 39
prometheus_net_metric_families{metric_type="summary"} 0
prometheus_net_metric_families{metric_type="histogram"} 0
# HELP prometheus_net_metric_instances Number of metric instances currently registered across all metric families.
# TYPE prometheus_net_metric_instances gauge
prometheus_net_metric_instances{metric_type="counter"} 4
prometheus_net_metric_instances{metric_type="gauge"} 244
prometheus_net_metric_instances{metric_type="summary"} 0
prometheus_net_metric_instances{metric_type="histogram"} 0
# HELP prometheus_net_metric_timeseries Number of metric timeseries currently generated from all metric instances.
# TYPE prometheus_net_metric_timeseries gauge
prometheus_net_metric_timeseries{metric_type="counter"} 4
prometheus_net_metric_timeseries{metric_type="gauge"} 244
prometheus_net_metric_timeseries{metric_type="summary"} 0
prometheus_net_metric_timeseries{metric_type="histogram"} 0
# HELP prometheus_net_exemplars_recorded_total Number of exemplars that were accepted into in-memory storage in the prometheus-net SDK.
# TYPE prometheus_net_exemplars_recorded_total counter
prometheus_net_exemplars_recorded_total 0
# HELP prometheus_net_eventcounteradapter_sources_connected_total Number of event sources that are currently connected to the adapter.
# TYPE prometheus_net_eventcounteradapter_sources_connected_total gauge
prometheus_net_eventcounteradapter_sources_connected_total 2
# HELP prometheus_net_meteradapter_instruments_connected Number of instruments that are currently connected to the adapter.
# TYPE prometheus_net_meteradapter_instruments_connected gauge
prometheus_net_meteradapter_instruments_connected 0
# HELP ohm_cpu_load_percent Metric reported by open hardware sensor
# TYPE ohm_cpu_load_percent gauge
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #2 Thread #2",hw_instance="0"} 4.675263404846191
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #7",hw_instance="0"} 11.800318717956543
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #9",hw_instance="0"} 13.17574405670166
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Total",hw_instance="0"} 20.554088592529297
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #4 Thread #1",hw_instance="0"} 34.832942962646484
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #6 Thread #2",hw_instance="0"} 6.812745094299316
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #10",hw_instance="0"} 12.631082534790039
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #8",hw_instance="0"} 11.979734420776367
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #1 Thread #1",hw_instance="0"} 46.9296875
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #5 Thread #2",hw_instance="0"} 12.011045455932617
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core Max",hw_instance="0"} 61.38889694213867
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #12",hw_instance="0"} 13.739824295043945
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #14",hw_instance="0"} 10.545879364013672
ohm_cpu_load_percent{hardware="13th Gen Intel Core i5-13600K",sensor="CPU Core #13",hw_instance="0"} 12.383783340454102
. . .

any help is much appreciated!

nickbabcock commented 3 months ago

The prometheus output starts with the following?

/ # curl http://host.docker.internal:9090/metrics

Is that the gibberish?

mpkonmbk commented 3 months ago

Sorry, I was under the impression, based on the error output, that the # HELP and # TYPE annotations are not valid - and what Grafana doesn't like. I also haven't seen anything like it in my (perhaps limited) experience with Prometheus.

The initial line of course is not part of the output, just wanted to be clear where did I get the output from.

I have since switched to using Graphite, but I'd much rather use Prometheus as I find its data format and ecosystem much superior in my use case. If there's something you can tell I am missing in my config or whatever, I'd appreciate the support!

nickbabcock commented 3 months ago

and what Grafana doesn't like

The output from the prometheus endpoint in OhmGraphite is for prometheus, not for Grafana (ie: you don't point Grafana directly at OhmGraphtie but instead at an prometheus instance). So Grafana should never see the # Help and # Type output.

Prometheus has a builtin expression browser if you want to verify that OhmGraphite data is being stored: https://prometheus.io/docs/prometheus/latest/getting_started/#using-the-expression-browser

mpkonmbk commented 3 months ago

Indeed, putting an actual Prometheus server between OhmGraphite and Grafana "fixed" the issue (though there's no fixing me being lazy for not doing that in the first place). Thanks for the assist :v: and for this incredibly useful tool