crash loop with google coral metric

[amigthea]

The container go in a crash loop if frigate push the google coral's metric to the mqtt server

INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_inference_speed
INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_detection_start
INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_pid
INFO:mqtt-exporter:creating prometheus metric: mqtt_detection_fps
INFO:mqtt-exporter:creating prometheus metric: mqtt_cpu_usages_Tasks:_mem
Traceback (most recent call last):
  File "/opt/mqtt-exporter/exporter.py", line 5, in <module>
    main()
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 342, in main
    client.loop_forever()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3327, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 297, in expose_metrics
    _parse_metrics(payload, topic, userdata["client_id"])
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 128, in _parse_metrics
    _parse_metrics(value, topic, client_id, f"{prefix}{metric}_")
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 128, in _parse_metrics
    _parse_metrics(value, topic, client_id, f"{prefix}{metric}_")
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 145, in _parse_metrics
    _create_prometheus_metric(prom_metric_name)
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 71, in _create_prometheus_metric
    prom_metrics[prom_metric_name] = Gauge(
  File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 365, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 130, in __init__
    raise ValueError('Invalid metric name: ' + self._name)
ValueError: Invalid metric name: mqtt_cpu_usages_%Cpu:_mem

[amigthea]

I need at least a workaround to exclude certain metric to be picked up by mqtt-exporter, how can I achieve that?

[kpetremann]

hello @amigthea,

Thanks for raising the issue.

mqtt_cpu_usages_%Cpu:_mem means that your original topic was cpu_usages_%Cpu:_mem. % is not a valid character for Prometheus. See details here.

[kpetremann]

56 should fix your issue. Could you please try?

[amigthea]

hello @amigthea,

Thanks for raising the issue.

mqtt_cpu_usages_%Cpu:_mem means that your original topic was cpu_usages_%Cpu:_mem. % is not a valid character for Prometheus. See details here.

56 should fix your issue. Could you please try?

hi @kpetremann and thank you for joining the discussion It's indeed a kind of metric not parsed for prometheus, I was trying to figure out a workaround on how to esclude that topic with the MQTT_IGNORED_TOPICS enviroment variable, but I'll gladly try out that fix instead! I have a docker compose mqtt-exporter setup, how could I implement that pull request?

[kpetremann]

To use the filter you simply need to put the original topic in: MQTT_IGNORED_TOPICS. I don't know what is the original topic in your case from the info you provided.

I have a docker compose mqtt-exporter setup, how could I implement that pull request?

The easiest way is to

• git clone the repository
• checkout the crash_bad_char_metric_name branch
• run the docker compose with docker-compose.dev.yml file

As an example, I have myself a docker-compose with both stable mqtt-exporter + the dev version built from the disk:

version: "3"
  services:

    mqtt-exporter:
      image: kpetrem/mqtt-exporter
    environment:
      ...

    mqtt-exporter-dev:
      build: /home/kevin/dev/mqtt-exporter
      environment:
      ...

[amigthea]

That's awesome, thank you for that workflow! However now I get a KeyError instead of a ValueError

INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_inference_speed
INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_detection_start
INFO:mqtt-exporter:creating prometheus metric: mqtt_detectors_coral_pid
INFO:mqtt-exporter:creating prometheus metric: mqtt_detection_fps
INFO:mqtt-exporter:creating prometheus metric: mqtt_cpu_usages_Tasks:_mem
INFO:mqtt-exporter:creating prometheus metric: mqtt_cpu_usages_Cpu:_mem
Traceback (most recent call last):
  File "/opt/mqtt-exporter/exporter.py", line 5, in <module>
    main()
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 365, in main
    client.loop_forever()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3327, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 320, in expose_metrics
    _parse_metrics(payload, topic, userdata["client_id"])
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 151, in _parse_metrics
    _parse_metrics(value, topic, client_id, f"{prefix}{metric}_")
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 151, in _parse_metrics
    _parse_metrics(value, topic, client_id, f"{prefix}{metric}_")
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 171, in _parse_metrics
    _add_prometheus_sample(topic, prom_metric_name, metric_value, client_id)
  File "/opt/mqtt-exporter/mqtt_exporter/main.py", line 105, in _add_prometheus_sample
    prom_metrics[prom_metric_name].labels(**labels).set(metric_value)
KeyError: 'mqtt_cpu_usages_%Cpu:_mem'
mqtt-exporter-dev exited with code 0

[kpetremann]

ok I see. I'll fix that. @amigthea could you provide the original MQTT message please? It could help to add a functional test.

[kpetremann]

actually it is not needed. I have an easy way to test that.

[kpetremann]

should be better now. Could you git pull and try again?

[amigthea]

perfect! it doesn't crash anymore, thank you for this amazing fix!

[kpetremann]

you are welcome. This has been included in the release 1.2.0