search

roflcoopter / viseron

Self-hosted, local only NVR and AI Computer Vision software. With features such as object detection, motion detection, face recognition and more, it gives you the power to keep an eye on your home, office or any other place you want to monitor.
MIT License
1.66k stars 171 forks source link

Viseron is bombing MQTT Messages in a loop #614

Open lyncker opened 1 year ago

lyncker commented 1 year ago

Hi , I am running Viseron with 3 cams sending MQTT messages to my broker service. I noticed that sometimes , if someone was doing something in front of the cams a lot of images are generated and send within MQTT. Everything allright so far .. But after such a high message rush, viseron seems to be continue sending MQTT messages... I noticed it after some hours because my receiving fhem server was lagging due to overload with these messages. Next time I will continue here to provide you with more detail informations like logs etc... but the question is : Is there something easy I can adjust like "do not send more then 1 Message / second " ?

here is what I found in the logs: 

[2023-05-29 19:12:34] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():
RuntimeError: dictionary changed size during iteration
[2023-05-29 19:12:36] [ERROR   ] [viseron.components.nvr.nvr.camera_2] - Failed to retrieve result for object_detector
[2023-05-29 19:12:39] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting
[2023-05-29 19:28:34] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():
RuntimeError: dictionary changed size during iteration
[2023-05-29 19:28:39] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting
[2023-05-29 20:24:34] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():
RuntimeError: dictionary changed size during iteration
[2023-05-29 20:24:36] [ERROR   ] [viseron.components.nvr.nvr.camera_2] - Failed to retrieve result for object_detector
[2023-05-29 20:24:36] [ERROR   ] [viseron.components.nvr.nvr.camera_1] - Failed to retrieve result for object_detector
[2023-05-29 20:24:39] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting
[2023-05-29 20:27:33] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():

and here I have a log part - you can see what kind of MQTT messages are beeing send nonstop: 

2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 118.7
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 118.7
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 118.7
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 15.6
2023-05-29 20:45:41 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 135.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 135.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 135.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 15.6
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 16.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 131.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_garage 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 16.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 131.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse 15.9
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 16.0
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-05-29 20:45:42 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 131.9

Is there anything I can adjust to avoid this? After a restart of Viseron everything works fine for the first time

roflcoopter commented 1 year ago

I saw your discussion about this but was awaitng the logs, so thanks for that!

First if all, it seems that you are not running the latest version, could you update to 2.2.0? The error in your logs, RuntimeError: dictionary changed size during iteration is fixed there.

The only image sent over MQTT is the latest_thumbnail which is only generated and sent once when a recording starts, which should not flood your MQTT server. The FPS sensor that you see in your MQTT log is only updated once every 30 seconds.

Try updating and see if that solves anything, if not, it would be great if you could catch the logs when the flooding begins

lyncker commented 1 year ago

as suggested, I updated to the last version, but the problem persists. Here are some new logs: 

[2023-06-28 21:47:39] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 135, in run_callbacks
    thread.start()
  File "/usr/lib/python3.8/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
[2023-06-28 21:47:41] [ERROR   ] [viseron.components.nvr.nvr.camera_1] - Failed to retrieve result for object_detector
[2023-06-28 21:47:53] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting
[2023-06-28 21:47:55] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 9
[2023-06-28 21:47:56] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 8
[2023-06-28 21:47:57] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 7
[2023-06-28 21:47:58] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 6
[2023-06-28 21:47:59] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 5
[2023-06-28 21:48:00] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 4
[2023-06-28 21:48:01] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 3
[2023-06-28 21:48:02] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 2
[2023-06-28 21:48:03] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 1
[2023-06-28 21:48:04] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Stopping recording in: 0
[2023-06-28 21:48:04] [INFO    ] [viseron.components.nvr.nvr.camera_3] - Pausing motion detector
[2023-06-28 21:48:04] [INFO    ] [viseron.components.ffmpeg.recorder.camera_3] - Stopping recorder
[2023-06-28 21:56:47] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():
RuntimeError: dictionary changed size during iteration
[2023-06-28 21:56:53] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting
[2023-06-28 21:57:17] [ERROR   ] [root] - Uncaught thread exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/src/viseron/components/data_stream/__init__.py", line 185, in consume_data
    self.static_subscriptions(data_item)
  File "/src/viseron/components/data_stream/__init__.py", line 165, in static_subscriptions
    self.run_callbacks(
  File "/src/viseron/components/data_stream/__init__.py", line 122, in run_callbacks
    for callback in callbacks.values():
RuntimeError: dictionary changed size during iteration
[2023-06-28 21:57:23] [ERROR   ] [viseron.watchdog.thread_watchdog] - Thread data_stream is dead, restarting

the last entry was written 2 hours ago from now ...!

and here what in my mqtt server is arriving: 

2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse 18.6
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 165.2
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:38 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 185.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: garage Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: garage Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 185.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 185.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: garage Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 23.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 165.2
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 165.2
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 165.2
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 18.6
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 18.7
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 160.0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_vorn Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 18.6
2023-06-28 22:36:39 at ScriptAnwesendCheck Next: 22:37:34
2023-06-28 22:36:39 at atVictronOutputPow Next: 22:36:38
2023-06-28 22:36:39 ModbusAttr VictronMP DailyPowerToBattery: 5252.79166667171
2023-06-28 22:36:39 ModbusAttr JKBMS AlarmCellInbalance: 0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_preprocessor_fps: 204.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_name: terrasse_hinten Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_inference_fps: 16.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn 16.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_theoretical_max_fps: 16.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse_vorn attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage 16.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_theoretical_max_fps: 16.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_name: terrasse_hinten Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_preprocessor_fps: 204.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_garage attributes_inference_fps: 16.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 16.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_hinten Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 204.3
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 16.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 16.3
2023-06-28 22:36:39 Shelly ShellyBalkon Power_GUI: 0
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_domain: sensor
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse 18.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_theoretical_max_fps: 18.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_preprocessor_fps: 203.4
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_name: terrasse_hinten Object Detector FPS
2023-06-28 22:36:39 MQTT2_DEVICE MQTT2_viseron_terrasse attributes_inference_fps: 18.5

as told before these messages are arriving in a permanent loop, and the main process of my FHEM Server is consuming 100% CPU... When I reboot viseron the problem is resolved for some days...

lyncker commented 1 year ago

hmm ... no ideas with his problem ? Arent there any things I can try? Am I the only person with such a problem?

roflcoopter commented 1 year ago

I have not had any other reports like this unfortunately. What could be helpful for me is if you can redirect the logs for both Viseron and your MQTT server to files, that way we can see exactly what happens before the flooding begins.

For docker you can do this: docker logs viseron>& viseron.log

roflcoopter commented 1 year ago

I just now noticed the RuntimeError: can't start new thread error in your log. Seems that Viseron is spawning more threads than your machine can handle, that might then spiral into some edge case, causing the flood.

I just pushed a new version to dev which includes some extra safe guards against that. Can you pull latest dev and see if there are any improvements?

lyncker commented 1 year ago

wow cool ... im just pulling the dev version ... it will take some days , maybe some weeks to verify if the problem persists or not ... thanks a lot!

lyncker commented 1 year ago

unfortunately it happend 2 more times... yesterday and today... I cant see this "cant start thread" message anymore in the logs... I also did not see any crashes or something strange in the logs... Is there anything I can adjust the logs to see more?

roflcoopter commented 1 year ago

Sad to hear that :(

Could you enable debug logging? 

logger:
  default_level: debug

And also redirect the logs to a file so we can see the whole event process like this: docker logs viseron>& viseron.log