search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.58k stars 29.91k forks source link

Recorder 30000 queue reach in two hours 2021.9 #56987

Closed egranto closed 2 years ago

egranto commented 2 years ago

The problem

I'm getting this error "The recorder queue reached the maximum size of 30000; Events are no longer being recorded"

I had a heavy logging system which was working fine previews version 2021.9.8 and OS 6.4

I had 1140 sensors logging between 1 minute and 1 hour

What is version of Home Assistant Core has the issue?

core-2021.9.7

What was the last working version of Home Assistant Core?

core-2021.8.8

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Recorder

Link to integration documentation on our website

https://www.home-assistant.io/integrations/recorder/

Example YAML snippet

recorder:
  db_url: !secret maria_db
  db_retry_wait: 20
  db_max_retries: 30  
  auto_purge: true
  purge_keep_days: 7
  commit_interval: 1
  exclude:
    entities:
      - sun.sun
    domains:
      - alarm_control_panel
      - light
      - automation
      - binary_sensor
      - updater
      - calendar
      - camera
      - device_tracker
      - group
      - image_processing
      - input_boolean
      - input_datetime
      - input_number
      - input_select
      - input_text
      - person
      - proximity
      - sun
      - utility_meter
    entity_globs:
      - sensor.*_time  
      - sensor.*time* 
      - sensor.*uptim*
      - sensor.*_current 
      - sensor.*_movimiento
    event_types:
      - call_service # Don't record service calls
      - deepstack.object_detected

Anything in the logs that might be useful for us?

2021-10-02 20:20:28 ERROR (MainThread) [homeassistant.components.recorder] The recorder queue reached the maximum size of 30000; Events are no longer being recorded

2021-10-03 04:10:06 WARNING (Thread-10785) [homeassistant.util.executor] Thread[SyncWorker_34] is still running at shutdown: File "/usr/local/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 75, in _worker
    work_item = work_queue.get(block=True)

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 446, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 641, in async_device_update
    raise exc
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/command_line/sensor.py", line 127, in update
    self._state = self._value_template.render_with_possible_json_value(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 527, in render_with_possible_json_value
    return run_callback_threadsafe(
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 58, in run_callback
    future.set_result(callback(*args))
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 557, in async_render_with_possible_json_value
    return _render_with_context(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1477, in _render_with_context
    return template.render(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1395, in regex_findall_index
    return re.findall(find, value, flags)[index]
IndexError: list index out of range

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 446, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 641, in async_device_update
    raise exc
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/command_line/sensor.py", line 127, in update
    self._state = self._value_template.render_with_possible_json_value(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 527, in render_with_possible_json_value
    return run_callback_threadsafe(
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 58, in run_callback
    future.set_result(callback(*args))
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 557, in async_render_with_possible_json_value
    return _render_with_context(
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1477, in _render_with_context
    return template.render(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1395, in regex_findall_index
    return re.findall(find, value, flags)[index]
IndexError: list index out of range

2021-10-02 20:51:06 ERROR (MainThread) [homeassistant] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)

Additional information

data is been past trough to influxdb where i store long term data, but recorder even the default one i'm getting this error

i remove all ENERGY stuff because i was thinking it was all the statistics the system create but don't i'm still couldn't find the cause

probot-home-assistant[bot] commented 2 years ago

recorder documentation recorder source (message by IssueLinks)

rostislav-palivoda commented 2 years ago

Did you check disk performance?

egranto commented 2 years ago

Yes first was on a SSD 256gb, now is on my 4tb RAID 0 NAS with M2 cache, i also run it with the default db same issue, two different instances of MariaDB one local on the virtual and the normal i run it on my Nas since over two years now

image

NO slow queries either

image

image

rostislav-palivoda commented 2 years ago

Do you have slow queries or errors in MariaDB? https://mariadb.com/kb/en/slow-query-log-overview/ It looks like memory and CPU usage is high, do you know why?

From version core-2021.8.8 to core-2021.9.7 the Recorder had changes for statisitcs https://github.com/home-assistant/core/commits/dev/homeassistant/components/recorder

rostislav-palivoda commented 2 years ago

Is it always fails on the same command line sensor when parsing json? ...from the web?

egranto commented 2 years ago

Memory an Cpu are high because the virtual environment is on the same system i run my Home assistant with 8gb plus 3 cores dedicate to it also run MariaDB i had a Synology +918 with 16gb an 1tb m2 cache, the main os of my synology runs on a the SSD

image

no slow queries trigger as you can see on the graph,

the one i see weird on my logs is this

` Logger: homeassistant Source: /usr/src/homeassistant/homeassistant/runner.py:87 First occurred: 6:19:41 AM (4 occurrences) Last logged: 7:49:41 AM

Error doing job: Future exception was never retrieved Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) TypeError: update() takes 1 positional argument but 2 were given

`

the error of template is because is trying to divide by zero that happends at soon as i reload the system because some sensors still haven't collect data

rostislav-palivoda commented 2 years ago

Is it always the same command line sensor exception when you reach recorder limit?

Why you setup retry wait to 20 when default is 3? 

 db_retry_wait: 20
 db_max_retries: 30

Do you see any dropped connections to the DB?

egranto commented 2 years ago

i add those to see if giving more time will solve the problem but all start with default values, because on the past i wasn't even excluding anything and my system was able to handle it.

no is random the command line that check the motion of my camera is the one the most trigger on my system that is why appear there in between (scan every 5 seconds)

image the one is constant and honestly i can't relate what is that trigger this error (see for example had been increase the count during this topic of today)

No error on the log related to connection drop i even stop my db to see if there where a connection, because i was thinking was disconnect for some reason that was why it was using the queue system (which i remember Paul explain it in one release party), so my first test was that, but at soon as i restart the db, i saw the error of mysql missing, which makes install the MariaDB add on, thinking it was something with my external maria, same issue, then i think well maybe is something with mysql engine, then i run the default for a few hours until it also fail, so i go back to my MariaDb on Synology start playing with times and excluding what i really don't need stored, in order to see if i'm able to reduce the amount queue generate, but i still not able to find it, that is when i decide open the issue, because on the 2 years that i had running Home Assistant i had never seen that error of runner, as you mention on the last releases they had change the way of the data is been log, and i know my system is heavy logging (i can't use the default db system because it logs around 1gb peer hour while on mysql i got around 20% of that or less )

the issue is not data collection because i also run influxdb as long storage and the data is getting there

rostislav-palivoda commented 2 years ago

All this excludes you have in the list.... was added with idea to reduce load to DB and see if new version can run with at least something? Can you exclude all and see if recorder does not hit the limit?

I would propose to run recorder in debug mode and see logs, because in debug it has output of queue size and other useful messages: https://github.com/home-assistant/core/blob/2f960e558f025d0e5f414c4e54058f96ff4e33dd/homeassistant/components/recorder/__init__.py#L440

You can find logger setup here - https://www.home-assistant.io/integrations/logger/

Probably Recorder debug log could give an idea what might be wrong.

egranto commented 2 years ago

image

2021-10-04 14:24:51 DEBUG (Recorder) [homeassistant.components.recorder] Connected to recorder database ... 2021-10-04 14:34:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3761 ... 2021-10-04 14:44:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 5397 ... 2021-10-04 14:54:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 6722 ... 2021-10-04 15:04:52 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 7314

egranto commented 2 years ago

image

2021-10-04 15:09:32 DEBUG (Recorder) [homeassistant.components.recorder] Connected to recorder database ... 2021-10-04 15:19:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0

egranto commented 2 years ago

image

2021-10-04 15:24:24 DEBUG (Recorder) [homeassistant.components.recorder] Connected to recorder database ... 2021-10-04 15:34:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1776

image

egranto commented 2 years ago

2021-10-04 15:34:04 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive 2021-10-04 15:34:16 DEBUG (SyncWorker_5) [homeassistant.components.recorder.util] querying 0 rows took 0.019838s 2021-10-04 15:34:16 DEBUG (SyncWorker_37) [homeassistant.components.recorder.util] querying 0 rows took 0.070096s 2021-10-04 15:34:17 DEBUG (SyncWorker_35) [homeassistant.components.recorder.util] querying 12 rows took 0.153912s 2021-10-04 15:34:17 DEBUG (SyncWorker_30) [homeassistant.components.recorder.util] querying 5 rows took 0.149738s 2021-10-04 15:34:17 DEBUG (SyncWorker_5) [homeassistant.components.recorder.util] querying 1 rows took 0.141629s 2021-10-04 15:34:17 DEBUG (SyncWorker_5) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.142982s 2021-10-04 15:34:17 DEBUG (SyncWorker_37) [homeassistant.components.recorder.util] querying 1 rows took 0.132156s 2021-10-04 15:34:17 DEBUG (SyncWorker_37) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.132852s 2021-10-04 15:34:17 DEBUG (SyncWorker_30) [homeassistant.components.recorder.util] querying 1 rows took 0.054713s 2021-10-04 15:34:17 DEBUG (SyncWorker_30) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.055522s 2021-10-04 15:34:17 DEBUG (SyncWorker_35) [homeassistant.components.recorder.util] querying 1 rows took 0.082582s 2021-10-04 15:34:17 DEBUG (SyncWorker_35) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.083627s 2021-10-04 15:34:17 DEBUG (SyncWorker_41) [homeassistant.components.recorder.util] querying 5 rows took 0.218668s 2021-10-04 15:34:17 DEBUG (SyncWorker_37) [homeassistant.components.recorder.util] querying 1 rows took 0.019072s 2021-10-04 15:34:17 DEBUG (SyncWorker_9) [homeassistant.components.recorder.util] querying 0 rows took 0.252842s 2021-10-04 15:34:17 DEBUG (SyncWorker_27) [homeassistant.components.recorder.util] querying 0 rows took 0.219833s 2021-10-04 15:34:17 DEBUG (SyncWorker_0) [homeassistant.components.recorder.util] querying 1 rows took 0.127901s 2021-10-04 15:34:17 DEBUG (SyncWorker_9) [homeassistant.components.recorder.util] querying 1 rows took 0.060052s 2021-10-04 15:34:17 DEBUG (SyncWorker_9) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.060552s 2021-10-04 15:34:17 DEBUG (SyncWorker_20) [homeassistant.components.recorder.util] querying 0 rows took 0.292573s 2021-10-04 15:34:17 DEBUG (SyncWorker_29) [homeassistant.components.recorder.util] querying 13 rows took 0.210614s 2021-10-04 15:34:17 DEBUG (SyncWorker_26) [homeassistant.components.recorder.util] querying 16 rows took 0.155691s 2021-10-04 15:34:17 DEBUG (SyncWorker_11) [homeassistant.components.recorder.util] querying 1 rows took 0.141761s 2021-10-04 15:34:17 DEBUG (SyncWorker_36) [homeassistant.components.recorder.util] querying 0 rows took 0.209314s 2021-10-04 15:34:17 DEBUG (SyncWorker_20) [homeassistant.components.recorder.util] querying 1 rows took 0.033214s 2021-10-04 15:34:17 DEBUG (SyncWorker_20) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.033937s 2021-10-04 15:34:17 DEBUG (SyncWorker_27) [homeassistant.components.recorder.util] querying 1 rows took 0.093695s 2021-10-04 15:34:17 DEBUG (SyncWorker_27) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.094157s 2021-10-04 15:34:17 DEBUG (SyncWorker_41) [homeassistant.components.recorder.util] querying 1 rows took 0.114142s 2021-10-04 15:34:17 DEBUG (SyncWorker_41) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.114895s 2021-10-04 15:34:17 DEBUG (SyncWorker_7) [homeassistant.components.recorder.util] querying 0 rows took 0.346837s 2021-10-04 15:34:17 DEBUG (SyncWorker_36) [homeassistant.components.recorder.util] querying 1 rows took 0.035462s 2021-10-04 15:34:17 DEBUG (SyncWorker_36) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.036030s 2021-10-04 15:34:17 DEBUG (SyncWorker_7) [homeassistant.components.recorder.util] querying 1 rows took 0.021692s 2021-10-04 15:34:17 DEBUG (SyncWorker_7) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.022243s 2021-10-04 15:34:17 DEBUG (SyncWorker_25) [homeassistant.components.recorder.util] querying 2 rows took 0.163959s 2021-10-04 15:34:17 DEBUG (SyncWorker_25) [homeassistant.components.recorder.util] querying 1 rows took 0.011950s 2021-10-04 15:34:17 DEBUG (SyncWorker_8) [homeassistant.components.recorder.util] querying 19 rows took 0.369771s 2021-10-04 15:34:17 DEBUG (SyncWorker_11) [homeassistant.components.recorder.util] querying 1 rows took 0.061304s 2021-10-04 15:34:17 DEBUG (SyncWorker_11) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.061907s 2021-10-04 15:34:17 DEBUG (SyncWorker_5) [homeassistant.components.recorder.util] querying 1 rows took 0.237300s 2021-10-04 15:34:17 DEBUG (SyncWorker_7) [homeassistant.components.recorder.util] querying 1 rows took 0.040958s 2021-10-04 15:34:17 DEBUG (SyncWorker_0) [homeassistant.components.recorder.util] querying 1 rows took 0.142082s 2021-10-04 15:34:17 DEBUG (SyncWorker_0) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.142578s 2021-10-04 15:34:17 DEBUG (SyncWorker_25) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.012708s 2021-10-04 15:34:17 DEBUG (SyncWorker_40) [homeassistant.components.recorder.util] querying 15 rows took 0.460616s 2021-10-04 15:34:17 DEBUG (SyncWorker_33) [homeassistant.components.recorder.util] querying 1 rows took 0.225143s 2021-10-04 15:34:17 DEBUG (SyncWorker_40) [homeassistant.components.recorder.util] querying 1 rows took 0.008991s 2021-10-04 15:34:17 DEBUG (SyncWorker_40) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.009770s 2021-10-04 15:34:17 DEBUG (SyncWorker_27) [homeassistant.components.recorder.util] querying 1 rows took 0.109744s 2021-10-04 15:34:17 DEBUG (SyncWorker_26) [homeassistant.components.recorder.util] querying 1 rows took 0.176488s 2021-10-04 15:34:17 DEBUG (SyncWorker_26) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.177137s 2021-10-04 15:34:17 DEBUG (SyncWorker_33) [homeassistant.components.recorder.util] querying 1 rows took 0.039672s 2021-10-04 15:34:17 DEBUG (SyncWorker_33) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.040542s 2021-10-04 15:34:17 DEBUG (SyncWorker_8) [homeassistant.components.recorder.util] querying 1 rows took 0.146697s 2021-10-04 15:34:17 DEBUG (SyncWorker_8) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.147252s 2021-10-04 15:34:17 DEBUG (SyncWorker_36) [homeassistant.components.recorder.util] querying 1 rows took 0.022539s 2021-10-04 15:34:17 DEBUG (SyncWorker_29) [homeassistant.components.recorder.util] querying 1 rows took 0.235781s 2021-10-04 15:34:17 DEBUG (SyncWorker_29) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.236450s 2021-10-04 15:34:17 DEBUG (SyncWorker_9) [homeassistant.components.recorder.util] querying 1 rows took 0.116264s 2021-10-04 15:34:17 DEBUG (SyncWorker_16) [homeassistant.components.recorder.util] querying 0 rows took 0.344815s 2021-10-04 15:34:17 DEBUG (SyncWorker_24) [homeassistant.components.recorder.util] querying 1 rows took 0.377032s 2021-10-04 15:34:17 DEBUG (SyncWorker_16) [homeassistant.components.recorder.util] querying 0 rows took 0.027353s 2021-10-04 15:34:17 DEBUG (SyncWorker_16) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.027927s 2021-10-04 15:34:17 DEBUG (SyncWorker_20) [homeassistant.components.recorder.util] querying 1 rows took 0.121645s 2021-10-04 15:34:17 DEBUG (SyncWorker_24) [homeassistant.components.recorder.util] querying 1 rows took 0.060603s 2021-10-04 15:34:17 DEBUG (SyncWorker_24) [homeassistant.components.recorder.history] getting 1 first datapoints took 0.061147s 2021-10-04 15:34:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1776 2021-10-04 15:34:28 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive

rostislav-palivoda commented 2 years ago

Is there any specific sensor like 'command_line' that causes rapid growth of queue and hit the limit of MAX_QUEUE_BACKLOG = 30000?

egranto commented 2 years ago

2021-10-04 15:44:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 679 .... 2021-10-04 15:54:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 ... 2021-10-04 16:04:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 113 ... 2021-10-04 16:14:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 182 ... 2021-10-04 16:24:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 88 ... 2021-10-04 16:34:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 82 .. 2021-10-04 16:44:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 38 ... 2021-10-04 16:54:24 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 29

i don't looks like is on the sensor domain

egranto commented 2 years ago

image image

2021-10-04 17:11:49 DEBUG (Recorder) [homeassistant.components.recorder] Connected to recorder database ... 2021-10-04 17:21:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2958 ... 2021-10-04 17:31:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3063 .. 2021-10-04 17:41:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2512 ... 2021-10-04 17:51:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3192 2021-10-04 18:01:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 4227 2021-10-04 18:11:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 4614 2021-10-04 18:21:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 6176 2021-10-04 18:31:49 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 7315

i think is on switches domain

egranto commented 2 years ago

Ok this is weird i had to be honest, queue it seems able to empty it self unless is full, i'm not sure what is happening on the back-end, but i wonder why so many first data points, and so many 0 rows

2021-10-04 20:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 19523 2021-10-04 20:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 21206 2021-10-04 20:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 22561 2021-10-04 20:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 21477 2021-10-04 21:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 20831 2021-10-04 21:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 19929 2021-10-04 21:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 18268 2021-10-04 21:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 16863 2021-10-04 21:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 15742 2021-10-04 21:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 14196 2021-10-04 22:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 12180 2021-10-04 22:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 10121 2021-10-04 22:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 9224 2021-10-04 22:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 7100 2021-10-04 22:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3332 2021-10-04 22:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 882 2021-10-04 23:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 47 2021-10-04 23:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 7 2021-10-04 23:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-04 23:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-04 23:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 35 2021-10-04 23:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1 2021-10-05 00:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 721 2021-10-05 00:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2714 2021-10-05 00:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1613 2021-10-05 00:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 330 2021-10-05 00:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 293 2021-10-05 00:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 54 2021-10-05 01:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 35 2021-10-05 01:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 33 2021-10-05 01:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 9 2021-10-05 01:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 49 2021-10-05 01:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2 2021-10-05 01:51:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 27 2021-10-05 02:01:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 10 2021-10-05 02:11:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 6 2021-10-05 02:21:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-05 02:31:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-05 02:41:50 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 22 2021-10-05 02:51:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 5 2021-10-05 03:01:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 6 2021-10-05 03:11:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1 2021-10-05 03:21:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 5 2021-10-05 03:31:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 16 2021-10-05 03:41:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2 2021-10-05 03:51:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-05 04:01:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2 2021-10-05 04:11:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-05 04:21:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 827 2021-10-05 04:31:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2055 2021-10-05 04:41:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3472 2021-10-05 04:51:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3155 2021-10-05 05:01:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3093 2021-10-05 05:11:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2366 2021-10-05 05:21:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 3692 2021-10-05 05:31:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 4598 2021-10-05 05:41:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 4657 2021-10-05 05:51:51 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 4891

egranto commented 2 years ago

No spikes on disk image Memory looks stable (mostly reserved by the VM) image CPU is normally high on this system but still had room to handle load image

egranto commented 2 years ago

2021-10-05 07:54:21 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 07:54:21 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 07:54:21 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 980 state ids to remove 2021-10-05 07:54:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 980 states to remove old_state_id 2021-10-05 07:54:25 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 980 states 2021-10-05 07:54:27 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 07:54:27 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet

2021-10-05 08:01:52 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 28081

2021-10-05 08:21:52 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 28976

2021-10-05 08:28:38 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive 2021-10-05 08:28:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 08:28:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 08:28:49 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 973 state ids to remove

2021-10-05 08:31:52 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 31076 2021-10-05 08:31:52 ERROR (MainThread) [homeassistant.components.recorder] The recorder queue reached the maximum size of 30000; Events are no longer being recorded

egranto commented 2 years ago

2021-10-05 09:33:35 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 977 state ids to remove 2021-10-05 09:33:35 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 977 states to remove old_state_id 2021-10-05 09:33:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 977 states 2021-10-05 09:33:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:33:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:33:39 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:33:41 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:33:42 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 977 state ids to remove 2021-10-05 09:33:43 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 977 states to remove old_state_id 2021-10-05 09:33:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 977 states 2021-10-05 09:33:54 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:33:54 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:33:55 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:33:55 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:33:59 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 977 state ids to remove 2021-10-05 09:34:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:34:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:34:39 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:34:42 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:34:44 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 974 state ids to remove 2021-10-05 09:34:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 974 states to remove old_state_id 2021-10-05 09:35:09 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:35:09 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:35:10 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:35:10 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:35:12 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 983 state ids to remove 2021-10-05 09:35:14 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 983 states to remove old_state_id 2021-10-05 09:35:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 983 states

2021-10-05 09:35:36 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:35:38 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 975 state ids to remove 2021-10-05 09:35:40 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 975 states to remove old_state_id 2021-10-05 09:35:40 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 975 states 2021-10-05 09:35:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:35:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:35:47 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:35:48 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:35:48 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 982 state ids to remove 2021-10-05 09:35:48 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 982 states to remove old_state_id 2021-10-05 09:35:52 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 982 states 2021-10-05 09:35:57 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:35:57 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:35:57 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:35:57 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:35:58 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 971 state ids to remove 2021-10-05 09:35:59 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 971 states to remove old_state_id 2021-10-05 09:36:09 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:36:09 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 965 state ids to remove 2021-10-05 09:36:09 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 965 states to remove old_state_id 2021-10-05 09:36:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 965 states 2021-10-05 09:36:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:36:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:36:14 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:36:16 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:36:16 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 955 state ids to remove 2021-10-05 09:36:19 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 954 states to remove old_state_id 2021-10-05 09:36:19 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 955 states 2021-10-05 09:36:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:36:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:36:23 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:36:23 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 998 event ids to remove 2021-10-05 09:36:23 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 978 state ids to remove 2021-10-05 09:36:24 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 978 states to remove old_state_id 2021-10-05 09:37:36 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 998 events 2021-10-05 09:37:36 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet 2021-10-05 09:37:37 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00

looks like purge is looping

rostislav-palivoda commented 2 years ago

There is MAX_ROWS_TO_PURGE in https://github.com/home-assistant/core/blob/0099b5448960cd596329d7ccb08318486698f11c/homeassistant/components/recorder/const.py#L15

Since you are on MariaDB you can try to increase it x 10 and see if queue is cleaning faster. If not then I would suspect something is wrong with performance of SQL DELETE statements.

Therefore I'm curious how long it takes to delete 998 records on your MariaDB, you can try to add miliseconds measurement into logging https://github.com/home-assistant/core/blob/8567aa9e1332bc89fb7b8922f63f0ce89865d977/homeassistant/components/recorder/purge.py#L135

Another option would be to pass Echo=True here https://github.com/home-assistant/core/blob/565a9fea6be97c2facb838701992db9df825d3a9/homeassistant/components/recorder/__init__.py#L993

To set echo=True change code:

Remove:       987 else:
Tab back to execute it always:            988 kwargs["echo"] = False
and set it to True.

Then in log you will see actual SQLs executed on your Maria DB.

There is also another way to enable logging for SQL Alchemy ORM: https://docs.sqlalchemy.org/en/14/core/engines.html#configuring-logging

rostislav-palivoda commented 2 years ago

What version of MariaDB do you use? Do you use InnoDB? Please check https://mariadb.com/kb/en/big-deletes/#innodb-chunking-recommendation

egranto commented 2 years ago

image

Is basically out of the box

rostislav-palivoda commented 2 years ago

Out of the box does not mean it's perfectly configured for 1k+ sensors.

2021-10-05 09:36:14 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00 2021-10-05 09:36:23 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-09-28 10:12:00+00:00

To me it looks like deletion of 1k states and 1k events takes 9 seconds. Sometimes more.

I would recommend to increase defaults on your DB: https://www.percona.com/blog/2007/11/01/innodb-performance-optimization-basics/

... or change purge_keep_days from 7 to e.g. 3 days. Then tables will be smaller and you will fit into your RAM size. Of course to overcome the queue overflow problem you have to delete old data manually before you start HA.

rostislav-palivoda commented 2 years ago

i think is on switches domain

What kind of switches do you have?

egranto commented 2 years ago

TP-Link, Sonoff, Meross, a few DIY from ESP home and Shellys but this are more set as like lights

egranto commented 2 years ago

I was able to handle way bigger DBs on the same system with the same amount of sensors
image

egranto commented 2 years ago

other is that we are unable to handle quick sensor

I just disable the command line sensor of the cameras (foscam)
are 5 of them checking the cgi info every 5 seconds (again not new sensor is even running since 0.### versions) 

  - platform: command_line  
    command: curl -k --silent "http://CAMIP:88/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=USER&pwd=PASSWORD"
    name: "Camera_Cochera_HumanAI"
    value_template: >
          {% set humanAI = value | regex_findall_index('AlarmState>(\d+)</humanDetect') %}
            {% if humanAI == "0" %}
              Disabled
            {%- elif humanAI == "1" -%}
              None
            {%- elif humanAI == "2" -%}
              Detected
            {% else %}
              Not Determined
          {%- endif -%}
    command_timeout: 20   
    scan_interval: 5

2021-10-05 10:31:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 31 2021-10-05 10:41:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 117 2021-10-05 10:51:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 23 2021-10-05 11:01:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 24 2021-10-05 11:11:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 865 2021-10-05 11:21:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2028 2021-10-05 11:31:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 2088 2021-10-05 11:41:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1873 2021-10-05 11:51:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1205 2021-10-05 12:01:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1007 2021-10-05 12:11:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 377 2021-10-05 12:21:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 36 2021-10-05 12:31:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 128 2021-10-05 12:41:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 64 2021-10-05 12:51:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 44 2021-10-05 13:01:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 100 2021-10-05 13:11:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 73 2021-10-05 13:21:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 20 2021-10-05 13:31:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 144 2021-10-05 13:41:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 19 2021-10-05 13:51:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 14 2021-10-05 14:01:32 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 106

rostislav-palivoda commented 2 years ago

Did I understand correct - if the JSON values from command line sensor are not recorded into DB then queue does not growing?

Queue is a set of tasks to execute. If it grows then something is slow. An independent process watch size of the queue and stops recorder when it reaches the limit. I propose to find out slow task in queue. Candidate are: purge task...or calculation of long term statistics.

Another option would be to understand cause of spikes in the queue. If the spikes are temporarily then increasing max queue size could solve your problem.

egranto commented 2 years ago

looks like, also now i had purge disable

image

rostislav-palivoda commented 2 years ago

How long it takes if you manually run SQL on states tables to DELETE a set of records for 1 day?

egranto commented 2 years ago

sure help me with the query because i try this one

delete FROM homeassistant.states WHERE last_updated LIKE '2021-09-20%'

but i got 

SQL query:

delete FROM homeassistant.states WHERE `last_updated` LIKE '2021-09-20%'

MySQL said: Documentation
#1451 - Cannot delete or update a parent row: a foreign key constraint fails (`homeassistant`.`states`, CONSTRAINT `states_ibfk_2` FOREIGN KEY (`old_state_id`) REFERENCES `states` (`state_id`))

By the way without the heavy logging text sensor of my camera movement looks like the queue never gets full

i could consider change those outputs instead of text send numbers and see if that helps

rostislav-palivoda commented 2 years ago

There is cascade FK on old_state_id. Start delete from records where its empty by adding to where clause old_state_id IS NULL. Not sure about last_updated, maybe created is better (need to look in to code how recorder does it). Anyway just delete for one day. Check how much records was deleted and time.

egranto commented 2 years ago

well i Narrow the scope to command line sensors

either the foscam or the other 2 i had one extract data cost of my energy provider, once a day the other extract data from my local weather 

  - platform: command_line  
    command: 'python3 "/config/python_scripts/IMN_Avg_Temp_1h.py"'
    name: "IMN Temperatura Promedio Por Hora"
    unit_of_measurement: '°C' 
    command_timeout: 20   
    scan_interval: 120

the script i use for all my data extractions is this 

try:
    from urllib2 import urlopen
except ImportError:
    from urllib.request import urlopen
from lxml import etree

url =  "https://www.imn.ac.cr/especial/tablas/imn.html"
response = urlopen(url)
htmlparser = etree.HTMLParser()
tree = etree.parse(response, htmlparser)

lluvia = tree.xpath('/html/body/table[1]/tr[2]/td[3]/text()')[0]

x = lluvia.replace(",", ".")
print(x)

i can't use scrape for that because the page had an issue on their source code

  - platform: scrape
    resource: https://www.imn.ac.cr/especial/tablas/imn.html
    name: "IMN Temperatura Promedio Por Hora"
    unit_of_measurement: '°C' 
    select: "body > table:nth-child(2) > tbody:nth-child(1) > tr:nth-child(2) > td:nth-child(2)"
    value_template: '{{ ((value.split(" ")[0]) | replace (",", ".")) }}'

first i was thinking was because heavy use of my foscam motion 5cams each 5 seconds, so i disable all command line sensors to start work for an entire day, then we update version, so i add all back, few hours later i hit the 30000 mark, so i start where we was, remove all command line, works again a day again, so i said ok could be my foscam by heavy use, so i add the other two back, then i got a FULL queue once again, been running without any command line sensor since.

2021-10-10 06:06:41 DEBUG (Recorder) [homeassistant.components.recorder] Recorder processing the queue

2021-10-10 06:11:44 DEBUG (Recorder) [homeassistant.components.recorder.statistics] Compiling statistics for 2021-10-10 12:05:00+00:00-2021-10-10 12:10:00+00:00 2021-10-10 06:11:44 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 577 rows took 0.036625s 2021-10-10 06:11:45 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 522 rows took 0.754330s 2021-10-10 06:11:45 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.754944s 2021-10-10 06:11:45 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 260 first datapoints took 0.009908s 2021-10-10 06:11:46 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 1841 rows took 0.390895s 2021-10-10 06:11:46 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.391528s 2021-10-10 06:11:46 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 325 first datapoints took 0.009935s

2021-10-10 06:15:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1525

2021-10-10 06:16:21 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive 2021-10-10 06:16:30 DEBUG (Recorder) [homeassistant.components.recorder.statistics] Compiling statistics for 2021-10-10 12:10:00+00:00-2021-10-10 12:15:00+00:00 2021-10-10 06:16:30 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 577 rows took 0.047457s 2021-10-10 06:16:31 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 320 rows took 0.269660s 2021-10-10 06:16:31 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.270322s 2021-10-10 06:16:31 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 260 rows took 0.087244s 2021-10-10 06:16:31 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 260 first datapoints took 0.107565s 2021-10-10 06:16:33 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 1931 rows took 2.624454s 2021-10-10 06:16:33 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 2.625708s 2021-10-10 06:16:35 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 325 rows took 1.008057s 2021-10-10 06:16:35 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 325 first datapoints took 1.031884s

2021-10-10 06:25:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 481 2021-10-10 06:35:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 649 2021-10-10 06:45:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 712 2021-10-10 07:05:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 83 2021-10-10 07:15:12 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 32

2021-10-10 07:25:19 DEBUG (Recorder) [homeassistant.components.recorder.statistics] Compiling statistics for 2021-10-10 13:20:00+00:00-2021-10-10 13:25:00+00:00 2021-10-10 07:25:19 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 577 rows took 0.032559s 2021-10-10 07:25:20 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 259 rows took 0.756838s 2021-10-10 07:25:20 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.757716s 2021-10-10 07:25:20 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 260 rows took 0.141780s 2021-10-10 07:25:20 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 260 first datapoints took 0.152013s 2021-10-10 07:25:21 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 2274 rows took 0.821934s 2021-10-10 07:25:21 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.823394s 2021-10-10 07:25:22 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 325 rows took 1.450438s 2021-10-10 07:25:22 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 325 first datapoints took 1.484854s 2021-10-10 07:35:15 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive

2021-10-10 07:35:23 DEBUG (Recorder) [homeassistant.components.recorder.statistics] Compiling statistics for 2021-10-10 13:30:00+00:00-2021-10-10 13:35:00+00:00 2021-10-10 07:35:23 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 577 rows took 0.051315s 2021-10-10 07:35:24 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 285 rows took 0.076943s 2021-10-10 07:35:24 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.077574s 2021-10-10 07:35:25 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 260 rows took 1.527013s 2021-10-10 07:35:25 DEBUG (Recorder) [homeassistant.components.recorder.history] getting 260 first datapoints took 1.539757s 2021-10-10 07:35:25 DEBUG (Recorder) [homeassistant.components.recorder.util] querying 2099 rows took 0.203640s 2021-10-10 07:35:25 DEBUG (Recorder) [homeassistant.components.recorder.history] get_significant_states took 0.204216s

rostislav-palivoda commented 2 years ago

Could you check ram and swap usage during this queue growth? Its interesting why queue grows only on command line sensors. I would suspect that you have some massive memory operations in this sensors and hit ram limit, the OS starts to swap the ram and in this moment the HA slows down and hits the queue limit, then queue watchdog steps in and kills everything - game over.

egranto commented 2 years ago

image image

the system running all don't present spikes image

image

well the queue always growth but is able to empty their self that is my first question why we are using queue if we still had connection, queries looks fast for me, the longer i had seen is less than 3 seconds

i can't notice bottlenecks by the lack of resources on the host

rostislav-palivoda commented 2 years ago

Try to reduce recorder history to 1 day, increase commit interval to lets say 5 and after tables are reduced (next day) enable command line sensors.

  purge_keep_days: 1
  commit_interval: 5
egranto commented 2 years ago

for the moment queue is very low use

2021-10-13 07:51:59 DEBUG (Recorder) [homeassistant.components.recorder] Connected to recorder database 2021-10-13 08:01:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1893 2021-10-13 08:11:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1046 2021-10-13 08:21:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 1582 2021-10-13 08:31:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 08:41:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 08:51:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 461 2021-10-13 09:01:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 09:11:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 09:21:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 09:31:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 10 2021-10-13 09:41:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 2021-10-13 09:51:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0 ... 2021-10-13 12:01:59 DEBUG (MainThread) [homeassistant.components.recorder] Recorder queue size is: 0

rostislav-palivoda commented 2 years ago

Try to increase days of history and watch queue for a week or two. If you start hitting the limit then either reduce days or tune innodb to speed up inserts - https://www.percona.com/blog/2007/11/01/innodb-performance-optimization-basics/

egranto commented 2 years ago

now i'm adding back the sensors that i remove to reduce the load, but i think the trick was the commit interval

rostislav-palivoda commented 2 years ago

Can you post here your InnoDB config? What's in innodb_buffer_pool_size and other values?

egranto commented 2 years ago

image

Host system had 16 gb of ram

rostislav-palivoda commented 2 years ago

Try to return buffer_pool_size to default 128mb or 256mb - https://mariadb.com/kb/en/innodb-system-variables/#innodb_buffer_pool_size

miniontechy commented 2 years ago

I'm seeing this problem too. Seems to have occured during changeover from BST to GMT since the clocks went back an hour last night.

Logger: homeassistant.components.recorder Source: components/recorder/init.py:444 Integration: Recorder (documentation, issues) First occurred: 1:03:16 AM (1 occurrences) Last logged: 1:03:16 AM

The recorder queue reached the maximum size of 30000; Events are no longer being recorded

CPU load usually 3%-5% but after this event, CPU load is typ 37% Glances shows that addon_core_mariadb is using about 70% of CPU use. Am using "core-2021.10.6" on "Home Assistant OS 6.5" with "MariaDB 2.4.0"

valepe commented 2 years ago

I'm seeing this problem too. Seems to have occured during changeover from BST to GMT since the clocks went back an hour last night."

Same thing happened to me. Never seen this before sunday on which clock changed. I run stock Home Assistant on docker on a Synology device.

rostislav-palivoda commented 2 years ago

There is confirmed performance problem caused by daylight time savings. Just restart or update to latest version. Can someone close the issue? :)

egranto commented 2 years ago

not really i'm on 2011.11.1

image

and just had to restart for the 30000 queue

and i'm not in a DTS country or region

ShadowRep commented 2 years ago

I have the same problem. "The recorder queue reached the maximum size of 30000; Events are no longer being recorded." At first I thought it was because I installed Mosquitto but uninstalling it did not help.

I don't have a lot of sensors. A handful of Shellys, less than 20 Z-Wave and ZHA entities. Everything else is "external" via integrations like "Hue", but even then my installation is quite small.

I am using MariaDB and because I want long time statistics I have set "purge_keep_days: 737".

The problem first appeared on the 19. November, never happened before. After a reboot it works ok for a short time before it stops recording again.

Screenshot 2021-11-25 083849

egranto commented 2 years ago

mine happen when i active a command line sensor to read the movement of my camara are 5 camaras and i check that sensor every 5 seconds

My assumption is that now it can't with so much data, because i don't think is a performance thing of MariaDB, because the same happened on my system with the normal DB even happened faster

ShadowRep commented 2 years ago

Is there any progress on this bug? It keeps happening all the time.