Amcrest on 2024.5.0 with offline camera causing HA to fail loading completely, got stuck with increased CPU usage

The problem

Updated to 2024.5.0 and 2024.5.1 and noted that HA failed to load competely. I also noticed an increase in CPU usage (around 10x higher) resulting in CPU usage stabilizing around 30% instead of 3%.

Rolled back to 2024.4.4 and HA loaded correctly, CP immediately dropped.
removed the configuration of camera with the Amcrest integration and 2024.5.1 loaded correctly, with CPU being back at 3%.

I have a camera configured that was offline at time of starting HA. (the camera is offline most of the time, I just turn it on when needed)

What version of Home Assistant Core has the issue?

core-2024.5.0 and core-2024.5.1

What was the last working version of Home Assistant Core?

core-2024.4.4

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Amcrest

Link to integration documentation on our website

https://www.home-assistant.io/integrations/amcrest/

Diagnostics information

No response

Example YAML snippet

amcrest:
  - host: 192.168.xxx.xxx
    username: !secret armcrest_camera_user
    password: !secret armcrest_camera_password
    name: "Amcrest Camera"
    binary_sensors:
      - online

Anything in the logs that might be useful for us?

amcrest.exceptions.CommError: HTTPConnectionPool(host='192.168.xxx.xxx', port=80): Max retries exceeded with url: /cgi-bin/magicBox.cgi?action=getMachineName (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/amcrest/camera.py", line 330, in async_on_demand_update
    self.async_schedule_update_ha_state(True)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1252, in async_schedule_update_ha_state
    self.hass.async_create_task(
  File "/usr/src/homeassistant/homeassistant/core.py", line 816, in async_create_task
    self.verify_event_loop_thread("async_create_task")
  File "/usr/src/homeassistant/homeassistant/core.py", line 440, in verify_event_loop_thread
    frame.report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 162, in report
    _report_integration(what, integration_frame, level, error_if_integration)
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 203, in _report_integration
    raise RuntimeError(
RuntimeError: Detected that integration 'amcrest' calls async_create_task from a thread at homeassistant/components/amcrest/camera.py, line 330: self.async_schedule_update_ha_state(True). Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+amcrest%22.

additional logs:

    self.connect()
  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.xxx.xxx', port=80): Max retries exceeded with url: /cgi-bin/magicBox.cgi?action=getMachineName (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/amcrest/http.py", line 305, in _command
    resp = session.get(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.xxx.xxx', port=80): Max retries exceeded with url: /cgi-bin/magicBox.cgi?action=getMachineName (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/amcrest/__init__.py", line 193, in command
    ret = super().command(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amcrest/http.py", line 253, in command
    self._generate_token()
  File "/usr/local/lib/python3.12/site-packages/amcrest/http.py", line 150, in _generate_token
    resp = self._command(cmd).content.decode()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/amcrest/http.py", line 327, in _command
    raise CommError(error) from error
amcrest.exceptions.CommError: HTTPConnectionPool(host='192.168.xxx.xxx', port=80): Max retries exceeded with url: /cgi-bin/magicBox.cgi?action=getMachineName (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/amcrest/camera.py", line 330, in async_on_demand_update
    self.async_schedule_update_ha_state(True)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1252, in async_schedule_update_ha_state
    self.hass.async_create_task(
  File "/usr/src/homeassistant/homeassistant/core.py", line 816, in async_create_task
    self.verify_event_loop_thread("async_create_task")
  File "/usr/src/homeassistant/homeassistant/core.py", line 440, in verify_event_loop_thread
    frame.report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 162, in report
    _report_integration(what, integration_frame, level, error_if_integration)
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 203, in _report_integration
    raise RuntimeError(
RuntimeError: Detected that integration 'amcrest' calls async_create_task from a thread at homeassistant/components/amcrest/camera.py, line 330: self.async_schedule_update_ha_state(True). Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+amcrest%22.
2024-05-05 11:08:47.681 WARNING (Amcrest Amcrest Camera) [py.warnings] /usr/src/homeassistant/homeassistant/util/logging.py:110: RuntimeWarning: coroutine 'Entity.async_update_ha_state' was never awaited
  log_exception(format_err, *args)
2024-05-05 11:08:47.681 WARNING (Amcrest Amcrest Camera) [homeassistant.components.amcrest] Error while processing events from Amcrest Camera camera: CommError(ConnectionError(MaxRetryError("HTTPConnectionPool(host='192.168.xxx.xxx', port=80): Max retries exceeded with url: /cgi-bin/magicBox.cgi?action=getMachineName (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6d6bc8c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))")))
2024-05-05 11:08:52.625 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.amcrest_camera is taking over 10 seconds
2024-05-05 11:08:57.624 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:09:12.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:09:27.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:09:42.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:09:57.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:10:08.524 WARNING (MainThread) [homeassistant.components.mqtt.client] Disconnected from MQTT server 192.168.xxx.xxx:1883 (7)
2024-05-05 11:10:12.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:10:27.625 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:10:42.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:10:57.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:11:12.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:11:27.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:11:42.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:11:50.548 WARNING (MainThread) [homeassistant.components.mqtt.client] Disconnected from MQTT server 192.168.xxx.xxx:1883 (7)
2024-05-05 11:11:57.626 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:12:12.627 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:12:27.627 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15
2024-05-05 11:12:42.627 WARNING (MainThread) [homeassistant.components.camera] Updating amcrest camera took longer than the scheduled update interval 0:00:15

Additional information

The camera was offline when starting HA, therefore the camera was not reachable during startup
when setting "debug: true" in the homeassistant configuration, HA starts normally. "debug: false" (or no setting) reproduces the issue

home-assistant / core

Amcrest on 2024.5.0 with offline camera causing HA to fail loading completely, got stuck with increased CPU usage #116850