amcrest: cannot schedule new futures after shutdown

xuefer commented 3 years ago

The problem

Happen randomly during shutdown, preventing shutdown Update for camera.men_kou fails, RuntimeError: cannot schedule new futures after shutdown

Environment

Home Assistant Core release with the issue: 0.118.4
Last working Home Assistant Core release (if known): unknown/random
Operating environment (OS/Container/Supervised/Core): openwrt on x86_64
Integration causing this issue: amcrest
Link to integration documentation on our website: https://www.home-assistant.io/integrations/amcrest/

Problem-relevant `configuration.yaml`

amcrest:
  - host: 192.16.26.83
    name: men_kou
    username: stream
    password: <removed>
    stream_source: rtsp

Traceback/Error logs

[pid 32059] epoll_wait(3, [], 7, 0)     = 0
[pid 32059] epoll_wait(3, [], 7, 0)     = 0
[pid 32059] epoll_wait(3, [], 7, 0)     = 0
[pid 32059] getpid()                    = 32059
[pid 32059] epoll_ctl(3, EPOLL_CTL_DEL, 8, 0x7ffdb62e6d24) = 0
[pid 32059] getpid()                    = 32059
[pid 32059] epoll_wait(3, [], 6, 0)     = 0
[pid 32059] close(8)                    = 0
[pid 32059] epoll_wait(3, [], 6, 1923)  = 0
[pid 32059] epoll_wait(3, [], 6, 4996)  = 0
[pid 32059] epoll_wait(3, [], 6, 3107)  = 0
[pid 32059] epoll_wait(3, [], 6, 0)     = 0
[pid 32059] getpid()                    = 32059
[pid 32059] getpid()                    = 32059
[pid 32059] epoll_wait(3, [], 6, 0)     = 0
[pid 32059] getpid()                    = 32059
[pid 32059] stat("/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/helpers/entity.py", {st_mode=S_IFREG|0644, st_size=23400, ...}) = 0
[pid 32059] stat("/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/core.py", {st_mode=S_IFREG|0644, st_size=58897, ...}) = 0
[pid 32059] stat("/usr/lib/python3.8/asyncio/base_events.py", 0x7ffdb62e5250) = -1 ENOENT (No such file or directory)
[pid 32059] stat("/usr/lib/python3.8/concurrent/futures/thread.py", 0x7ffdb62e5250) = -1 ENOENT (No such file or directory)
[pid 32059] write(2, "Update for camera.men_kou fails\nTraceback (most recent call last):\n  File \"/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/
homeassistant/helpers/entity.py\", line 278, in async_update_ha_state\n    await self.async_device_update()\n  File \"/custom/mnt/data/var/lib/homeassistant/venv/lib/python
3.8/site-packages/homeassistant/helpers/entity.py\", line 461, in async_device_update\n    task = self.hass.async_add_executor_job(self.update)  # type: ignore\n  File \"/c
ustom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/core.py\", line 406, in async_add_executor_job\n    task = self.loop.run_in_executor(Non
e, target, *args)\n  File \"/usr/lib/python3.8/asyncio/base_events.py\", line 783, in run_in_executor\n  File \"/usr/lib/python3.8/concurrent/futures/thread.py\", line 179,
 in submit\nRuntimeError: cannot schedule new futures after shutdown\n", 887) = 887

Traceback (most recent call last):
  File \"/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/helpers/entity.py\", line 278, in async_update_ha_state
    await self.async_device_update()
  File \"/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/helpers/entity.py\", line 461, in async_device_update
    task = self.hass.async_add_executor_job(self.update)  # type: ignore
  File \"/custom/mnt/data/var/lib/homeassistant/venv/lib/python3.8/site-packages/homeassistant/core.py\", line 406, in async_add_executor_job
    task = self.loop.run_in_executor(None, target, *args)
  File \"/usr/lib/python3.8/asyncio/base_events.py\", line 783, in run_in_executor
  File \"/usr/lib/python3.8/concurrent/futures/thread.py\", line 179, in submit
RuntimeError: cannot schedule new futures after shutdown

Additional information

@pnbruckner

probot-home-assistant[bot] commented 3 years ago

amcrest documentation amcrest source _{^{(message by IssueLinks)}}

xuefer commented 3 years ago

the IP does not belong to a device (misconfigured), but maybe this config make it easier to reproduce but not 100%

pnbruckner commented 3 years ago

I have no idea how this could happen. Are you saying it only happens if the configured camera doesn't actually exist, or isn't on?

It might help if you could capture more details from home-assistant.log when this happens. Please add at least the following to your configuration, then restart. The next time it happens, look in home-assistant.log for any lines that have to do with amcrest.

logger:
  default: info
  logs:
    amcrest: debug
    homeassistant.components.amcrest: debug
    homeassistant.core: debug

xuefer commented 3 years ago

not sure if the following state the same issue when restarting home assistant

# sv restart homeassistant
timeout: run: homeassistant: (pid 19846) 2211s, got TERM

# strace -fp 19846
strace: Process 19846 attached with 4 threads
[pid 19867] futex(0x7f867c2404c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 20198] futex(0x7f8638096190, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 19917] recvfrom(34,  <unfinished ...>

# lsof -np 19846
hass    19846 root   34u     IPv4              83122      0t0    TCP 172.16.26.1:61232->172.16.26.83:www (ESTABLISHED)

172.16.26.83 is

amcrest:
  - host: 172.16.26.83
    username: ...
    password: ...
    stream_source: rtsp
    binary_sensors:
      - audio_detected
      - motion_detected
      - online

device: dahua camera device exists working fine but make restarting home assistant impossible, stuck at "recvfrom(34,"

I'm not sure but I'll file a new issue ticket if this is not the same problem

github-actions[bot] commented 3 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core