I keep losing video from my camera. Integration reload helps. Error from stream worker: Error demuxing stream, Error opening stream (HTTP_UNAUTHORIZED,

dgtal1 commented 1 year ago

The problem

For many months now I have a problem with my Synology cameras preview in Home Assistant. It used to work super stable for years, but for over a year there's the same issue happening: after the last HA restart I keep losing video. Sometimes it's hours, sometimes days, but eventually it's gone and I need to reload thy Synology integration to have it working again.

I opened an issue for this a year ago: https://github.com/home-assistant/core/issues/72242. It looked like it helped (I was using the custom_component version of the fix provided in there), but HA version 2023.5 broke the compatibility with the custom fix, so I rolled back to the HA version to have the video working again. This time however the situation got worse. While the custom fix helped me to maintain video for days - now it's failing after several hours, sometimes below 1 hoour :(

What wonders me is that I've got 2 DSM cameras integrated with HA, but only 1 is configured to keep the stream running: HA entity: camera.synology_podjazd And this one never fails!

The one which fails (HA entity camera.synology_dach) with this error I described has the setting disabled:

I think it is relevant to highlight this. I don't want both cameras to have stream preloading as this consumes CPU on my HA server, which I want to be as little energy consuming and as fast as possible.

What version of Home Assistant Core has the issue?

core-2023.8.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

synology_dsm

Link to integration documentation on our website

https://www.home-assistant.io/integrations/synology_dsm

Diagnostics information

config_entry-synology_dsm-4e2b48aaaf05e4597d49bc2eb475af28.json (8).txt

I had to compress the log file and upload to GDrive as the debug log generated 90 MB of data in about 1 hour. Here's the debug log: https://drive.google.com/file/d/153txsGAcbQ5dFwjiwTqp8d3EpwzeWqNp/view?usp=drive_link

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.synology_dsm.coordinator
Source: components/synology_dsm/coordinator.py:126
Integration: Synology DSM (documentation, issues)
First occurred: August 17, 2023 at 5:22:52 PM (3 occurrences)
Last logged: August 21, 2023 at 11:23:22 PM

Unexpected error fetching 192.168.0.13 SynologyDSMCameraUpdateCoordinator data: {'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'TimeoutError'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 317, in _execute_request
    response = await self._session.get(url_encoded, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 899, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 316, in _execute_request
    async with async_timeout.timeout(self._timeout):
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/synology_dsm/coordinator.py", line 126, in _async_update_data
    await surveillance_station.update()
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/api/surveillance_station/__init__.py", line 25, in update
    raw_data = await self._dsm.get(self.CAMERA_API_KEY, "List", max_version=7)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 207, in get
    return await self._request("GET", api, method, params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 277, in _request
    url, params, kwargs = await self._prepare_request(api, method, params, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 237, in _prepare_request
    await self.login()
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 155, in login
    result = await self.get(API_AUTH, "login", params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 207, in get
    return await self._request("GET", api, method, params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 282, in _request
    response = await self._execute_request(request_method, url, params, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 358, in _execute_request
    raise SynologyDSMRequestException(exp) from exp
synology_dsm.exceptions.SynologyDSMRequestException: {'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'TimeoutError'}

Logger: homeassistant.components.synology_dsm.coordinator
Source: components/synology_dsm/coordinator.py:73
Integration: Synology DSM (documentation, issues)
First occurred: August 17, 2023 at 10:16:30 AM (6 occurrences)
Last logged: August 21, 2023 at 11:23:23 PM

Unexpected error fetching 192.168.0.13 SynologyDSMSwitchUpdateCoordinator data: {'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'TimeoutError'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 317, in _execute_request
    response = await self._session.get(url_encoded, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 628, in sock_connect
    return await fut
           ^^^^^^^^^
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 316, in _execute_request
    async with async_timeout.timeout(self._timeout):
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/synology_dsm/coordinator.py", line 73, in _async_update_data
    "switches": {"home_mode": await surveillance_station.get_home_mode_status()}
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/api/surveillance_station/__init__.py", line 153, in get_home_mode_status
    raw_data = await self._dsm.get(self.HOME_MODE_API_KEY, "GetInfo")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 207, in get
    return await self._request("GET", api, method, params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 282, in _request
    response = await self._execute_request(request_method, url, params, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synology_dsm/synology_dsm.py", line 358, in _execute_request
    raise SynologyDSMRequestException(exp) from exp
synology_dsm.exceptions.SynologyDSMRequestException: {'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'TimeoutError'}

Logger: homeassistant.components.stream.stream.camera.synology_podjazd
Source: components/stream/__init__.py:434
Integration: Stream (documentation, issues)
First occurred: August 13, 2023 at 7:33:46 PM (272 occurrences)
Last logged: 11:24:21 AM

Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:fc7b921d3bc48fb8764fab8fdf6c411f@192.168.0.13:554/Sms=8.unicast'
Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:661dd45dd925b86431ebf1eef33c3535@192.168.0.13:554/Sms=8.unicast'
Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:2d576aaaf3f0bea5bae907023d0022c9@192.168.0.13:554/Sms=8.unicast'
Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:8c80f533859cacc224f85b0363cfa5ff@192.168.0.13:554/Sms=8.unicast'
Error from stream worker: Error opening stream (HTTP_UNAUTHORIZED, Server returned 401 Unauthorized (authorization failed)) rtsp://****:****@192.168.0.13:554/Sms=8.unicast

Logger: homeassistant.components.stream.stream.camera.synology_dach
Source: components/stream/__init__.py:434
Integration: Stream (documentation, issues)
First occurred: August 15, 2023 at 5:58:53 AM (283 occurrences)
Last logged: 11:19:03 AM

Error from stream worker: Error demuxing stream: [Errno 1414092869] Immediate exit requested: 'rtsp://syno:dacf8de0ec844dedaa5471f999b1e755@192.168.0.13:554/Sms=9.unicast'
Error from stream worker: Error demuxing stream while finding first packet: [Errno 110] Operation timed out: 'rtsp://syno:e0164fdc80da98adc15766732b2b3bf6@192.168.0.13:554/Sms=9.unicast'; last error log: [h264] no frame!
Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:e62d6a0c17004aa39ee3f6ba77853ade@192.168.0.13:554/Sms=9.unicast'
Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://syno:e62d6a0c17004aa39ee3f6ba77853ade@192.168.0.13:554/Sms=9.unicast'; last error log: [h264] error while decoding MB 98 56, bytestream -27
Error from stream worker: Error opening stream (HTTP_UNAUTHORIZED, Server returned 401 Unauthorized (authorization failed)) rtsp://****:****@192.168.0.13:554/Sms=9.unicast

Additional information

Looks like it's related to stream preloading. Apparently when not retrieving video data from Synology for some time - the integration looses some access token, which probably needs to be refreshed every once and a while. Since integration reload solves this issue and this is the workaround I use to avoid restarting the whole HA server - this must be some buffering or token validity issue in my opinion.

I enabled debug logging for Synology DSM integration and downloaded it once the error occurred. The log is huge as it has all entries since the last HA bootup which is 10 days, but I saw it shows the DEBUG log entries right after I enabled it in settings. I think everything may be relevant and hoping you are able to filter out the irrelevant log entries.

Keeping fingers crossed that this time you'll be able to pinpoint the issue and solve it once and forever 🤞

home-assistant[bot] commented 1 year ago

Hey there @hacf-fr, @quentame, @mib1185, mind taking a look at this issue as it has been labeled with an integration (synology_dsm) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `synology_dsm` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign synology_dsm` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

synology_dsm documentation synology_dsm source _{^{(message by IssueLinks)}}

issue-triage-workflows[bot] commented 11 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

dgtal1 commented 11 months ago

That's still not solved. I'm still loosing video stream from my Synology camera even in the newest HA version.

daved3luxe commented 7 months ago

I have the same problem, if I activate "Preload Stream" it works bit without this beeing enabled, the stream stops after some time.

Dennis90BW commented 6 months ago

Same issue here. HA claims that my entity is no longer being provided but if I reload the integration it works for a few hours.

issue-triage-workflows[bot] commented 3 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core