Card stops working intermittently: Unknown error

felipecrs commented 2 years ago

I'm using the current latest v4.0.0. Nothing shows in the browser logs. Happens frequently, but I still can't determine why.

https://user-images.githubusercontent.com/29582865/167648294-1f719e1f-41d6-41f1-9982-80c5561327ba.mp4

This didn't happen in v3.

This is the configuration of one of my cards, but the problem happens to all of them:

- type: custom:frigate-card
    view:
      default: live
      update_force: false
    menu:
      buttons:
        frigate:
          enabled: false
        snapshots:
          enabled: false
        image:
          enabled: false
      style: outside
      position: top
    live:
      preload: false
      controls:
        next_previous:
          style: none
        title:
          mode: none
      lazy_load: true
      lazy_unload: all
      auto_unmute: never
    dimensions:
      aspect_ratio_mode: static
      aspect_ratio: '16:9'
    event_viewer:
      auto_play: never
    elements:
      - type: custom:frigate-card-conditional
        conditions:
          view:
            - live
          camera:
            - camera.rua
        elements:
          - type: icon
            icon: mdi:arrow-up
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 25px
              bottom: 115px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                tilt: UP
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-down
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 25px
              bottom: 65px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                tilt: DOWN
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-left
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 50px
              bottom: 90px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                pan: LEFT
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
          - type: icon
            icon: mdi:arrow-right
            style:
              background: rgba(255, 255, 255, 0.75)
              border-radius: 5px
              right: 0px
              bottom: 90px
            tap_action:
              action: call-service
              service: onvif.ptz
              service_data:
                entity_id: camera.rua_onvif
                pan: RIGHT
                move_mode: ContinuousMove
                speed: 0.05
                continuous_duration: 0.05
      - type: custom:frigate-card-conditional
        conditions:
          view:
            - live
          camera:
            - camera.rua
        elements:
          - type: custom:frigate-card-menu-submenu-select
            entity: select.camera_da_rua_night_vision
    cameras:
      - camera_entity: camera.rua
        live_provider: frigate-jsmpeg
        id: camera.rua
        title: Rua
        camera_name: rua
      - camera_entity: camera.morro
        live_provider: frigate-jsmpeg
      - camera_entity: camera.portao
        live_provider: frigate-jsmpeg
      - camera_entity: camera.video_porteiro
        live_provider: ha

dermotduffy commented 2 years ago

Thanks @felipecrs .

Ever see it in a view other than live? You are using multiple live providers, does it show up on providers or only 1? (If you happened to notice!)

felipecrs commented 2 years ago

It happens in clips view as well.

By the way I noticed a bunch of errors in the Hass Core logs:

I'm not sure what's the best way to dump these logs (and btw, if Frigate integration supported Diagnostics this would be a little easier).

felipecrs commented 2 years ago

Here is one of them:

Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/frigate/media_source.py:685
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: May 9, 2022, 11:53:12 AM (374 occurrences)
Last logged: 11:21:45 AM

[139964827719376] Error handling message: Unknown error (unknown_error)
[139964821966464] Error handling message: Unknown error (unknown_error)
[139965014090464] Error handling message: Unknown error (unknown_error)
[139962666910048] Error handling message: Unknown error (unknown_error)
[139964532992512] Error handling message: Unknown error (unknown_error)
Traceback (most recent call last):
  File "/config/custom_components/frigate/api.py", line 140, in api_wrapper
    response = await self._session.get(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/config/custom_components/frigate/api.py", line 154, in api_wrapper
    await self._session.post(url, headers=headers, json=data)
  File "/usr/local/lib/python3.9/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.9/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/frigate/media_source.py", line 681, in _get_event_summary_data
    summary_data = await self._get_client(identifier).async_get_event_summary(
  File "/config/custom_components/frigate/api.py", line 102, in async_get_event_summary
    await self.api_wrapper(
  File "/config/custom_components/frigate/api.py", line 162, in api_wrapper
    raise FrigateApiClientError from exc
custom_components.frigate.api.FrigateApiClientError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 27, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/media_source/__init__.py", line 152, in websocket_browse_media
    media = await async_browse_media(hass, msg.get("media_content_id", ""))
  File "/usr/src/homeassistant/homeassistant/components/media_source/__init__.py", line 111, in async_browse_media
    item = await _get_media_item(hass, media_content_id).async_browse()
  File "/usr/src/homeassistant/homeassistant/components/media_source/models.py", line 85, in async_browse
    return await self.async_media_source().async_browse_media(self)
  File "/config/custom_components/frigate/media_source.py", line 653, in async_browse_media
    await self._get_event_summary_data(identifier), identifier, events
  File "/config/custom_components/frigate/media_source.py", line 685, in _get_event_summary_data
    raise MediaSourceError from exc
homeassistant.components.media_source.error.MediaSourceError

dermotduffy commented 2 years ago

Looks like a communication issue between HA and the Frigate backend, which propagated to a failed media source error, which propagated into the card.

Not sure why this wouldn't have happened with v3 though. Did you change your Frigate backend version recently?

felipecrs commented 2 years ago

No, I didn't change anything else, except applying HA Core patch upgrade, I think.

I didn't reboot my server since a while, I will reboot it now and monitor too.

dermotduffy commented 2 years ago

Actually, I think the v4 card is probably fetching the events to show as thumbnails, but you don't have thumbnails enabled -- so that fetch does ~nothing useful. And when that fetch fails, it causes this error for you. I can try to optimize that away ... [and off the top of my head the v3 probably didn't behave this way]

(But still seems like there's something unrelated going on between your HA and your Frigate instance)

felipecrs commented 2 years ago

Ok, error remains happening after rebooting the server, just to confirm.

dermotduffy commented 2 years ago

Anything hints in your Frigate backend logs?

dermotduffy commented 2 years ago

BTW: Because of the behavior I described above, if you built your v4 before I fixed https://github.com/dermotduffy/frigate-hass-card/issues/557 , this problem may have been massively exasperated (as fetching was incorrectly continuous, instead of one-off).

felipecrs commented 2 years ago

Ok, but my card is up-to-date with the release-4.0.0 tree, including my own PR which was the last to be merged.

felipecrs commented 2 years ago

I will try to revert to 3.0.0 and see what happens, I will let you know of the result.

felipecrs commented 2 years ago

Anything hints in your Frigate backend logs?

No, nothing useful there. Only the regular WebSockets initiated and terminated messages.

felipecrs commented 2 years ago

I wasn't able to reproduce the issue in v3 (but testing it wasn't very sharp, as my card configuration was made for v4, what I did was to create another card with 3 cameras lacking a lot of the configuration from my real ones)
After rebuilding and upgrading again to the v4 tree (https://github.com/dermotduffy/frigate-hass-card/pull/569), I yet could not reproduce the issue.

Let me monitor for a while.

felipecrs commented 2 years ago

It just happened again. I noticed that it happens not only during initialization. Right now, it happened around 3 seconds after the camera was already being shown.

felipecrs commented 2 years ago

I think it has no relationship with cameras being Frigate or not (because I do mix them, but the third card in the above screenshot only mixes Frigate cameras).

felipecrs commented 2 years ago

Ok, the issue came from frequently to rare. I don't know what has changed, and I can't guarantee anymore that this was introduced by v4.

I still see tons of Unknown error in my HA Core logs, related to the Frigate WebSockets API, but, they are not causing my card to crash (it seems).

So, I'm closing this. Thank you very much for the support so far, if I find something useful I will let you know.

felipecrs commented 2 years ago

Out of the blue this started to plague me again:

https://user-images.githubusercontent.com/29582865/167893638-5647a520-f501-4635-85a8-6164474e0e72.mp4

I have the feeling that this has something to do with ongoing detections/clips.

felipecrs commented 2 years ago

Oh, I found a good way to test.

I was using HA 2022.5.3, and I restored the HA Core backup of 2022.5.2. This also restored my Lovelace dashboard, which in turn was before the Frigate card 4.0.0.

This way, I could revert to 3.0.0 keeping all my cards untouched.

I am testing now with v3.

felipecrs commented 2 years ago

Just to mention, restoring the backup caused a big side effect for me:

https://github.com/home-assistant/core/issues/68198

The correct thing to do was to restore the backup taken by the Google Drive add-on. That one restored everything, including my /config folder.

felipecrs commented 2 years ago

@dermotduffy, I think I was able to narrow the issue a bit further:

https://1drv.ms/v/s!AntpCaY1YjnMiukyVoZUxB_tmqUm7A?e=UCFE1S

(The video is too big to post on GitHub)

dermotduffy commented 2 years ago

Theory: The card is spamming the server with requests, causing the Frigate backend to get overloaded, time out on some of those requests -- causing this error.

dermotduffy commented 2 years ago

@felipecrs Would you mind re-testing with the merged #587 ? I hope/suspect this will help you!

felipecrs commented 2 years ago

@dermotduffy thank you so much! But unfortunately, it did not solve the issue:

https://user-images.githubusercontent.com/29582865/168440882-c48aa139-2281-43c2-87e4-da90fd3f860d.mp4

(Jump to 0:35)

Now I will try to update to the latest Frigate integration that was just released.

felipecrs commented 2 years ago

I have another theory:

Now the card fetches thumbnails (which didn't happen before)
Maybe there is a bug in the Frigate backend which affects my environment (but not yours), and it was related to the thumbnails API, and since now the card is pulling from it, it triggers the error.

dermotduffy commented 2 years ago

Thanks @felipecrs . Ah well, we'll get there!

In your configuration, the card should no longer fetch thumbnails in the live view (since you do not enable thumbnails in your config). I notice in your video shows the problem in timeline view (which does need to fetch events, obviously) -- so at least this might be resolved for you in live view?

Your theory could be right. Is your Frigate machine under heavy load perhaps? Does browsing the HA media browser ever fail for you?

I'll do a little research later on trying to reduce the number of requests the timeline is showing, but that'll be harder since events are kind of the whole point. To experiment, one simple way to cut the number of event requests in half, would be to do this:

timeline:
  media: clips

(By default it fetches events that are clips & snapshots, so limiting it to one of those instantly cuts the number of fetches necessary by 50%).

felipecrs commented 2 years ago

@dermotduffy thanks a lot for the insights, but please don't do anything yet. I'm testing the new Frigate integration, and I already see a different behavior:

Before, in the timeline view, nothing ever showed up for me (I was about to send you a Discord message asking if that was expected). Immediately after upgrading the integration, I now get proper timeline views:

And yet, no errors.

dermotduffy commented 2 years ago

Yes, that is expected. The new integration has https://github.com/blakeblackshear/frigate-hass-integration/pull/225 which is required for the timeline to work (I just hadn't done an integration build, since the card code is still very untested!).

OK -- I'll hold off until you tell me.

felipecrs commented 2 years ago

@dermotduffy I'm about to claim everything is good now. No errors in HA core logs too. If nothing shows up by the end of day, I will close the issue.

felipecrs commented 2 years ago

And just to answer some questions:

so at least this might be resolved for you in live view?

I haven't observed the issue happening in live view indeed, but my test (before upgrading the Frigate integration) was very shallow.

Is your Frigate machine under heavy load perhaps?

Yes, the machine is under a good load. I host everything on it through HassOS, and Frigate usually takes 40-60% of my 4-core J4125 CPU. The total load is usually at 70%:

I haven't yet determined the reason behind the 90-100% spikes, though.

Does browsing the HA media browser ever fail for you?

Never noticed it.

felipecrs commented 2 years ago

Ok, the issue didn't happen since I upgraded the Frigate integration from v2.2.2 to v2.3.

I'm closing the issue, if it happens again, I will come back.

I am very grateful about all the care you put in both the Frigate Card and the integration.

satyambaba commented 2 years ago

@dermotduffy, I am facing the same issue even on Frigate Integration v2.3 (Frigate Card - v3.0.0 and Frigate Addon - Frigate NVR v3.1). I am on HA Blue (HA Core 2022.6.2, Supervisor 2022.05.3, OS 8.1).

Screenshot 2022-06-06 221948

dermotduffy commented 2 years ago

Hi @satyambaba, I'm not sure that's the same thing. Felipe's error was showing up as "unknown", whereas yours is labeled as a media source issue.

Does the HA Media Browser work correctly for you for Frigate? Do you have any exceptions in your Home Assistant logs, or anything on your Javascript console?

satyambaba commented 2 years ago

Hi @satyambaba, I'm not sure that's the same thing. Felipe's error was showing up as "unknown", whereas yours is labeled as a media source issue.

Does the HA Media Browser work correctly for you for Frigate? Do you have any exceptions in your Home Assistant logs, or anything on your Javascript console?

Hi @dermotduffy, thanks a lot for your response. I can browse frigate clips and recordings using the inbuilt media browser. I don't have any exception in HA logs or in the JS console. Please see the attached screenshots.

media_browser

javascript_console

dermotduffy commented 2 years ago

@satyambaba What version number do you see when you navigate to your Frigate UI, then click debug on the left sidebar? You should see a version number near the top of the screen.

satyambaba commented 2 years ago

@satyambaba What version number do you see when you navigate to your Frigate UI, then click debug on the left sidebar? You should see a version number near the top of the screen.

@dermotduffy - I see 0.10.1-83481AF. I'm not sure if this is the version number you are looking for?

dermotduffy commented 2 years ago

Bit surprised you're seeing mp4 files in the media browser like that, thought you might be on a much older Frigate instance. Something feels wrong here :-)

Can you navigate in HA to Settings -> Devices & Services, then click through your Frigate entry, choose 'Devices' choose any arbitrary Frigate device that is other than the one named Frigate then take a screenshot? It should look something like this:

frigate

satyambaba commented 2 years ago

Bit surprised you're seeing mp4 files in the media browser like that, thought you might be on a much older Frigate instance. Something feels wrong here :-)

Can you navigate in HA to Settings -> Devices & Services, then click through your Frigate entry, choose 'Devices' choose any arbitrary Frigate device that is other than the one named Frigate then take a screenshot? It should look something like this:

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled. The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

dermotduffy commented 2 years ago

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled.

Aha!

The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

The integration only does the glue between the Frigate server (which captures events/records) and Home Assistant, so that is as expected.

satyambaba commented 2 years ago

While I was navigating to take the suggested screenshot, I noticed that the frigate integration was disabled.

Aha!

The moment I enabled that, the error vanished. Sorry for the trouble, it was an oversight at my end. However, now I'm wondering that how the frigate was working fine (including capturing events, recording, etc.) while the integration was disabled.

The integration only does the glue between the Frigate server (which captures events/records) and Home Assistant, so that is as expected.

Thanks for the clarification, and your help. I really appreciate it! Thanks a lot for this awesome integration.

dermotduffy / frigate-hass-card

Card stops working intermittently: Unknown error #568