API error whilst retrieving events for camera

btrudgill commented 3 months ago

Checklist:

[x] I updated to the latest version available
[x] I cleared the cache of my browser

Release with the issue: 5.2.0 Last working release (if known):

Browser and Operating System: Firefox, home assistant mobile app (ios and android).

Description of problem:

Probably 50% of the time when I check my cameras, the events do not load (the side tab thing does not show up) and after a few seconds I get the error message shown below.

Frigate version: 0.13.2-6476f8a

Frigate card diagnostic information This is the error I am getting on the card when the events fail to load:

{
  "request": {
    "type": "frigate/events/get",
    "instance_id": "frigate",
    "cameras": [
      "doorbell"
    ],
    "limit": 50,
    "id": 48
  },
  "response": {
    "code": "frigate_error",
    "message": "API error whilst retrieving events for cameras ['doorbell'] for Frigate instance frigate"
  }
}

{
  "ha_version": "2024.2.3",
  "card_version": "5.2.0",
  "browser": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
  "date": "2024-02-27T09:19:49.056Z",
  "frigate_version": {
    "47ac916ca5d1661b5fe5f01f9e5360dd": "5.0.1/0.13.2-6476f8a"
  },
  "lang": "en",
  "timezone": "Europe/London",
  "git": {
    "build_version": "5.2.0-HEAD+g69249b6",
    "build_date": "Fri, 23 Jun 2023 15:26:26 GMT",
    "commit_date": "Thu, 22 Jun 2023 09:21:26 -0600"
  },
  "config": {
    "type": "custom:frigate-card",
    "cameras": [
      {
        "live_provider": "go2rtc",
        "go2rtc": {
          "modes": [
            "webrtc"
          ],
          "stream": "doorbell_main"
        },
        "camera_entity": "camera.doorbell"
      }
    ],
    "menu": {
      "buttons": {
        "microphone": {
          "enabled": true,
          "priority": 100,
          "type": "toggle"
        },
        "download": {
          "enabled": false
        },
        "frigate": {
          "enabled": true
        },
        "cameras": {
          "enabled": false
        },
        "mute": {
          "enabled": true,
          "priority": 85
        },
        "media_player": {
          "enabled": false
        },
        "fullscreen": {
          "priority": 34
        }
      },
      "style": "hidden"
    },
    "live": {
      "controls": {
        "builtin": false
      }
    }
  }
}

Javascript errors shown in the web inspector (if applicable):

Additional information:

Frigate config included in case it is useful.

mqtt:
  host: 192.168.1.239
  port: 1883
  user: user
  password: pass

detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: True
  retain:
    days: 31
    mode: all
  events:
    retain:
      default: 31
      mode: active_objects

snapshots:
  enabled: True

ffmpeg:
  hwaccel_args: preset-vaapi

birdseye:
  enabled: false

go2rtc:
  streams:
    doorbell_main:
      - "ffmpeg:http://192.168.1.132/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass#video=copy#audio=copy#audio=opus"
      - rtsp://user:pass@192.168.1.132:554/h264Preview_01_main
    doorbell_sub:
      - "ffmpeg:http://192.168.1.132/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pass"
    rear_main:
      - rtsp://user:pass@192.168.1.210:554/h264Preview_01_main
    rear_sub:
      - rtsp://user:pass@192.168.1.210:554/h264Preview_01_sub
  webrtc:
    candidates:
      - 192.168.1.61:8555
      - stun:8555
  ffmpeg:
    bin: ffmpeg
    volume: "-af \"volume=30dB\""

cameras:
  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/doorbell_main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/doorbell_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      width: 896
      height: 672
      fps: 5
      enabled: true
    objects:
      track:
      - person
      - dog
      - cat
    motion:
      mask: 
        - 896,0,896,169,896,267,660,296,547,233,470,255,377,244,237,230,97,230,0,253,0,0
        - 0,672,425,672,425,630,0,631

  rear:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/rear_main?video=copy&audio=opus
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/rear_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      width: 640
      height: 360
      fps: 5
      enabled: true
    objects:
      track:
      - person
      - dog
      - cat
    motion:
      mask: 
        - 0,360,225,360,224,334,129,338,140,206,124,125,255,0,0,0
        - 296,360,640,360,640,0,367,0,490,67,513,315

FeatherKing commented 3 months ago

came here looking for a reported issue, i am seeing similar behavior. It seems like it is happening more often to me (at least 50%)

dermotduffy commented 3 months ago

This is because either the integration, or the Frigate backend, is returning an error to the card for some reason. I'd start by checking your Home Assistant logs and seeing if there's a log entry that aligns with the time you receive this message and you may get a clue. The request is likely triggered by fetching events/thumbnails, disabling the thumbnails would be a way to avoid the request being sent (and thus the errors being received).

btrudgill commented 3 months ago

This is the error log that is shown in HA, I don't know if there's a way of getting more detailed logs/info?

This error originated from a custom integration.

Logger: custom_components.frigate.api
Source: custom_components/frigate/api.py:244
Integration: Frigate (documentation, issues)
First occurred: 2:04:59 PM (1 occurrences)
Last logged: 2:04:59 PM

Timeout error fetching information from http://192.168.1.61:5005/api/events?cameras=doorbell&limit=50&include_thumbnails=0:

Sure, disabling the thumbnails/events may get rid of the error, but it isn't a fix to the issue.

FeatherKing commented 3 months ago

i run my frigate separate from HA. on the frigate side it is returning http 499

frigate | 2024-02-28 16:49:09.424868720 192.168.0.24 - - [28/Feb/2024:16:49:09 -0700] "GET /api/events?cameras=driveway-cam&limit=50&has_snapshot=1&include_thumbnails=0 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.2.4 aiohttp/3.9.3 Python/3.12" "-"

in my case, it also crashes my camera feeds

frigate | 2024-02-28 16:49:58.577091561 2024/02/28 16:49:58 [error] 146#146: *71483 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.24, server: , request: "GET /api/events?cameras=hagdrivewaycam&limit=50&has_snapshot=1&include_thumbnails=0 HTTP/1.1", upstream: "http://127.0.0.1:5001/events?cameras=hagdrivewaycam&limit=50&has_snapshot=1&include_thumbnails=0", host: "frigate.featherstone.xyz:15000"

and it seems to cause insane i/o (frigate isnt logging anything helpful during this i/o load)

i only experience this i/o load when i hit the frigate haas card. I have three instances of the card on my dashboard (three separate cameras). Frigate will never recover if i leave the haas card open. Once the dashboard/card is closed, frigate returns to normal.

The frigate logs show that there are several calls made from the haas card. Looks like 4 separate calls for each camera within about 3 seconds. I ran a watch command with a 0.1 timer

❯ watch -n 0.1 curl "http://a.b.xyz/api/events?cameras=driveway-cam&limit=50&has_snapshot=1&include_thumbnails=0"

and this also caused my i/o instantly. maybe this is an underlying frigate issue responding to several quick api calls

edit: is it possible to limit how many events the card pulls in from frigate?

dermotduffy commented 3 months ago

edit: is it possible to limit how many events the card pulls in from frigate?

Yes:

performance:
  features:
    media_chunk_size: 5  // <------------- Default 50

You can also disable thumbnails entirely. Take a look at this issue (https://github.com/dermotduffy/frigate-hass-card/issues/1304), it could be related. Do you see the duplicate log entries in your nginx log as that reporter shows?

btrudgill commented 3 months ago

changing media_chunk_size on mine to 10 seems to stop it throwing an error.

Trying to look at the nginx logs but they seem a mess on my browser at work, will check when I get home. The user in that issue looks to have a better viewer for the logs? Is there a way to view them that isn't the default 'logs' section in Frigate?

Edit: it seems like changing the media chunk size to 10 hasn't solved the issue, I still get the errors.

unrly commented 2 months ago

Having same issue. Reduced to 5 and still getting this message.

skykingjwc commented 2 months ago

I am getting this issue as well. Apparently it takes only a single occurrence of a failed API call to make the card crash.

This error originated from a custom integration.

Logger: custom_components.frigate.api
Source: custom_components/frigate/api.py:259
integration: Frigate (documentation, issues)
First occurred: 5:35:19 PM (1 occurrences)
Last logged: 5:35:19 PM

Error fetching information from http://cameranuc.local:5000/api/stats: Cannot connect to host cameranuc.local:5000 ssl:default [Invalid argument]

When I cut and paste the /api/stats URL into a browser, I get the correct JSON response as you would expect. Continuous pings from the HA server to the Frigate server during the time of the failure do not show any disconnects, and the Frigate server does not show any errors in the Frigate, Go2rtc or Nginx logs. Only 200 OK replies from the endpoint.

I also had VLC running a stream from the same Go2rtc as the Frigate card, and it continues to run correctly at the same time that Frigate card failed. This combined with the "invalid argument "card error log leads me to believe it's something in the card code itself causing the issue. The card eventually recovers the video stream on its own, but the stream could be down anywhere from a few seconds to several minutes.

I already reduced my media chunk size to 10, so that's not the issue.

ronluna commented 1 month ago

I’ve been trying to get to the bottom of this issue for a couple of month now. I really believe this is a SQLite issue not being able to respond to the queries in time the frigate-has-card sends it to the server . And for some reason it halts the frigate server and restart the entire docker container .

Frequent vacuuming (daily reboots) may improve the SQLite . Another way I have manage to improve SQLite response is by sending an event request to the frigate server for each camera every hour. This for some reason improves the response time for each query… it does not completely solves it but at least is something …

sstratoti commented 1 month ago

I've got one page where I have 6 cards on it, all displaying a live feed with a 5 chunk. When I load the page, it shows the feed for a split second and then they all crash. If I set the chunk to zero, they display fine. Is there any way to lazy load these images / events? I wonder if them all hitting the server at once is killing it.

ronluna commented 1 month ago

I've got one page where I have 6 cards on it, all displaying a live feed with a 5 chunk. When I load the page, it shows the feed for a split second and then they all crash. If I set the chunk to zero, they display fine. Is there any way to lazy load these images / events? I wonder if them all hitting the server at once is killing it.

Currently your best bet is to disable the thumbnails while on live view by disabling the thumbnails while live viewing, you can always access the event list by clicking on the event button.

live: controls: thumbnails: mode: none

also creating an automation to send a request to the server per camera will also helps the SQLite response time.

In your HA configuration.yaml file add shell commands as:

shell_command: refresh_frigate_events_camera_name_x : 'curl -k -X GET "http://frigate_ip:5000/events?cameras=camera_name_x"' refresh_frigate_events_camera_name_y : 'curl -k -X GET "http://frigate_ip:5000/events?cameras=camera_name_y"' refresh_frigate_events_camera_name_z : 'curl -k -X GET "http://frigate_ip:5000/events?cameras=camera_name_z"'

Then create an automation in HA that runs on a time pattern as:

` alias: Refresh Frigate Events Database Table per Camera description: Refresh Frigate Events Database Table per Camera trigger:

platform: time_pattern minutes: /45 action:
service: shell_command.refresh_frigate_events_camera_name_x data: {}
delay: hours: 0 minutes: 0 seconds: 15 milliseconds: 0
service: shell_command.refresh_frigate_events_camera_name_y data: {}
delay: hours: 0 minutes: 0 seconds: 15 milliseconds: 0
service: shell_command.refresh_frigate_events_camera_name_z data: {}
delay: hours: 0 minutes: 0 seconds: 15 milliseconds: 0 `

Give enough time between request for frigate's SQLite to respond to the request that has been sent by the automation. If you notice that accessing events for certain cameras are taking longer than 15 seconds then increase the delay for each request inside your ha automation.

lordbah commented 3 weeks ago

Just started getting this. Removing thumbnails and reducing chunk size down to 2 hasn't helped. It logs that timeout very quickly. HA log 2024-05-07 16:33:34.526 ERROR (MainThread) [custom_components.frigate.api] Timeout error fetching information from http://homeassistant.local:5000/api/events?cameras=driveway&limit=2&include_thumbnails=0: Frigate nginx log (seems happy) 2024-05-07 16:33:34.741711164 172.20.0.1 - - [07/May/2024:16:33:34 -0400] "GET /api/events?cameras=driveway&limit=2&include_thumbnails=0 HTTP/1.1" 200 0 "-" "HomeAssistant/2024.4.4 aiohttp/3.9.5 Python/3.12" "-"

Chunk size 0 seems to be working so far.

ADD: I am not seeing the duplicate log entries of #1304. I do see the error with chunk size 1. Quite frequently now. I had been running nearly a year without any config related to retaining recordings or events. So I hypothesized that the database may have become quite large. I've added some config to retain only 30 days, but I don't see any evidence in the logs that expiration is running (logger/logs/frigate.event: debug). Updated Frigate to 0.13.2 and still see the error.

ADD: In some cases I see this error on screen and the error above in the HA log, yet the Frigate nginx log does not show any access to an "events" URL at all, nor does the log of the Apache server which is the reverse proxy fronting Frigate. Odd.

ADD: Then one day the problem just vanished.

ADD: And it's back again ... and there was a 499 in the nginx log this time.

dermotduffy / frigate-hass-card

API error whilst retrieving events for camera #1382