blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.17k stars 1.75k forks source link

[Support]: Error with Frigate Addon and SMB storage on OS 11.3 #9218

Closed ovizii closed 10 months ago

ovizii commented 10 months ago

Describe the problem you are having

I always keep HA, the HACS integrations and HA add-ons up to date. Recently there have been Frigate updates as well as HA updates, so I am unsure what caused the problem.

I apologize in advance for the information overload :-(

The problem started with the Frigate lovelace card reporting:

{
  "request": {
    "type": "frigate/events/get",
    "instance_id": "frigate",
    "cameras": [
      "camera_1"
    ],
    "limit": 50,
    "id": 46
  },
  "response": {
    "code": "not_found",
    "message": "Unable to find Frigate instance with ID: frigate"
  }
}

Using:

HA logs: 2024-01-05 13:29:45.435 ERROR (MainThread) [custom_components.frigate.api] Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: 500, message='INTERNAL SERVER ERROR', url=URL('http://ccab4aaf-frigate-fa:5000/api/stats')

While checking everything for related problems, I saw the Frigate integration “needs attention”. image

I tried configuring the integration again with these settings but it says could not connect: I tried these settings with the Frigate integration:

http://ccab4aaf-frigate-fa:5000 and 
rtsp://ccab4aaf-frigate-fa:8554

I have double-checked and the settings of the Frigate add-on correspond to these ports.

If I visit HA dashboard => Frigate I see my live camera working just fine. If I click on “events” and “recordings” I see past events and recordings and can play them BUT going to Frigate => Logs there is a problem:

2024-01-05 11:49:13.005769404  [INFO] Starting Frigate...
2024-01-05 11:49:15.715862323  [2024-01-05 13:49:15] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2024-01-05 11:49:15.744532558  [2024-01-05 13:49:15] frigate.app                    INFO    : Creating directory: /tmp/cache
2024-01-05 11:49:15.747085544  [2024-01-05 13:49:15] peewee_migrate                 INFO    : Starting migrations
2024-01-05 11:49:15.758893603  [2024-01-05 13:49:15] peewee_migrate                 INFO    : There is nothing to migrate
2024-01-05 11:49:15.796172286  [2024-01-05 13:49:15] frigate.app                    INFO    : Output process started: 227
2024-01-05 11:49:15.824339629  [2024-01-05 13:49:15] frigate.app                    INFO    : Camera processor started for cam01: 230
2024-01-05 11:49:15.846534417  [2024-01-05 13:49:15] detector.cp                    INFO    : Starting detection process: 226
2024-01-05 11:49:15.846541844  [2024-01-05 13:49:15] frigate.detectors              WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
2024-01-05 11:49:15.856689386  [2024-01-05 13:49:15] frigate.app                    INFO    : Capture process started for cam01: 232
2024-01-05 11:49:24.300362899  [2024-01-05 13:49:24] frigate.http                   ERROR   : Exception on /stats [GET]
2024-01-05 11:49:24.300365055  Traceback (most recent call last):
2024-01-05 11:49:24.300366248    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2529, in wsgi_app
2024-01-05 11:49:24.300367183      response = self.full_dispatch_request()
2024-01-05 11:49:24.300368873    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
2024-01-05 11:49:24.300370981      rv = self.handle_user_exception(e)
2024-01-05 11:49:24.300371925    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
2024-01-05 11:49:24.300380645      rv = self.dispatch_request()
2024-01-05 11:49:24.300381570    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
2024-01-05 11:49:24.300394076      return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-01-05 11:49:24.300394931    File "/opt/frigate/frigate/http.py", line 827, in stats
2024-01-05 11:49:24.300395667      stats = stats_snapshot(
2024-01-05 11:49:24.300396442    File "/opt/frigate/frigate/stats.py", line 253, in stats_snapshot
2024-01-05 11:49:24.300406125      storage_stats = shutil.disk_usage(path)
2024-01-05 11:49:24.300407398    File "/usr/lib/python3.9/shutil.py", line 1262, in disk_usage
2024-01-05 11:49:24.300408491      st = os.statvfs(path)
2024-01-05 11:49:24.300409714  BlockingIOError: [Errno 11] Resource temporarily unavailable: '/media/frigate/recordings'

This error keeps repeating every minute: BlockingIOError: [Errno 11] Resource temporarily unavailable: '/media/frigate/recordings'

Not sure what is the problem since Frigate can actually view events and recordings and play them for me.

When setting up Frigate initially I followed this guide: https://docs.frigate.video/guides/ha_network_storage/ and made Frigate use the mounted SMB share.

On the terminal the folder does exist and is accessible and Frigate can indeed access it as shown above.: msOp2fCLDh

Btw. when visiting HA Dashboard => Frigate I can only access the initial page via https://mydomain.tld/ccab4aaf_frigate-fa/ingress and the birdseye, events, config and log page. The system and storage pages are resulting in server error 500. image

This last bit of info might be unrelated but it looks confusing.

Version

UnknownToMe

Frigate config file

# full config https://docs.frigate.video/configuration/index/#full-configuration-reference 
# further clarification: 
# https://docs.frigate.video/configuration/masks/#further-clarification
# https://docs.frigate.video/configuration/record/

# below is for saving frigate data to SMB share
# https://docs.frigate.video/guides/ha_network_storage/
database:
  path: /config/frigate.db

mqtt:
  host: core-mosquitto
  user: mosquitto
  password: secret

detectors:
    cp:
      type: cpu

ffmpeg:
  output_args:
# replaced by preset
#    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
#    record: preset-record-generic-audio-copy
    record: preset-record-generic # record without audio

# https://github.com/AlexxIT/go2rtc
# Format: ffmpeg:{input}#{param1}#{param2}#{param3}
# i.e. #video=h264 #video=copy #audio=opus #backchannel=0 
# these can be added to any type of stream, not necessarily to ffmpeg streams.
go2rtc:
  streams:
    cam01:
      - rtsp://frigate:secret@192.168.95.2:554/Streaming/channels/101?transportmode=unicast&profile=Profile_1
#      - "ffmpeg:cam01#audio=opus" # <- copy of the stream which transcodes audio to the missing codec (usually will be opus)
    cam01_sub:
      - rtsp://frigate:secret@192.168.95.2:554/Streaming/channels/102?transportmode=unicast&profile=Profile_2
#      - "ffmpeg:cam01_sub#audio=opus" # <- copy of the stream which transcodes audio to the missing codec (usually will be opus)
  webrtc:
    candidates:
      - 192.168.96.2:8555
      - stun:8555
# sample logging levels and their order ALL < TRACE < DEBUG < INFO < WARN < ERROR < FATAL < OFF
  log:
    level: info  # default level
    api: info
    exec: info
    ngrok: info
    rtsp: info
    streams: info
    webrtc: info

cameras:
# cam01 from Cartisoara
  cam01: # <------ Name the camera
    ffmpeg:
      input_args: preset-rtsp-restream
      inputs:
        - path: rtsp://127.0.0.1:8554/cam01
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cam01_sub
          roles:
            - detect

    detect:
      enabled: true
      fps: 6
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    live:
      quality: 6
#      height: 720      
  # this configures the snapshots sent over mqtt
    mqtt:
      enabled: true
      bounding_box: true
      timestamp: false
      crop: false
      height: 480
      quality: 95

    objects:
      track:
        - person
        - dog
      filters:
        person:
          threshold: 0.7
          min_area: 60000
        dog:
          threshold: 0.7
          min_area: 15000
# read more about zones at https://docs.frigate.video/configuration/zones
# combine zones with motion masks https://docs.frigate.video/configuration/masks/
    zones:
      door:
        coordinates: 933,592,1231,0,621,0,646,207,664,395
      window:
        coordinates: 118,67,235,0,380,0,460,241,293,379
      kitchen:
        coordinates: 1213,246,1055,720,859,720
      bedroom:
        coordinates: 328,720,496,720,107,288

# each event gets one snapshot. try to keep retain in sync with events    
# this is the global section      
snapshots:
  enabled: true
  bounding_box: true
  timestamp: false
  retain:
    default: 7
    objects:
      dog: 10
      person: 10

record:
  enabled: true
# save recordings with all / motion / active_objects for XX days
  retain:
    days: 7
    mode: motion
# save recordings overlapping with events with all / motion / active_objects for xx days
  events:
    pre_capture: 10
    post_capture: 20
# add more objects from https://docs.frigate.video/configuration/objects 
# this limits the types of objects recorded
    objects:
      - person
      - dog
    retain:
      default: 7
      mode: active_objects
      objects:
        dog: 3
        person: 14

motion:
  threshold: 25
  contour_area: 35

#timestamp_style:
#  position: "br"
#  format: "%d.%m.%Y %H:%M:%S"
#  thickness: 3

birdseye:
  enabled: true
# https://docs.frigate.video/configuration/birdseye
  mode: motion

Relevant log output

2024-01-05 13:29:45.435 ERROR (MainThread) [custom_components.frigate.api] Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: 500, message='INTERNAL SERVER ERROR', url=URL('http://ccab4aaf-frigate-fa:5000/api/stats')

FFprobe output from your camera

seems unnecessary for this issue

Frigate stats

not sure how to get this

Operating system

HassOS

Install method

HassOS Addon

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Hikvision but the problem is unrelated to the camera

Any other information that may be helpful

I am accessing my HA instance behind traefik as a reverse proxy in case it matters.

ovizii commented 10 months ago

Not sure if this is important but when I set this all up as shown here: https://docs.frigate.video/guides/ha_network_storage/

I noticed that there seem to be 2 frigate folder visible in HA Gui somehow. One is directly under "media" but it seems to differ from what is visible via the terminal :-(

Via terminal:

image

Via GUI:

4tBvcFQSKg

BUT via GUI: media => my media => frigate:

ZUs95jn5bs

blakeblackshear commented 10 months ago

The stats endpoint is failing to check how much storage space is left. What version of HassOS and Home Assistant are you running?

andonevris commented 10 months ago

For me this is related to the OS 11.3 after updating I get the exact same errors.

Rolling back to OS 11.2 and it's working again.

blakeblackshear commented 10 months ago

@andonevris are you also using an SMB share for recordings?

andonevris commented 10 months ago

@andonevris are you also using an SMB share for recordings?

Yes I am

NickM-27 commented 10 months ago

https://github.com/blakeblackshear/frigate/discussions/9196 is related as well, seems HA OS 11.3 broke something

ovizii commented 10 months ago

The stats endpoint is failing to check how much storage space is left. What version of HassOS and Home Assistant are you running?

HA OS 11.3 HA Core 2024.1.0 HA Supervisor 2023.12.0

NickM-27 commented 10 months ago

Can someone who is experiencing this please make an issue on the HA OS repo https://github.com/home-assistant/operating-system/issues

ovizii commented 10 months ago

Can someone who is experiencing this please make an issue on the HA OS repo https://github.com/home-assistant/operating-system/issues

I don't mind doing that but what logs could I submit? All I have is posted in this thread and those are all Frigate logs, right?

NickM-27 commented 10 months ago

you can show that 2024-01-05 11:49:24.300409714 BlockingIOError: [Errno 11] Resource temporarily unavailable: '/media/frigate/recordings' occurs when frigate runs shutil.disk_usage(path) and this started in HA OS 11.3, they should be able to direct to get logs from the HA side but maybe there is something in superivsor logs

ovizii commented 10 months ago

Here is the created issue: https://github.com/home-assistant/operating-system/issues/3041

In the section called “Anything in the Host logs that might be useful for us?” you will see there seems to be another built-ion integration called “systemmonitor” with the same problem.

thimon commented 10 months ago

Facing the same issue here, HA OS 11.3 broke something.

HA logs: 2024-01-05 10:13:43.150 ERROR (MainThread) [custom_components.frigate.api] Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: 500, message='INTERNAL SERVER ERROR', url=URL('http://ccab4aaf-frigate-fa:5000/api/stats')

Rolling back to OS 11.2 and everything is working again

rw377 commented 10 months ago

Confirming same issue here on OS 11.3. Reverting to 11.2 (SSH > os update --version 11.2) resolves this for me.

gednet commented 10 months ago

Same issue in my system - OS 11.3, SMB share with frigate media. Rollback to version 11.2 resolves problem. I'm waiting for more info.

kiloptero commented 10 months ago

how you reverse the os? i cant his backups;(

gednet commented 10 months ago

In terminal

ha os update --version 11.2

kiloptero commented 10 months ago

thanks:) and then restart off course;)

Andres Roepke Del Solar. Ingeniero Civil Industrial Ingeniero Electronico Mobile 88196932

On Sat, Jan 6, 2024 at 08:14 gednet @.***> wrote:

ha os update --version 11.2

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/9218#issuecomment-1879639972, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIOTKWRGUWWKQHP73ADBXDTYNEWZJAVCNFSM6AAAAABBOL2BJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGYZTSOJXGI . You are receiving this because you commented.Message ID: @.***>

kiloptero commented 10 months ago

now its working thanks for all.. 11.3 has some issues

Andres Roepke Del Solar. Ingeniero Civil Industrial Ingeniero Electronico Mobile 88196932

On Sat, Jan 6, 2024 at 08:26 Andres Roepke @.***> wrote:

thanks:) and then restart off course;)

Andres Roepke Del Solar. Ingeniero Civil Industrial Ingeniero Electronico Mobile 88196932

On Sat, Jan 6, 2024 at 08:14 gednet @.***> wrote:

ha os update --version 11.2

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/9218#issuecomment-1879639972, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIOTKWRGUWWKQHP73ADBXDTYNEWZJAVCNFSM6AAAAABBOL2BJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGYZTSOJXGI . You are receiving this because you commented.Message ID: @.***>

alexlmiller commented 10 months ago

Just to echo, had the same issue here as other folks.

Rolled back to 11.2 per https://github.com/blakeblackshear/frigate/issues/9218#issuecomment-1879639972 and is working again

emilianogetino commented 10 months ago

The same for me, likewise the solution was to go back to the previous version of OS. Without a doubt Home Assistant Operating System has a problem. Please report if there is any news, thank

mclever commented 10 months ago

Haven't tested yet but looks like haos rc has found the fix https://github.com/home-assistant/operating-system/pull/3051

MaBeniu commented 10 months ago

New update with downgraded kernel version Screenshot_2024-01-08-21-27-55-97_c3a231c25ed346e59462e84656a70e50

Tested now and Nas storage now works fine for frigate with latest HAOS update

chatziko commented 10 months ago

HA OS 11.4 is out, I confirm it solves the issue.

Although this was a HA issue, I think frigate should tolerate shutil.disk_usage exceptions here, since they're not that important (it already tolerates FileNotFoundError) and it's at least the second time users report such exceptions breaking their system.

NickM-27 commented 10 months ago

frigate uses the storage info for a number of different things including the retention by available storage to ensure that it is not going to fill the recording storage. Going to close this since the actual issue is solved, but having the function tolerate no storage info will take more time and planning

kspearrin commented 7 months ago

I am facing this same issue now, however, I do not have Frigate running on HAOS. I use Ubuntu 22.04 with Docker 26.

/media/frigate is a CIFS-mounted docker volume. This has worked for years. Not sure what changed to break things. From looking at the patch HAOS repo made, it seems like they reverted some changes to fs/smb/client/smb2pdu.c, which I guess is the SMB client running on the OS?? Any ideas how I can resolve this on an Ubuntu 22 host using docker?

eohlde commented 7 months ago

I'm also facing this issue, running on ubuntu server.

I did find this: https://forums.linuxmint.com/viewtopic.php?t=417371. looks like there is a linux kernel, 5.15.0-102.

walkerleigh commented 7 months ago

Me too, but as @eohlde says the issue seems to be with kernel 5.15.0-102 rather than Frigate. This landed on my Frigate machine yesterday when I ran apt update/upgrade. Following the clues in @eohlde's link, I reverted to 5.15.0-101 and the issue disappeared. I guess I'll leave it like this until the fix trickles through to Ubuntu.

backcountrymountains commented 7 months ago

Me too, but as @eohlde says the issue seems to be with kernel 5.15.0-102 rather than Frigate. This landed on my Frigate machine yesterday when I ran apt update/upgrade. Following the clues in @eohlde's link, I reverted to 5.15.0-101 and the issue disappeared. I guess I'll leave it like this until the fix trickles through to Ubuntu.

Many thanks, I had the same issue on my Ubuntu. I'm running a Hyper-V VM and I couldn't get grub to show alternate boot options. Reference of what I did for other linux noobs: uname -r showed that I was on kernel 102. I had to sudo nano /etc/default/grub and comment out GRUB_TIMEOUT_STYLE=hidden-->#GRUB_TIMEOUT_STYLE=hidden then sudo update-grub and reboot. On reboot I selected "Advanced options for Ubuntu" and select the entry for the 5.15.0-101 kernel

Anyway, I'm glad to have found this issue as I never would've known it was an Ubuntu kernel problem otherwise.

mnl1121 commented 7 months ago

I'm also running into this issue. I have the latest HA (version 2024.4.3) in a Docker Container.

Lectere commented 1 month ago

I just had the same problems when updating to 2024.10.1 OS 13.1, Frigate 5.4.0; error message;

Logger: custom_components.frigate.api Source: custom_components/frigate/api.py:259 integration: Frigate (documentation, issues) First occurred: 21:16:54 (12 occurrences) Last logged: 21:28:13

Error fetching information from http://ccab4aaf-frigate:5000/api/stats: Cannot connect to host ccab4aaf-frigate:5000 ssl:default [Connect call failed ('172.30.33.4', 5000)]

So besides rolling back to 11.2, any information on the root cause?,

I have HAOS on hardware

NickM-27 commented 1 month ago

it means frigate addon is likely not running

Lectere commented 1 month ago

Thank you so much for pointing me in the right direction NickM-27, your right, it's not running.

That's a mighty complicated way of stating that the Frigate Addon is not running.

Somehow the update broke this, the add-on won't start, I found the root cause;

https://github.com/blakeblackshear/frigate/discussions/12980

Would be nice if the intergration would check and state 'Frigate Add-on is not running...'

NickM-27 commented 1 month ago

Would be nice if the intergration would check and state 'Frigate Add-on is not running...'

there's no way for the integration to know if it is indeed not running or something else is the problem