blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.06k stars 1.74k forks source link

[Support]: Recordings folder deleted after Frigate crash #2573

Closed jasonpstokes closed 2 years ago

jasonpstokes commented 2 years ago

Describe the problem you are having

Over the last few days Frigate 0.10 beta has crashed and needed restarting a couple times. Today though when it failed it erased my entire recordings folder!? Screenshots are still there.

image (this is from inside the Frigate container)

Full log dump below. If there is anything else I can give please let me know?

Version

0.10.0-DB1255A

Frigate config file

database:
  path: /media/frigate/database/frigate.db

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: 192.168.1.5
  port: 1883
  user: mqtt
  password: "{FRIGATE_MQTT_PASSWORD}"

birdseye:
  mode: continuous
  width: 1792
  height: 1344

detect:
  width: 896
  height: 672
  fps: 5

ffmpeg:
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
  input_args: -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000 # https://github.com/blakeblackshear/frigate/issues/1713#issuecomment-932976305
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy

live:
  height: 672
  quality: 12

motion:
  mask:
    - 265,650,265,625,630,625,630,650

record:
  enabled: True
  retain:
    days: 0
  events:
    retain:
      default: 14
      mode: active_objects

rtmp:
  enabled: False

snapshots:
  enabled: true
  clean_copy: false
  bounding_box: true
  retain:
    default: 14

timestamp_style:
  format: "%d/%m/%Y %H:%M:%S"

cameras:
  Driveway:
    ffmpeg:
      inputs:
        - path: http://192.168.1.71/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.71/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person
        - vehicle
        - dog
      filters:
        dog:
          max_area: 30000
        person:
          mask: 
            - 510,120,540,120,540,160,510,160

  Entrance:
    ffmpeg:
      inputs:
        - path: http://192.168.1.72/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.72/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person
        - dog
      filters:
        person:
          mask: 72,449,106,568,209,569,214,449

  Rumpus:
    ffmpeg:
      inputs:
        - path: http://192.168.1.73/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.73/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person
        - dog

  Deck:
    ffmpeg:
      inputs:
        - path: http://192.168.1.74/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.74/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person

  Laundry:
    ffmpeg:
      inputs:
        - path: http://192.168.1.75/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.75/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person

  Balcony:
    ffmpeg:
      inputs:
        - path: http://192.168.1.76/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect
        - path: http://192.168.1.76/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - record
    objects:
      track:
        - person
        - vehicle

Relevant log output

log uploaded below.

FFprobe output from your camera

**RTSP stream, which I don't use.

ffprobe version 4.4-full_build-www.gyan.dev Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 10.2.0 (Rev6, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 6 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 7 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 9 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 4 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 6 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 7 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 5 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 9 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[h264 @ 00000223019f6100] left block unavailable for requested intra4x4 mode -1
[h264 @ 00000223019f6100] error while decoding MB 0 98, bytestream 75495
[h264 @ 00000223019f6100] concealing 3569 DC, 3569 AC, 3569 MV errors in I frame
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 7 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 7 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 10 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 8 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[h264 @ 00000223019f6100] concealing 18267 DC, 18267 AC, 18267 MV errors in P frame
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 7 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[h264 @ 00000223019f6100] concealing 15891 DC, 15891 AC, 15891 MV errors in P frame
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 5 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 4 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 6 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 5 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 2 packets
[h264 @ 00000223019f6100] concealing 18386 DC, 18386 AC, 18386 MV errors in P frame
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 4 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 15 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 14 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 3 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
[h264 @ 00000223019f6100] concealing 10311 DC, 10311 AC, 10311 MV errors in P frame
[rtsp @ 00000223019da980] max delay reached. need to consume packet
[rtsp @ 00000223019da980] RTP: missed 1 packets
Input #0, rtsp, from 'rtsp://user:pass@192.168.1.71:554//h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000500, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 30 fps, 30 tbr, 90k tbn, 180k tbc
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Frigate stats

{"Balcony":{"camera_fps":5.1,"capture_pid":259,"detection_fps":0.6,"pid":234,"process_fps":5.1,"skipped_fps":0.0},"Deck":{"camera_fps":5.0,"capture_pid":244,"detection_fps":0.1,"pid":231,"process_fps":5.0,"skipped_fps":0.0},"Driveway":{"camera_fps":5.0,"capture_pid":236,"detection_fps":2.1,"pid":226,"process_fps":5.0,"skipped_fps":0.0},"Entrance":{"camera_fps":5.1,"capture_pid":238,"detection_fps":0.0,"pid":229,"process_fps":5.1,"skipped_fps":0.0},"Laundry":{"camera_fps":5.1,"capture_pid":248,"detection_fps":0.0,"pid":233,"process_fps":5.1,"skipped_fps":0.0},"Rumpus":{"camera_fps":5.1,"capture_pid":242,"detection_fps":0.3,"pid":230,"process_fps":5.1,"skipped_fps":0.0},"detection_fps":3.1,"detectors":{"coral":{"detection_start":0.0,"inference_speed":14.58,"pid":218}},"service":{"storage":{"/dev/shm":{"free":59.0,"mount_type":"tmpfs","total":67.1,"used":8.1},"/media/frigate/clips":{"free":1731111.2,"mount_type":"nfs4","total":1921292.6,"used":189644.5},"/media/frigate/recordings":{"free":1731111.2,"mount_type":"nfs4","total":1921292.6,"used":189644.5},"/tmp/cache":{"free":966.2,"mount_type":"tmpfs","total":1073.7,"used":107.5}},"temperatures":{},"uptime":466,"version":"0.10.0-db1255a"}}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Reolink RLC-520A

Any other information that may be helpful

_frigate_logs.txt

blakeblackshear commented 2 years ago

Can you post your compose file?

jasonpstokes commented 2 years ago
services:
  frigate:
    container_name: frigate
    image: blakeblackshear/frigate:0.10.0-beta6-amd64
    hostname: frigate
    restart: always
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/bus/usb:/dev/bus/usb
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /usr/docker/frigate10/config.yml:/config/config.yml:ro
      - /usr/docker/frigate10/labelmap.txt:/labelmap.txt:ro
      - /usr/docker/frigate10/database:/media/frigate/database
      - /mnt/nas/frigate10:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: '1gb'
    network_mode: bridge
    ports:
      - 5000:5000
    environment:
      FRIGATE_MQTT_PASSWORD: ""
      FRIGATE_USER: ""
      FRIGATE_PASSWORD: ""
      LIBVA_DRIVER_NAME: "radeonsi"
blakeblackshear commented 2 years ago

There are tons of segmentation faults in your logs, but I don't think that's related to the recordings being deleted. I would recommend setting a shm_size in your compose file to see if that resolves the frequent segmentation faults.

I assume you started seeing the Unable to find file from recordings database messages in the logs after the files were deleted. If so, this means the recording entries still exist in the database. There are 2 places where frigate deletes files from the recordings directory and both of them remove the database entry at the same time. This indicates that the files were deleted external to frigate.

There is one other location that cleans up old recordings that do not have a database entry, but it only deletes files older than the last entry in the database, so I don't see how this could have happened there either.

Unfortunately, I don't see any answer as to why this happened. You may want to turn on debug output for the record process to get more info for what the cleanup jobs are doing in case it happens again.

logger:
  default: info
  logs:
    frigate.record: debug
jasonpstokes commented 2 years ago

Thanks for looking into this, and your response - appreciated!

No idea what might have deleted those recordings outside of Frigate, and it wasn't a NAS mount issue. They're just gone! Oh well, I'll see if it happens again and what else might be going on.

I've added shm_size: '256m' to my compose file and restarted, however the SHM size required is a little confusing. The docs say 64m is fine for 2 cameras, so I'd assume I need approx. 4x more with 6 cameras, but using the provided formula: (width * height * 1.5 * 9 + 270480)/1048576 I get: (896 * 672 * 1.5 * 9 + 270480)/1048576 = 8.00990295410156 6 cameras * 8.00990295410156 = 48m ...well below 64m! Have I got that wrong, or should there be an extra zero somehow?

PS Updated website looks great!

EDIT: updated for the revised calculation

jasonpstokes commented 2 years ago

In 24 hours I've had a cache full error and 7 seg faults - is it something with my host or docker setup, or something in Frigate? (Plain Debian 11 with Docker, all up to date.)

This is after deleting the database and recording folder yesterday, to start from scratch after further issues. Full log since attached.

_frigate_logs.txt

I also increased the shm size to 320m when I cleared the database yesterday.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

blakeblackshear commented 2 years ago

Wanted to check in on this and point out that beta 8 is now available. I don't think anything specifically would have addressed any of this, but wasn't sure if you were still having these issues.

jasonpstokes commented 2 years ago

Yesterday I set up Docker on a desktop to test and Frigate has run without issue since. Mixed feelings about this!! It's good that Frigate is running well now (not Frigate's fault) but sadly it points to my homelab server having issues.

Can you please confirm the shm formula though? If my calculations above are right I only need 48mb...which doesn't seem right! (I'm running 320m)

Love the development. Any chance a UI update (esp. for mobile) will make the roadmap soon? :-)

blakeblackshear commented 2 years ago

Your calculation looks right to me. That should be enough shm storage unless something is not getting cleaned up properly.

There are some UI updates brewing from a few users. I have some major backend updates planned for the next version, but I definitely have my eye on some UI improvements after that.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.