blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.81k stars 1.71k forks source link

Non-monotonous DTS in output stream 1:0 #1826

Closed fabltd closed 2 years ago

fabltd commented 3 years ago

Describe the bug

Soon after starting frigate the logs fill up with ffmeg errors. I have been trying to find a solution and I see this issue has come up a number of times but there does not seem to be a comprehensive fix.

Is there a way to check what ffmeg command is being called and to test that to prove its ffmeg and not frigate?

The errors are as follows:

frigate    |  * Starting nginx nginx
frigate    |    ...done.
frigate    | Starting migrations
frigate    | peewee_migrate                 INFO    : Starting migrations
frigate    | There is nothing to migrate
frigate    | peewee_migrate                 INFO    : There is nothing to migrate
frigate    | frigate.mqtt                   INFO    : MQTT connected
frigate    | frigate.app                    INFO    : Camera processor started for front_door: 34
frigate    | frigate.app                    INFO    : Capture process started for front_door: 35
frigate    | detector.coral                 INFO    : Starting detection process: 31
frigate    | frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate    | frigate.edgetpu                INFO    : TPU found
frigate    | frigate.video                  INFO    : front_door: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : front_door: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : front_door: ffmpeg process is not running. exiting capture thread...
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5633c2eaf2c0] Non-monotonous DTS in output stream 0:0; previous: 56132518, current: 1257247; changing to 56132519. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5633c2eb9d80] Non-monotonous DTS in output stream 1:0; previous: 3654452, current: 81852; changing to 3654452. This may result in incorrect timestamps in the output file.

I also see:

frigate    | ffmpeg.front_door.detect       ERROR   : [h264 @ 0x55cb4c2a2c80] error while decoding MB 94 57, bytestream -31
frigate    | ffmpeg.front_door.detect       ERROR   : rtsp://**(Changed)@cameraIP **/rtsph2641080p: corrupt decoded frame in stream 0 
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x55cb4c0ea800] Failed to update header with correct duration.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x55cb4c0ea800] Failed to update header with correct filesize.

Note have removed camera authentication details.

Version of frigate Output from /api/version

0.8.4-5043040

Config file Include your full config file wrapped in triple back ticks.

 mqtt:
   # Required: host name
   host: *********
   # Optional: port (default: shown below)
   port: 1883
   # Optional: topic prefix (default: shown below)
   # WARNING: must be unique if you are running multiple instances
   topic_prefix: frigate
   # Optional: client id (default: shown below)
   # WARNING: must be unique if you are running multiple instances
   client_id: frigate
   # Optional: user
   user: mqtt
   # Optional: password
   # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
   #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
   password: *******
   # Optional: interval in seconds for publishing stats (default: shown below)
   stats_interval: 60

 cameras:
   front_door:
     ffmpeg:
       inputs:
         - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD_0}@******/rtsph2641080p
           roles:
             - detect
             - rtmp
             - clips
     width: 1920
     height: 1080
     fps: 5

     # Optional: Camera level detect settings
     detect:
       enabled: True
       # Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
       max_disappeared: 25

     # Optional: save clips configuration
     clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 15

     rtmp:
     # Required: Enable the live stream (default: True)
        enabled: True```

**Frigate container logs**

See above


**Frigate stats**
```json

{
  "detection_fps": 0.0, 
  "detectors": {
    "coral": {
      "detection_start": 0.0, 
      "inference_speed": 10.34, 
      "pid": 31
    }
  }, 
  "front_door": {
    "camera_fps": 5.0, 
    "capture_pid": 35, 
    "detection_fps": 0.0, 
    "pid": 34, 
    "process_fps": 5.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 63.4, 
        "mount_type": "tmpfs", 
        "total": 67.1, 
        "used": 3.7
      }, 
      "/media/frigate/clips": {
        "free": 3265659.9, 
        "mount_type": "nfs4", 
        "total": 3266837.0, 
        "used": 1177.0
      }, 
      "/media/frigate/recordings": {
        "free": 3265659.9, 
        "mount_type": "nfs4", 
        "total": 3266837.0, 
        "used": 1177.0
      }, 
      "/tmp/cache": {
        "free": 985.3, 
        "mount_type": "tmpfs", 
        "total": 1000.0, 
        "used": 14.7
      }
    }, 
    "uptime": 42, 
    "version": "0.8.4-5043040"
  }

FFprobe from your camera

Run the following command and paste output below

ffprobe version 4.2.4-1ubuntu0.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, rtsp, from 'rtsp://*******/rtsph2641080p':
  Metadata:
    title           : RTSP/RTP stream from HDIPCam
  Duration: N/A, start: 0.327189, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 15 fps, 25.08 tbr, 90k tbn, 30 tbc

Screenshots If applicable, add screenshots to help explain your problem.

Computer Hardware

Ubuntu 20.04.3 LTS

Docker Compose - See config:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: blakeblackshear/frigate:stable-amd64
  #  shm_size: '256mb'
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./frigate.yml:/config/config.yml:ro
      - /mnt/cctv:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds

Esxi V7

USB

Wired

``` Camera Info:**

blakeblackshear commented 3 years ago

This just means ffmpeg is struggling to consistently read the video stream from your camera. If it isn't recoverable, the process exits and frigate restarts ffmpeg.

You could try a less aggressive set of input_args for ffmpeg. Example here: https://github.com/blakeblackshear/frigate/issues/1249#issuecomment-864394891

fabltd commented 3 years ago

Ok will try. As FFMEG crashes. Is the bottle neck the camera?

blakeblackshear commented 3 years ago

Could be lots of things. Camera could be overloaded if you are connected from multiple clients. It could be network issues, or your camera just may not output a very reliable stream.

fabltd commented 3 years ago

Thanks.

Testing now.

I have one other issue I assume was related.

Clips all seem to show a time of 1:00:38. The clip still is correct but the play back doesn't appear to show the event.

blakeblackshear commented 3 years ago

If ffmpeg is crashing, its going to break the clips too.

fabltd commented 3 years ago

Testing still shows error:

frigate    | ffmpeg.front_door.detect       ERROR   : More than 1000 frames duplicated
frigate    | ffmpeg.front_door.detect       ERROR   : [rtsp @ 0x5628a7157cc0] DTS 146912157824107 < 146912479227747 out of order
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963678, current: 5110790; changing to 59963679. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332734; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963679, current: 5110909; changing to 59963680. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332741; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963680, current: 5111091; changing to 59963681. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332753; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963681, current: 5111146; changing to 59963682. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332757; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963682, current: 5111301; changing to 59963683. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332767; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963683, current: 5111393; changing to 59963684. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332773; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [h264 @ 0x5628a7326d40] error while decoding MB 29 24, bytestream -6
frigate    | ffmpeg.front_door.detect       ERROR   : [segment @ 0x5628a717f0c0] Non-monotonous DTS in output stream 0:0; previous: 59963684, current: 5111442; changing to 59963685. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Non-monotonous DTS in output stream 1:0; previous: 3903885, current: 332776; changing to 3903885. This may result in incorrect timestamps in the output file.
frigate    | ffmpeg.front_door.detect       ERROR   : rtsp://*******/rtsph2641080p: corrupt decoded frame in stream 0
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Failed to update header with correct duration.
frigate    | ffmpeg.front_door.detect       ERROR   : [flv @ 0x5628a7199c40] Failed to update header with correct filesize.
fabltd commented 3 years ago

Ok so the odd time of 1 hour is rreleated not some configuration mismatch

blakeblackshear commented 3 years ago

Note that all messages from the underlying ffmpeg process come across as type ERROR regardless of what the actual message is. Some of these are just informational. These messages are up to the last 100 lines output by ffmpeg before the process exited. Try increasing the log level of ffmpeg:

 cameras:
   front_door:
     ffmpeg:
       inputs:
         - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD_0}@******/rtsph2641080p
           global_args: -hide_banner -loglevel info
           roles:
             - detect
             - rtmp
             - clips
fabltd commented 3 years ago

Thanks will add.

I have changed to the 720P stream from the camera and this seems to have stopped the errors for now.

Any idea why the cameras are not presented to homekit. I have added them as a source but not appearing on my phone.

blakeblackshear commented 3 years ago

I don't know enough about homekit or your setup to help with that. Frigate cameras in home assistant should work like any other camera.

fabltd commented 3 years ago

Thanks. I got it working now the stream is more stable.

Is there anyway to stop the camera user name and password being printed in the logs.

blakeblackshear commented 3 years ago

Not at the moment

fabltd commented 3 years ago

Thanks. Also any way to clear the last person detected.

fabltd commented 3 years ago

Still seeing errors but less than there were. Is there any way to check FFMPEGS memory allocation.

fabltd commented 3 years ago

Bump

blakeblackshear commented 3 years ago

It hasn't even been 24 hours. You haven't provided any additional information since the last suggestion I made to increase the log level for ffmpeg.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.