search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.93k stars 1.73k forks source link

Ability to record "Clips" from non-h264 sources #1096

Closed MrSiO closed 3 years ago

MrSiO commented 3 years ago

Many cheap cams are now available (eg: ESP32-CAM) and growing in popularity but do not support h.264.

And for those trying to migrate from other systems (MotionEye,BlueIris) this can be a show stopper.

Thx for considering.

blakeblackshear commented 3 years ago

This is already possible. It just requires modifying output args. See the recently updated docs for mjpeg cameras.

MrSiO commented 3 years ago

Thx alot. Oversaw that. Here's the info for others:

      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -r
        - '3'  # <---- adjust depending on your desired frame rate from the mjpeg image
        - -use_wallclock_as_timestamps
        - '1'
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        clips: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        rtmp: -c:v libx264 -an -f flv

Testing in progress

MrSiO commented 3 years ago

Getting some errors in log and no available clips:

ffmpeg.ESP32-CAM-02.detect     ERROR   : [swscaler @ 0x55cf863cc100] deprecated pixel format used, make sure you did set range correctly

frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1620951355.991992-di83bs. There were no cache files for this event.

Any hints on this? Current resolution from the ESP32-CAM: 800x600

Thx

blakeblackshear commented 3 years ago

The first is not an error. All ffmpeg messages come across that way, and that shouldn't cause issues.

The second means there aren't any cache files. Post your complete config.

MrSiO commented 3 years ago

Thx for you reply. Here is the complete config.

mqtt:
  host: 192.168.107.128
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqtt
  password: mqtt_password
  stats_interval: 60

cameras:
  ESP32-CAM-02:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -r
        - '3'  # <---- adjust depending on your desired frame rate from the mjpeg image
        - -use_wallclock_as_timestamps
        - '1'
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        clips: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        rtmp: -c:v libx264 -an -f flv

      hwaccel_args: []
#        - -hwaccel
#        - qsv
#        - -qsv_device
#        - /dev/dri/renderD128        
      inputs:
        - path: rtsp://192.168.107.38:8554/mjpeg/1
          roles:
            - detect
            - clips
    width: 800
    height: 600
    fps: 5

    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 5
    clips:
      enabled: True       
#      max_seconds: 300
#      tmpfs_cache_size: 256m
      retain:
        default: 10
        objects:
          person: 15

objects:
# Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - cup  

detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu
#  coral:
#    type: edg

Note 1: I had to leave the max_seconds & tmpfs_cache_size commented out otherwise I get the following error:

`Error parsing config: extra keys not allowed @ data['cameras']['ESP32-CAM-02']['clips']['max_seconds']
 * Starting nginx nginx
   ...done.
Error parsing config: extra keys not allowed @ data['cameras']['ESP32-CAM-02']['clips']['tmpfs_cache_size']`

Note 2: Currently running out of CPU but Coral is ordered and on the way.

Note 3: Running Docker on IntelNUC

Thx

blakeblackshear commented 3 years ago

Do you ever get any clips? If not, you must have more messages in your logs.

MrSiO commented 3 years ago

I used to get clips before adding input/output arguments but the .mp4 was not playable from browser.

Since I added the input/output arguments I do not have any clips and get the cache-file warning.

Will try again a person detection and send complete log. Standby

MrSiO commented 3 years ago

Tried again. Only log is the following and in the web interface it metions: No clip available

2021-05-14T03:08:52.618028373Z frigate.app                    WARNING : Camera ESP32-CAM-02 has record assigned to an input, but record is not enabled.
2021-05-14T03:08:52.618118078Z frigate.app                    WARNING : Camera ESP32-CAM-02 has rtmp enabled, but rtmp is not assigned to an input.
2021-05-14T03:08:52.620867441Z Starting migrations
2021-05-14T03:08:52.620959441Z peewee_migrate                 INFO    : Starting migrations
2021-05-14T03:08:52.647489859Z There is nothing to migrate
2021-05-14T03:08:52.647557786Z peewee_migrate                 INFO    : There is nothing to migrate
2021-05-14T03:08:52.654426479Z frigate.mqtt                   INFO    : MQTT connected
2021-05-14T03:08:52.682795423Z detector.cpu1                  INFO    : Starting detection process: 34
2021-05-14T03:08:52.685050516Z detector.cpu2                  INFO    : Starting detection process: 35
2021-05-14T03:08:52.693148146Z frigate.app                    INFO    : Camera processor started for ESP32-CAM-02: 39
2021-05-14T03:08:52.699776797Z frigate.app                    INFO    : Capture process started for ESP32-CAM-02: 40
2021-05-14T11:02:32.463738304Z frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1620990099.458797-8ku1sz. There were no cache files for this event.
2021-05-14T11:04:52.363737079Z frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1620990175.94513-xpanky. There were no cache files for this event.

In my media folder there are my snapshots (jpg) but no mp4.

Thx for you help.

blakeblackshear commented 3 years ago

Try changing the clips args to this. It may be an error in the docs:

clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
MrSiO commented 3 years ago

Hi. Tried this new config. Log appears a little cleaner but still no clips available (screenshot works) and no new files in my /media/clips


 * Starting nginx nginx
   ...done.
frigate.app                    WARNING : Camera ESP32-CAM-02 has record assigned to an input, but record is not enabled.
frigate.app                    WARNING : Camera ESP32-CAM-02 has rtmp enabled, but rtmp is not assigned to an input.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.cpu1                  INFO    : Starting detection process: 36
frigate.app                    INFO    : Camera processor started for ESP32-CAM-02: 41
detector.cpu2                  INFO    : Starting detection process: 37
frigate.app                    INFO    : Capture process started for ESP32-CAM-02: 42
frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1621021446.855422-8bazgt. There were no cache files for this event.
ffmpeg.ESP32-CAM-02.detect     ERROR   : Finishing stream 0:0 without any data written to it.

ffmpeg.ESP32-CAM-02.detect     ERROR   : Finishing stream 1:0 without any data written to it.

ffmpeg.ESP32-CAM-02.detect     ERROR   : [swscaler @ 0x5594f419b140] deprecated pixel format used, make sure you did set range correctly

frigate.video                  INFO    : ESP32-CAM-02: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

frigate.video                  INFO    : ESP32-CAM-02: ffmpeg process is not running. exiting capture thread...

My stream in VLC works fine with specs:

Codec: Motion JPEG Video (MJPG)
Type: Video
Video Dimension: 800x600
Buffer dimensions: 800x608
Decoded Format: Planar 4:2:2 YUV full scale
Orientation: Top Left
Color Space: ITU-R BT.601 Range
Chroma location: Center
MrSiO commented 3 years ago

Hi. Tried this new config. Log appears a little cleaner but still no clips available (screenshot works) and no new files in my /media/clips


 * Starting nginx nginx
   ...done.
frigate.app                    WARNING : Camera ESP32-CAM-02 has record assigned to an input, but record is not enabled.
frigate.app                    WARNING : Camera ESP32-CAM-02 has rtmp enabled, but rtmp is not assigned to an input.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.cpu1                  INFO    : Starting detection process: 36
frigate.app                    INFO    : Camera processor started for ESP32-CAM-02: 41
detector.cpu2                  INFO    : Starting detection process: 37
frigate.app                    INFO    : Capture process started for ESP32-CAM-02: 42
frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1621021446.855422-8bazgt. There were no cache files for this event.
ffmpeg.ESP32-CAM-02.detect     ERROR   : Finishing stream 0:0 without any data written to it.

ffmpeg.ESP32-CAM-02.detect     ERROR   : Finishing stream 1:0 without any data written to it.

ffmpeg.ESP32-CAM-02.detect     ERROR   : [swscaler @ 0x5594f419b140] deprecated pixel format used, make sure you did set range correctly

frigate.video                  INFO    : ESP32-CAM-02: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

frigate.video                  INFO    : ESP32-CAM-02: ffmpeg process is not running. exiting capture thread...

My stream in VLC works fine with specs:

Codec: Motion JPEG Video (MJPG)
Type: Video
Video Dimension: 800x600
Buffer dimensions: 800x608
Decoded Format: Planar 4:2:2 YUV full scale
Orientation: Top Left
Color Space: ITU-R BT.601 Range
Chroma location: Center

STANDBY! After restarting container and Camera, I did get a successfull Clip! I do get warnings in logs but seems to work:

frigate.events WARNING : Unable to create clip for ESP32-CAM-02 and event 1621022242.913473-wdztbr. There were no cache files for this event.

MrSiO commented 3 years ago

UPDATE: After a few tests clips do not get created every time a clip is taken. For example a snapshot was taken for this event, but no clips were produce, and this log appeared at the same time:

frigate.events WARNING : Unable to create clip for ESP32-CAM-02 and event 1621023021.050275-v89c5b. There were no cache files for this event.

Note: I do have a JPG with the event 1621023021.050275-v89c5b but no clips.

Thx for your help

MrSiO commented 3 years ago

Latest Update:

Captured clips that actually make their way to the events can be viewed using Chrome on PC but not on mobile (android) at this time.

MrSiO commented 3 years ago

@blakeblackshear I have simplified my config to the basic. With basic config (no output arguments) I do get my clips in the media folder. I get an MP4. From Frigate UI I do see there is a clip available but it wont play brower (Firefox/Chrome) and wont play on mobile Chrome. If I download the MP4 and play it from VLC, it does play.

Following your instructions, I added the ouput arguements:

clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an With the arguments in, I completely loose my live feed and get errors in logs:

`peewee_migrate INFO : There is nothing to migrate

frigate.mqtt INFO : MQTT connected frigate.app INFO : Camera processor started for ESP32-CAM-02: 41 frigate.app INFO : Capture process started for ESP32-CAM-02: 43 detector.coral INFO : Starting detection process: 38 frigate.edgetpu INFO : Attempting to load TPU as usb frigate.edgetpu INFO : TPU found watchdog.ESP32-CAM-02 INFO : No frames received from ESP32-CAM-02 in 20 seconds. Exiting ffmpeg... watchdog.ESP32-CAM-02 INFO : Waiting for ffmpeg to exit gracefully... watchdog.ESP32-CAM-02 INFO : FFmpeg didnt exit. Force killing... frigate.video INFO : ESP32-CAM-02: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : ESP32-CAM-02: ffmpeg sent a broken frame. read of closed file frigate.video INFO : ESP32-CAM-02: ffmpeg sent a broken frame. read of closed file frigate.video INFO : ESP32-CAM-02: ffmpeg sent a broken frame. read of closed file frigate.video INFO : ESP32-CAM-02: ffmpeg process is not running. exiting capture thread... [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e4e0da3140] moov atom not found /tmp/cache/ESP32-CAM-02-20210525115539.mp4: Invalid data found when processing input frigate.events INFO : bad file: ESP32-CAM-02-20210525115539.mp4 ffmpeg.ESP32-CAM-02.detect ERROR : [segment @ 0x5626fbb43900] Frame rate very high for a muxer not efficiently supporting it. ffmpeg.ESP32-CAM-02.detect ERROR : Please consider specifying a lower framerate, a different muxer or -vsync 2 ffmpeg.ESP32-CAM-02.detect ERROR : [libx264 @ 0x5626fbb45680] MB rate (171000000) > level limit (2073600) ffmpeg.ESP32-CAM-02.detect ERROR : [libx264 @ 0x5626fbb40900] MB rate (171000000) > level limit (2073600) ffmpeg.ESP32-CAM-02.detect ERROR : More than 1000 frames duplicated`

Otherwise everything else works:

Thx for any other help you can give.

blakeblackshear commented 3 years ago

Can you post your latest config so I can suggest a change? I think we need to tweak the input params so they aren't pulling from the camera so aggressively.

MrSiO commented 3 years ago

@blakeblackshear Sure! Here it is:

cameras:                                                                          
  ESP32-CAM-02:                                                                   
    ffmpeg:                                                                       
      inputs:                                                                     
        - path: rtsp://10.10.10.38:8554/mjpeg/1                                  
          roles:                                                                  
            - detect                                                              
            - rtmp                                                                
            - clips                                                               

      output_args:                                                                
        rtmp: -c:a copy -c:v libx264 -f flv                                       
    #   clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an                                                     

      hwaccel_args:                                                               
        - -hwaccel                                                                
        - vaapi                                                                   
        - -hwaccel_device                                                         
        - /dev/dri/renderD128                                                     
        - -hwaccel_output_format                                                  
        - yuv420p                                                                 

    width: 800                                                                    
    height: 600                                                                   
    fps: 5                                                                        
    best_image_timeout: 15                                                        
    detect:                                                                       
      enabled: True                                                               
      max_disappeared: 15                                                         
    clips:                                                                        
      enabled: True                                                               
    snapshots:                                                                    
      enabled: True                                                               
      timestamp: True                                                             
      bounding_box: True                                                          
      retain:                                                                     
        default: 5                                                                
    objects:                                                                      
      track:                                                                      
        - person                                                                  
        - cup                                                                     
        - cat                                                                     
#    rtmp:                                                                        
#      enabled: True                                                              

detectors:                                                                        
  coral:                                                                          
    type: edgetpu                                                                 
    device: usb

Comment out lines were causing different issues.

Side question: is RTMP stream sensitive to poor camera quality/frame rate/lost packets? Since I can't save clips (for now) I'm sending an RTMP feed to MotionEye for recording. But it's really a flip of the coin if it will work or not. I restart containers (Frigate, motioneye) and then reboot the cam. At some point I get the RTMP feed trough to MotionEye but seems completely random.

Thx alot for your time.

blakeblackshear commented 3 years ago

Give this a try. I removed the hwaccel args because they don't help for mjpeg cameras.

cameras:                                                                          
  ESP32-CAM-02:                                                                   
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - +genpts
        - -r
        - '5'
        - -use_wallclock_as_timestamps
        - '1'                                                                       
      inputs:                                                                     
        - path: rtsp://10.10.10.38:8554/mjpeg/1                                  
          roles:                                                                  
            - detect                                                              
            - rtmp                                                                
            - clips                                                               

      output_args:                                                                
        rtmp: -c:a copy -c:v libx264 -f flv                                       
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an   

    width: 800                                                                    
    height: 600                                                                   
    fps: 5                                                                        
    best_image_timeout: 15                                                        
    detect:                                                                       
      enabled: True                                                               
      max_disappeared: 15                                                         
    clips:                                                                        
      enabled: True                                                               
    snapshots:                                                                    
      enabled: True                                                               
      timestamp: True                                                             
      bounding_box: True                                                          
      retain:                                                                     
        default: 5                                                                
    objects:                                                                      
      track:                                                                      
        - person                                                                  
        - cup                                                                     
        - cat                                                                     
#    rtmp:                                                                        
#      enabled: True                                                              

detectors:                                                                        
  coral:                                                                          
    type: edgetpu                                                                 
    device: usb
MrSiO commented 3 years ago

Almost there! It's working from my PC now! I'm using Chrome on Linux Mint.

Only issue now is on mobile Android (chrome).

I can see the "built in" video player has changed a bit, but when I hit play the player jerks a little and won't play.

blakeblackshear commented 3 years ago

Can you download a recorded clip and post here?

MrSiO commented 3 years ago

@blakeblackshear

Will do first thing in the morning. (What is usually used to share large video files here?)

I did download the .mp3 and played it from VLC no problem (on mobile).

VLC File info says the following:

CODEC H624 - MPEG-4 AVC (part 10)
Resolution: 800x600
Frame Rate: 5.000

Thx alot!

blakeblackshear commented 3 years ago

Seems like that should play fine. I think you can drag and drop to upload video files.

MrSiO commented 3 years ago

Here is the latest video.
Working fine on PC but not playing on Android browser... (last issue!)

https://user-images.githubusercontent.com/49870576/119657811-604c0300-bdfa-11eb-98e9-d11934dae289.mp4

Thx!

blakeblackshear commented 3 years ago

Try this:

cameras:                                                                          
  ESP32-CAM-02:                                                                   
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - +genpts
        - -r
        - '5'
        - -use_wallclock_as_timestamps
        - '1'                                                                       
      inputs:                                                                     
        - path: rtsp://10.10.10.38:8554/mjpeg/1                                  
          roles:                                                                  
            - detect                                                              
            - rtmp                                                                
            - clips                                                               

      output_args:                                                                
        rtmp: -c:a copy -c:v libx264 -f flv                                       
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -pix_fmt yuv420p -an   

    width: 800                                                                    
    height: 600                                                                   
    fps: 5                                                                        
    best_image_timeout: 15                                                        
    detect:                                                                       
      enabled: True                                                               
      max_disappeared: 15                                                         
    clips:                                                                        
      enabled: True                                                               
    snapshots:                                                                    
      enabled: True                                                               
      timestamp: True                                                             
      bounding_box: True                                                          
      retain:                                                                     
        default: 5                                                                
    objects:                                                                      
      track:                                                                      
        - person                                                                  
        - cup                                                                     
        - cat                                                                     
#    rtmp:                                                                        
#      enabled: True                                                              

detectors:                                                                        
  coral:                                                                          
    type: edgetpu                                                                 
    device: usb
MrSiO commented 3 years ago

It works on mobile! Great work thx!

Can you add this to your offical documentation? Pretty useful information.

I did get "no clips available" on my first 2 attempts but it's probably related to my ESP32-CAM bad frame rate?

frigate.events                 WARNING : Unable to create clip for ESP32-CAM-02 and event 1622290980.461175-ygy5sd. There were no cache files for this event.

I just go a new 1080p cam so I will do the same test on it and get back to you.

Thx!

blakeblackshear commented 3 years ago

If ffmpeg can't consistently maintain a connection to the esp32 cam, then this will happen sometimes. I can add it to the docs.

MrSiO commented 3 years ago

Working perfectly on esp32-cam and Amcrest cam. Thx for your help, case closed.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tve commented 3 years ago

I'm also seeing the "no clip" error for esp32 mjpeg cameras:

frigate.events                 WARNING : Unable to create clip for birdbath and event 1626231187.443
023-h51oih. There were no cache files for this event.

One camera is on a bird bath, so it gets regular detections. I scanned some 20 and a bit more than half are OK but the rest have no clip. The config I have is:

  birdbath:
    ffmpeg:
      inputs:
        - path: http://esp32-cam6.voneicken.com:81
          roles:
            - detect
            - rtmp
            - clips
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -use_wallclock_as_timestamps
        - "1"
        - -r
        - "4"
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v
 h264_vaapi -vf format=nv12,hwupload -r 4 -an
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v
h264_vaapi -vf format=nv12,hwupload -r 4 -an
        rtmp: -c:v h264_vaapi -vf format=nv12,hwupload -r 4 -an -f flv
    width: 1024
    height: 768
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 5
      retain:
        default: 7
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
    objects:
      track: [ 'bird', 'cat', 'dog', 'sheep' ]

How can I troubleshoot what causes the issue?

blakeblackshear commented 3 years ago

@tve I would create a new issue. There should be some related messages in the logs. This would either happen because you are running out of cache space for clips or because the ffmpeg process is being restarted and causing invalid clip cache segments.

tve commented 3 years ago

Hmm, cache space... Thanks for the two suggestions, I'll check on those and open an issue if I don't get anywhere.