[Support]: Max cpu, memory, swap in beta 7

nabahr commented 1 year ago

Describe the problem you are having

I run frigate using the docker image under a debian lxc in proxmox. I upgraded to beta 7 and after a few hours I noticed frigate was not working. Checked the proxmox and it showed the lxc had maxed out the cpu, memory, and swap. I was completely unable to log in to see what was going on and had to reboot the entire proxmox node just to get it working again. This happened a few times before I switched back to the beta 6 docker image and everything has been solid now for over a day now.

Version

0.12.0-beta7

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: pci
ffmpeg:
  hwaccel_args: preset-vaapi
mqtt:
  host: 192.168.0.235
  user: ****
  password: ****
birdseye:
  enabled: False
  width: 1920
  height: 1080
  quality: 12
  mode: motion
detect:
  width: 1920
  height: 1080
  fps: 5
objects:
  track:
    - person
record:
  enabled: True
  retain:
    days: 2
    mode: all
  events:
    retain:
      default: 10
      mode: active_objects
snapshots:
  enabled: True
  crop: False
  bounding_box: True
  timestamp: False
  height: 1080
  retain:
    default: 10
go2rtc:
  streams:
    malcolms_bedroom: ffmpeg:rtsp://192.168.0.81:554/live0#video=copy#audio=aac#audio=opus
    front_door: ffmpeg:rtsp://****:****@192.168.0.98:554/cam/realmonitor?channel=1&subtype=0#video=copy#audio=aac#audio=opus
    main_floor: ffmpeg:rtsp://****:****@192.168.0.38/live#video=copy#audio=aac#audio=opus
    backyard_sw: ffmpeg:rtsp://****:****@192.168.0.108:554/cam/realmonitor?channel=1&subtype=0#video=copy#audio=aac#audio=opus
    family_room: ffmpeg:rtsp://****:****@192.168.0.30:8554/unicast#video=copy#audio=aac#audio=opus
    driveway: ffmpeg:rtsp://192.168.0.31/live0#video=copy#audio=aac#audio=opus
    garage: ffmpeg:rtsp://****:****@192.168.0.55/live#video=copy#audio=aac#audio=opus
  webrtc:
    candidates:
      - 192.168.0.16:8555
      - stun:8555
cameras:
  malcolms_bedroom:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/malcolms_bedroom?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    zones:
      malcolms_bed:
        coordinates: 1850,490,1437,518,1260,50,1578,80
      malcolms_room:
        coordinates: 668,208,1242,202,1393,629,1818,609,1920,1080,178,1080,219,943,480,927
    detect:
      stationary:
        max_frames:
          default: 10
    objects:
      filters:
        person:
          min_score: 0.65
          threshold: 0.75
    motion:
      improve_contrast: True
      threshold: 30
      contour_area: 100
      delta_alpha: 0.15
      frame_alpha: 0.2
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_door?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    zones:
      porch:
        coordinates: 0,1080,1127,1080,1028,853,624,841,0,985
    objects:
      track:
        - person
        - dog
        - cat
      filters:
        person:
          min_area: 2500
          min_score: 0.6
          threshold: 0.75
  main_floor:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/main_floor?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    zones:
      dining_table:
        coordinates: 1486,872,1302,1051,830,1047,741,765,819,435,951,390,1203,389,1328,450
      kitchen:
        coordinates: 1504,390,1536,161,1479,126,1314,90,1085,68,1081,149,946,150,950,368
      piano:
        coordinates: 645,400,671,525,569,697,451,691,406,484,553,326
    objects:
      filters:
        person:
          min_score: 0.65
          threshold: 0.76
  backyard_sw:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/backyard_sw?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    zones:
      backyard:
        coordinates: 408,350,885,326,1035,363,1048,288,1632,523,1559,695,1745,845,1884,555,1488,223,1292,151,973,85,514,79,0,132,43,474,403,408
      deck:
        coordinates: 0,1080,0,488,893,397,1048,415,1899,1080
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.65
          threshold: 0.75
    motion:
      mask:
        - 1920,443,1920,0,1073,0
      improve_contrast: True
      threshold: 30
      contour_area: 110
      delta_alpha: 0.15
      frame_alpha: 0.2
  family_room:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/family_room?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    objects:
      filters:
        person:
          min_score: 0.6
          threshold: 0.8
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/driveway?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    objects:
      track:
        - person
        - dog
        - cat
        - bicycle
        - car
        - motorcycle
    zones:
      culdesac:
        coordinates: 653,92,420,135,251,191,88,275,33,337,116,359,405,297,813,218,963,177,1009,145,968,119,894,94,764,70
      mailbox:
        coordinates: 926,206,902,116,788,127,805,228
      parking:
        coordinates: 1386,403,0,823,0,1080,1573,1080,1867,592
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    objects:
      track:
        - person
        - dog
        - cat
        - bicycle
        - car
        - motorcycle
      filters:
        car:
          min_score: 0.6
          threshold: 0.75
        motorcycle:
          min_score: 0.6
          threshold: 0.75
        person:
          min_score: 0.6
          threshold: 0.8
          min_area: 3000

Relevant log output

na

FFprobe output from your camera

na

Frigate stats

No response

Operating system

Proxmox

Install method

Docker Compose

Coral version

M.2

Network connection

Mixed

Camera make and model

na

Any other information that may be helpful

I can get more debug info later if needed. Been a crazy busy work week.

NickM-27 commented 1 year ago

Without a config there's nothing to go off of. Beta 6 was the same as beta 7 besides go2rtc version bump and was only the latest for half a day.

nabahr commented 1 year ago

I should have some time this weekend to gather some more details. All I know is beta6 is still working fine but beta7 had this problem repeatedly.

skupi20 commented 1 year ago

i have the same problem, today i have freeze my whole server i cant do nothing even locally. Only reboot help. I saw after start frigate docker that after some time i get a lot of ffmpeg processes.

NickM-27 commented 1 year ago

i have the same problem, today i have freeze my whole server i cant do nothing even locally. Only reboot help. I saw after start frigate docker that after some time i get a lot of ffmpeg processes.

many ffmpeg processes are expected, that is used all over this project and go2rtc. More info is needed otherwise there is nothing that can be done.

skupi20 commented 1 year ago

i will try to find some logs, but those ffmpeg processes eat my 24GB ram, the strange is that happen today even i update images yesterday. After reboot its happen very fast. Maybe its some memory leak.

NickM-27 commented 1 year ago

i will try to find some logs, but those ffmpeg processes eat my 24GB ram, the strange is that happen today even i update images yesterday. After reboot its happen very fast. Maybe its some memory leak.

would need to know which ffmpeg processes are using how much RAM. How many cameras do you have?

skupi20 commented 1 year ago

I have 3 cameras, 2 are problematic cheap wifi camera, loose connection sometimes. But its not happen before. Before it eat all memory i have some errors like:

ffmpeg.garage.detect ERROR : [rtsp @ 0x55d02e174100] RTP: PT=60: bad cseq d9ea expected=8859 ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] Packet duration: 4182805182 / dts: 4183165178 is out of range ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] pts has no value ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] Packet duration: 4182808181 / dts: 4183168178 is out of range ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] pts has no value ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] Packet duration: 4182811180 / dts: 4183171178 is out of range ffmpeg.garage.detect ERROR : [mp4 @ 0x55d02f649440] pts has no value ffmpeg.garage.detect ERROR : [out_1_0 @ 0x55d02ee4b900] 100 buffers queued in out_1_0, something may be wrong. ffmpeg.garage.detect ERROR : [out_1_0 @ 0x55d02ee4b900] 1000 buffers queued in out_1_0, something may be wrong.

edit: I see some commits for those type of cameras on go2rtc github :) maybe its that problem

nabahr commented 1 year ago

I should have mentioned this before too. I do have an m.2 coral tpu and Intel hw decoding enabled. I normally sit at around 20-30% cpu utilization with 7 cameras. Memory use is normally around 1.5-2G. When things locked up previously, I could see in proxmox that the container was using about 7 of the 8 Gb available to it, and was using about 90% cpu and the disk i/o was extremely high, it was reporting basically no network traffic too.

My server locked up again last night after I switched back to beta7. I tried to capture some relevant data but failed to do so. Investigating the all logs afterwords, everything looked normal up until the server locked up, then nothing until I had to hard power cycle this morning. I will keep trying to gather data.

NickM-27 commented 1 year ago

@nabahr please update your post to include your config. and logs Otherwise there is no way to help.

Carve commented 1 year ago

I also had this issue, swap and memory (eventually) were both being used to 100% capacity (8gb swap and 64gb total).

Only thing that helped was downgrading back to beta 6. As @nabahr mentioned I believe this has to do with the beta7 update.

NickM-27 commented 1 year ago

I also had this issue, swap and memory (eventually) were both being used to 100% capacity (8gb swap and 64gb total).

Only thing that helped was downgrading back to beta 6. As @nabahr mentioned I believe this has to do with the beta7 update.

as has been said multiple times above, more info is needed otherwise there is nothing we can do. The only change between beta6 and beta7 was an update to go2rtc which fixed one quick issue that go2rtc had. beta6 was only out for a few hours before beta7 was released

nabahr commented 1 year ago

I've been trying to recreate the crash again to capture some relevant logs, etc. but unfortunately (or fortunately) it's been stable ever since I had to hard-reboot my server a couple of days ago.

Carve commented 1 year ago

I also had this issue, swap and memory (eventually) were both being used to 100% capacity (8gb swap and 64gb total). Only thing that helped was downgrading back to beta 6. As @nabahr mentioned I believe this has to do with the beta7 update.

as has been said multiple times above, more info is needed otherwise there is nothing we can do. The only change between beta6 and beta7 was an update to go2rtc which fixed one quick issue that go2rtc had. beta6 was only out for a few hours before beta7 was released

Yes, of course, I forgot to attach my config, please see below. Also want to mention that it's hard to get logs because of the server locking up and needing a reboot.

My latest config:

mqtt:
  host: 192.168.1.69

detectors:
  coral:
    type: edgetpu
    device: pci:0

birdseye:
  enabled: True
  mode: continuous

ffmpeg:
  hwaccel_args: preset-nvidia-h264
  output_args:
    record: preset-record-generic-audio-aac

objects:
  track:
    - package
    - person
    - bicycle
    - dog
    - cat

record:
  enabled: true
  expire_interval: 180
  retain:
    days: 4
    mode: motion
  events:
    pre_capture: 15
    post_capture: 15
    objects:
      - package
      - person
      - bicycle
      - dog
      - cat
    required_zones: []
    retain:
      default: 4
      mode: all
      objects:
        person: 10
        package: 10

snapshots:
  enabled: false
  clean_copy: false
  timestamp: true
  bounding_box: false
  crop: false
  retain:
    default: 10
    objects:
      person: 45
      package: 45

go2rtc:
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    # doorbell_sub:
    #   - rtsp://admin:UNOHASMYPW@192.168.2.200:554/cam/realmonitor?channel=1&subtype=1&authbasic=64
    #   - "ffmpeg:doorbell_sub#audio=opus"
    ashers_room:
      - rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast
      - "ffmpeg:ashers_room#audio=opus"
  webrtc:
    candidates:
      - 192.168.1.9:8555
      - 192.168.1.10:8555
      - 192.168.1.67:8555
      - 192.168.1.68:8555
      - 192.168.1.69:8555
      - stun:8555

cameras:
  doorbell:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/doorbell?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
        # - path: rtsp://127.0.0.1:8554/doorbell_sub?video=copy&audio=aac
        #   input_args: preset-rtsp-restream
        #   roles:
        #     - detect
    detect:
      enabled: true
    mqtt:
      timestamp: true
      bounding_box: false
      crop: true
      height: 500
    record:
      enabled: true
      retain:
        days: 4
        mode: motion
      events:
        retain:
          default: 10
    snapshots:
      enabled: true
  ashers_room:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/ashers_room?video=copy&audio=aac
          input_args: preset-rtsp-restream
    record:
      enabled: false
    snapshots:
      enabled: false
    detect:
      enabled: false

NickM-27 commented 1 year ago

config looks alright, what kind of doorbell do you have? you may need to add #backchannel=0 to the end of its url to ensure that it isn't trying to use rtsp backchannel which can cause some issues

NickM-27 commented 1 year ago

the most important info is going to be which processes are actually using this memory, you should find this by running top or htop inside the container

Carve commented 1 year ago

config looks alright, what kind of doorbell do you have? you may need to add #backchannel=0 to the end of its url to ensure that it isn't trying to use rtsp backchannel which can cause some issues

Amcrest AD410

I swapped back to Beta 6 and I no longer have an issue with memory leaking. htop shows the usual culprit of memory usage being: ffmpeg with a dozen or more processes typically when running normal, when using Beta 7 it would recreate the ffmpeg process multiple times until the memory was gone entirely.

NickM-27 commented 1 year ago

config looks alright, what kind of doorbell do you have? you may need to add #backchannel=0 to the end of its url to ensure that it isn't trying to use rtsp backchannel which can cause some issues

Amcrest AD410

then yes, your problem was most likely the backchannel. Adding #backchannel=0 to the end of the URL would probably fix it

I swapped back to Beta 6 and I no longer have an issue with memory leaking. htop shows the usual culprit of memory usage being: ffmpeg with a dozen or more processes typically when running normal, when using Beta 7 it would recreate the ffmpeg process multiple times until the memory was gone entirely.

okay, you'll need to use htop then to see the name of the ffmpeg process so we can see if it is a frigate ffmpeg process or a go2rtc frigate process.

nabahr commented 1 year ago

then yes, your problem was most likely the backchannel. Adding #backchannel=0 to the end of the URL would probably fix it

Does this apply to the AD110? This is my front door camera.

NickM-27 commented 1 year ago

It might, I'm not sure, wouldn't hurt to put it to be sure.

Carve commented 1 year ago

It might, I'm not sure, wouldn't hurt to put it to be sure.

It did help, however I still am running into the problem;

NickM-27 commented 1 year ago

What do the go2rtc logs show? What about the systems page?

Carve commented 1 year ago

What do the go2rtc logs show? What about the systems page?

2023-02-06 11:44:42.955454942  11:44:42.955 INF go2rtc version 1.1.1 linux/amd64
2023-02-06 11:44:42.955544358  11:44:42.955 INF [api] listen addr=:1984
2023-02-06 11:44:42.955649873  11:44:42.955 INF [rtsp] listen addr=:8554
2023-02-06 11:44:42.955749561  11:44:42.955 INF [srtp] listen addr=:8443
2023-02-06 11:44:42.955809214  11:44:42.955 INF [webrtc] listen addr=:8555
2023-02-06 11:45:17.263818445  11:45:17.263 ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=opus
2023-02-06 11:56:47.457021571  11:56:47.456 WRN github.com/AlexxIT/go2rtc/cmd/streams/producer.go:132 > error="RTSP wrong input" url=rtsp://admin:MYpassword@192.168.2.201:8554/unicast
2023-02-06 11:57:34.530863516  11:57:34.530 ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=opus
2023-02-06 12:00:33.638867557  12:00:33.638 WRN github.com/AlexxIT/go2rtc/cmd/streams/producer.go:132 > error="RTSP wrong input" url=rtsp://admin:MYpassword@192.168.2.201:8554/unicast
2023-02-06 12:02:09.512626056  12:02:09.512 ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=opus
2023-02-06 12:06:09.263417418  12:06:09.263 WRN github.com/AlexxIT/go2rtc/cmd/streams/producer.go:132 > error="RTSP wrong input" url=rtsp://admin:MYpassword@192.168.2.201:8554/unicast

Systems Page:

NickM-27 commented 1 year ago

Looks like the camera is sending bad data causing the stream to restart causing confusion.

For the ashers_room camera you may have better success with

inputs:
        - path: rtsp://127.0.0.1:8554/ashers_room?video
          input_args: preset-rtsp-restream

Since you're not using recordings with it.

skupi20 commented 1 year ago

I think too its problem of camera (my are yi outdoor), maybe some connection problem etc. The frigate and go2rtc are confused as @NickM-27 said. But i dont have such problems last 6 months even with dev or latest beta until some commits were added. Maybe in frigate or go2rtc. I also see ram eat frigate ffmpeg processes. When i check all my logs i saw that many time with that called "confusion" happen on those cameras with connection etc.

It happen even on save & restart from web gui, suddenly frigate eat all my ram. At some log i saw also fps limit warning and i think that after that frigate eat my all ram too. Maybe its problem of killing ffmpeg process, maybe there is some problem.

I put memory limit in my docker compose i hope it help me to achieve more logs before my server hang. At least dont eat all my ram :)

As i said before those 2 cameras i have with wifi make some connection problem or unstable rtsp on that camera, but ffmpeg process restart without a problem. First on frigate directly and also with restream from go2rtc unitil last week.

NickM-27 commented 1 year ago

There's also no requirement to use go2rtc restream as the source for frigate cameras. It is easily possible to keep frigate consuming directly from the camera if your cameras have issues with go2rtc

skupi20 commented 1 year ago

@NickM-27 I use go2rtc from time you changed restream behavior, and have no problem till now :) And i must said clips recorded on restrem are much better i have no idea why :D but for that yi outdoor cameras on direct ffmpeg i have some micro freeze. The problem is that on some situation frigate can hold whole server. I hope that limit memory on docker will help, also i will limit cpu.

Carve commented 1 year ago

ashers_room?video

I like to have audio so I can listen in on my baby boy if needed. Should I change from ?video=copy&audio=aac to ?video&audio=aac? or just leave it as-is.

If it helps this is a Wyze 3 cam.

NickM-27 commented 1 year ago

ashers_room?video

I like to have audio so I can listen in on my baby boy if needed. Should I change from ?video=copy&audio=aac to ?video&audio=aac? or just leave it as-is.

If it helps this is a Wyze 3 cam.

I think you are misunderstanding. Changing the part I mentioned has no affect on the live view having audio or not.

Carve commented 1 year ago

ashers_room?video

I like to have audio so I can listen in on my baby boy if needed. Should I change from ?video=copy&audio=aac to ?video&audio=aac? or just leave it as-is. If it helps this is a Wyze 3 cam.

I think you are misunderstanding. Changing the part I mentioned has no affect on the live view having audio or not.

Regardless if I use ?video or what I had previously, it unfortuantely does not solve the memory issue.

NickM-27 commented 1 year ago

ashers_room?video

I like to have audio so I can listen in on my baby boy if needed. Should I change from ?video=copy&audio=aac to ?video&audio=aac? or just leave it as-is. If it helps this is a Wyze 3 cam.

I think you are misunderstanding. Changing the part I mentioned has no affect on the live view having audio or not.

It in-fact does cause the loss of audio on the stream.

where are you viewing this stream? If you are changing the part I mentioned all that does is change what the recordings and object detection processes receive which is unrelated to the live stream, unless you changed a different part I wasn't referring to.

Regardless if I use ?video or what I had previously, it unfortuantely does not solve the memory issue.

Maybe try using ffmpeg for the go2rtc stream

go2rtc:
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=opus"

Carve commented 1 year ago

@NickM-27 thanks, I've enabled it and now get the following in the go2rtc log:

ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=opus

NickM-27 commented 1 year ago

can you enable trace logs for go2rtc exec ? That seems odd

Carve commented 1 year ago

can you enable trace logs for go2rtc exec ? That seems odd

Yes, how would I go about that?

NickM-27 commented 1 year ago

go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=opus"

Carve commented 1 year ago

go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=opus"

2023-02-06 14:02:14.773222627  14:02:14.773 DBG [exec] run url="exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_transport tcp -i rtsp://admin:MyPassword@192.168.2.201:8554/unicast -c copy -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://localhost:8554/9dfa86075241991edf47b7ec0195e5cd"
2023-02-06 14:02:16.319817917  Input #0, rtsp, from 'rtsp://admin:MyPassword@192.168.2.201:8554/unicast':
2023-02-06 14:02:16.319829083    Metadata:
2023-02-06 14:02:16.319833277      title           : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:02:16.319863569      comment         : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:02:16.319866991    Duration: N/A, start: 0.000000, bitrate: N/A
2023-02-06 14:02:16.319916075    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 20 tbr, 90k tbn
2023-02-06 14:02:16.319958277    Stream #0:1: Audio: pcm_s16be, 16000 Hz, mono, s16, 256 kb/s
2023-02-06 14:02:16.366237607  Output #0, rtsp, to 'rtsp://localhost:8554/9dfa86075241991edf47b7ec0195e5cd':
2023-02-06 14:02:16.366240537    Metadata:
2023-02-06 14:02:16.366254964      title           : 14:02:16.366 DBG [exec] run launch=1.59296305s
2023-02-06 14:02:16.366258254  LIVE555 Streaming Media v2022.04.26
2023-02-06 14:02:16.366261856      comment         : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:02:16.366274339      encoder         : Lavf59.27.100
2023-02-06 14:02:16.366323123    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 20 tbr, 90k tbn
2023-02-06 14:02:16.366361807    Stream #0:1: Audio: pcm_s16be, 16000 Hz, mono, s16, 256 kb/s
2023-02-06 14:02:16.366374581  Stream mapping:
2023-02-06 14:02:16.366377959    Stream #0:0 -> #0:0 (copy)
2023-02-06 14:02:16.366390224    Stream #0:1 -> #0:1 (copy)
2023-02-06 14:02:16.366393382  Press [q] to stop, [?] for help
2023-02-06 14:02:16.366443974  14:02:16.366 DBG [exec] run url="exec:ffmpeg -hide_banner -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/ashers_room?audio -vn -c:a libopus -ar:a 48000 -ac:a 2 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://localhost:8554/972832f3fd682a431e432601d169bc73"
2023-02-06 14:02:16.372295219  frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:00.00 bitrate=N/A speed=N/A    
Input #0, rtsp, from 'rtsp://localhost:8554/ashers_room?audio':
2023-02-06 14:02:16.372295752    Metadata:
2023-02-06 14:02:16.372298628      title           : go2rtc/1.1.1
2023-02-06 14:02:16.372300862    Duration: N/A, bitrate: N/A
2023-02-06 14:02:16.372398903  Codec AVOption flags () specified for input file #0 (rtsp://localhost:8554/ashers_room?audio) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some decoder which was not actually used for any stream.
2023-02-06 14:02:16.372420966  Output #0, rtsp, to 'rtsp://localhost:8554/972832f3fd682a431e432601d169bc73':
2023-02-06 14:02:16.372421495  Output file #0 does not contain any stream
2023-02-06 14:02:16.372687830  14:02:16.372 ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=opus
2023-02-06 14:03:08.765744664  frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:02.73 bitrate=N/A speed= 5.2x    
frame=   10 fps=9.6 q=-1.0 size=N/A time=00:00:03.24 bitrate=N/A speed=3.12x    
frame=   20 fps= 13 q=-1.0 size=N/A time=00:00:03.75 bitrate=N/A speed=2.43x    
frame=   30 fps= 14 q=-1.0 size=N/A time=00:00:04.26 bitrate=N/A speed=2.06x    
frame=   40 fps= 15 q=-1.0 size=N/A time=00:00:04.77 bitrate=N/A speed=1.84x    
frame=   50 fps= 16 q=-1.0 size=N/A time=00:00:05.29 bitrate=N/A speed=1.71x    
frame=   61 fps= 17 q=-1.0 size=N/A time=00:00:05.79 bitrate=N/A speed=1.61x    
frame=   71 fps= 17 q=-1.0 size=N/A time=00:00:06.31 bitrate=N/A speed=1.54x    
frame=   81 fps= 18 q=-1.0 size=N/A time=00:00:06.82 bitrate=N/A speed=1.47x    
frame=   91 fps= 18 q=-1.0 size=N/A time=00:00:07.33 bitrate=N/A speed=1.43x    
frame=  102 fps= 18 q=-1.0 size=N/A time=00:00:07.85 bitrate=N/A speed=1.39x    
frame=  112 fps= 18 q=-1.0 size=N/A time=00:00:08.36 bitrate=N/A speed=1.36x    
frame=  122 fps= 18 q=-1.0 size=N/A time=00:00:08.87 bitrate=N/A speed=1.33x    
frame=  132 fps= 18 q=-1.0 size=N/A time=00:00:09.38 bitrate=N/A speed=1.31x    
frame=  143 fps= 19 q=-1.0 size=N/A time=00:00:09.89 bitrate=N/A speed=1.29x    
frame=  153 fps= 19 q=-1.0 size=N/A time=00:00:10.41 bitrate=N/A speed=1.27x    
frame=  163 fps= 19 q=-1.0 size=N/A time=00:00:10.92 bitrate=N/A speed=1.25x    
frame=  173 fps= 19 q=-1.0 size=N/A time=00:00:11.43 bitrate=N/A speed=1.24x    
frame=  184 fps= 19 q=-1.0 size=N/A time=00:00:11.94 bitrate=N/A speed=1.23x    
frame=  194 fps= 19 q=-1.0 size=N/A time=00:00:12.45 bitrate=N/A speed=1.22x    
frame=  204 fps= 19 q=-1.0 size=N/A time=00:00:12.97 bitrate=N/A speed=1.21x    
frame=  214 fps= 19 q=-1.0 size=N/A time=00:00:13.48 bitrate=N/A speed= 1.2x    
frame=  225 fps= 19 q=-1.0 size=N/A time=00:00:13.99 bitrate=N/A speed=1.19x    
frame=  235 fps= 19 q=-1.0 size=N/A time=00:00:14.50 bitrate=N/A speed=1.18x    
frame=  245 fps= 19 q=-1.0 size=N/A time=00:00:15.01 bitrate=N/A speed=1.17x    
frame=  255 fps= 19 q=-1.0 size=N/A time=00:00:15.53 bitrate=N/A speed=1.17x    
frame=  265 fps= 19 q=-1.0 size=N/A time=00:00:16.04 bitrate=N/A speed=1.16x    
frame=  276 fps= 19 q=-1.0 size=N/A time=00:00:16.55 bitrate=N/A speed=1.15x    
frame=  286 fps= 19 q=-1.0 size=N/A time=00:00:17.06 bitrate=N/A speed=1.15x    
frame=  296 fps= 19 q=-1.0 size=N/A time=00:00:17.57 bitrate=N/A speed=1.14x    
frame=  307 fps= 19 q=-1.0 size=N/A time=00:00:18.09 bitrate=N/A speed=1.14x    
frame=  317 fps= 19 q=-1.0 size=N/A time=00:00:18.60 bitrate=N/A speed=1.13x    
frame=  327 fps= 19 q=-1.0 size=N/A time=00:00:19.11 bitrate=N/A speed=1.13x    
frame=  337 fps= 19 q=-1.0 size=N/A time=00:00:19.62 bitrate=N/A speed=1.13x    
frame=  348 fps= 19 q=-1.0 size=N/A time=00:00:20.13 bitrate=N/A speed=1.12x    
frame=  358 fps= 19 q=-1.0 size=N/A time=00:00:20.65 bitrate=N/A speed=1.12x    
frame=  368 fps= 19 q=-1.0 size=N/A time=00:00:21.16 bitrate=N/A speed=1.12x    
frame=  380 fps= 19 q=-1.0 size=N/A time=00:00:21.71 bitrate=N/A speed=1.11x    
frame=  391 fps= 20 q=-1.0 size=N/A time=00:00:22.26 bitrate=N/A speed=1.11x    
frame=  401 fps= 19 q=-1.0 size=N/A time=00:00:22.82 bitrate=N/A speed=1.11x    
frame=  412 fps= 19 q=-1.0 size=N/A time=00:00:23.33 bitrate=N/A speed= 1.1x    
frame=  422 fps= 19 q=-1.0 size=N/A time=00:00:23.85 bitrate=N/A speed= 1.1x    
frame=  432 fps= 19 q=-1.0 size=N/A time=00:00:24.36 bitrate=N/A speed= 1.1x    
frame=  441 fps= 19 q=-1.0 size=N/A time=00:00:24.87 bitrate=N/A speed= 1.1x    
frame=  452 fps= 19 q=-1.0 size=N/A time=00:00:25.38 bitrate=N/A speed=1.09x    
frame=  462 fps= 19 q=-1.0 size=N/A time=00:00:25.89 bitrate=N/A speed=1.09x    
frame=  474 fps= 20 q=-1.0 size=N/A time=00:00:26.46 bitrate=N/A speed=1.09x    
frame=  484 fps= 20 q=-1.0 size=N/A time=00:00:26.96 bitrate=N/A speed=1.09x    
frame=  494 fps= 20 q=-1.0 size=N/A time=00:00:27.46 bitrate=N/A speed=1.09x    
frame=  503 fps= 20 q=-1.0 size=N/A time=00:00:27.94 bitrate=N/A speed=1.09x    
frame=  513 fps= 20 q=-1.0 size=N/A time=00:00:28.44 bitrate=N/A speed=1.08x    
frame=  523 fps= 20 q=-1.0 size=N/A time=00:00:28.97 bitrate=N/A speed=1.08x    
frame=  533 fps= 20 q=-1.0 size=N/A time=00:00:29.48 bitrate=N/A speed=1.08x    
frame=  544 fps= 20 q=-1.0 size=N/A time=00:00:29.99 bitrate=N/A speed=1.08x    
frame=  554 fps= 20 q=-1.0 size=N/A time=00:00:30.50 bitrate=N/A speed=1.08x    
frame=  564 fps= 20 q=-1.0 size=N/A time=00:00:31.01 bitrate=N/A speed=1.08x    
frame=  574 fps= 20 q=-1.0 size=N/A time=00:00:31.53 bitrate=N/A speed=1.08x    
frame=  585 fps= 20 q=-1.0 size=N/A time=00:00:32.04 bitrate=N/A speed=1.07x    
frame=  595 fps= 20 q=-1.0 size=N/A time=00:00:32.55 bitrate=N/A speed=1.07x    
frame=  605 fps= 20 q=-1.0 size=N/A time=00:00:33.06 bitrate=N/A speed=1.07x    
frame=  615 fps= 20 q=-1.0 size=N/A time=00:00:33.57 bitrate=N/A speed=1.07x    
frame=  626 fps= 20 q=-1.0 size=N/A time=00:00:34.09 bitrate=N/A speed=1.07x    
frame=  636 fps= 20 q=-1.0 size=N/A time=00:00:34.60 bitrate=N/A speed=1.07x    
frame=  646 fps= 20 q=-1.0 size=N/A time=00:00:35.11 bitrate=N/A speed=1.07x    
frame=  656 fps= 20 q=-1.0 size=N/A time=00:00:35.62 bitrate=N/A speed=1.07x    
frame=  667 fps= 20 q=-1.0 size=N/A time=00:00:36.13 bitrate=N/A speed=1.06x    
frame=  677 fps= 20 q=-1.0 size=N/A time=00:00:36.65 bitrate=N/A speed=1.06x    
frame=  687 fps= 20 q=-1.0 size=N/A time=00:00:37.16 bitrate=N/A speed=1.06x    
frame=  697 fps= 20 q=-1.0 size=N/A time=00:00:37.67 bitrate=N/A speed=1.06x    
frame=  708 fps= 20 q=-1.0 size=N/A time=00:00:38.18 bitrate=N/A speed=1.06x    
frame=  718 fps= 20 q=-1.0 size=N/A time=00:00:38.69 bitrate=N/A speed=1.06x    
frame=  728 fps= 20 q=-1.0 size=N/A time=00:00:39.21 bitrate=N/A speed=1.06x    
frame=  738 fps= 20 q=-1.0 size=N/A time=00:00:39.72 bitrate=N/A speed=1.06x    
frame=  748 fps= 20 q=-1.0 size=N/A time=00:00:40.23 bitrate=N/A speed=1.06x    
frame=  759 fps= 20 q=-1.0 size=N/A time=00:00:40.74 bitrate=N/A speed=1.06x    
frame=  769 fps= 20 q=-1.0 size=N/A time=00:00:41.25 bitrate=N/A speed=1.06x    
frame=  779 fps= 20 q=-1.0 size=N/A time=00:00:41.77 bitrate=N/A speed=1.06x    
frame=  789 fps= 20 q=-1.0 size=N/A time=00:00:42.28 bitrate=N/A speed=1.06x    
frame=  800 fps= 20 q=-1.0 size=N/A time=00:00:42.79 bitrate=N/A speed=1.05x    
frame=  810 fps= 20 q=-1.0 size=N/A time=00:00:43.30 bitrate=N/A speed=1.05x    
frame=  820 fps= 20 q=-1.0 size=N/A time=00:00:43.81 bitrate=N/A speed=1.05x    
frame=  831 fps= 20 q=-1.0 size=N/A time=00:00:44.33 bitrate=N/A speed=1.05x    
frame=  841 fps= 20 q=-1.0 size=N/A time=00:00:44.84 bitrate=N/A speed=1.05x    
frame=  851 fps= 20 q=-1.0 size=N/A time=00:00:45.35 bitrate=N/A speed=1.05x    
frame=  861 fps= 20 q=-1.0 size=N/A time=00:00:45.86 bitrate=N/A speed=1.05x    
frame=  872 fps= 20 q=-1.0 size=N/A time=00:00:46.37 bitrate=N/A speed=1.05x    
frame=  882 fps= 20 q=-1.0 size=N/A time=00:00:46.89 bitrate=N/A speed=1.05x    
frame=  892 fps= 20 q=-1.0 size=N/A time=00:00:47.40 bitrate=N/A speed=1.05x    
frame=  902 fps= 20 q=-1.0 size=N/A time=00:00:47.91 bitrate=N/A speed=1.05x    
frame=  913 fps= 20 q=-1.0 size=N/A time=00:00:48.42 bitrate=N/A speed=1.05x    
frame=  923 fps= 20 q=-1.0 size=N/A time=00:00:48.92 bitrate=N/A speed=1.05x    
frame=  933 fps= 20 q=-1.0 size=N/A time=00:00:49.45 bitrate=N/A speed=1.05x    
frame=  943 fps= 20 q=-1.0 size=N/A time=00:00:49.96 bitrate=N/A speed=1.05x    
frame=  953 fps= 20 q=-1.0 size=N/A time=00:00:50.47 bitrate=N/A speed=1.05x    
frame=  964 fps= 20 q=-1.0 size=N/A time=00:00:50.98 bitrate=N/A speed=1.05x    
frame=  974 fps= 20 q=-1.0 size=N/A time=00:00:51.49 bitrate=N/A speed=1.04x    
frame=  984 fps= 20 q=-1.0 size=N/A time=00:00:52.01 bitrate=N/A speed=1.04x    
frame=  994 fps= 20 q=-1.0 size=N/A time=00:00:52.52 bitrate=N/A speed=1.04x    
frame= 1004 fps= 20 q=-1.0 size=N/A time=00:00:53.03 bitrate=N/A speed=1.04x    
frame= 1015 fps= 20 q=-1.0 size=N/A time=00:00:53.54 bitrate=N/A speed=1.04x    
frame= 1025 fps= 20 q=-1.0 size=N/A time=00:00:54.05 bitrate=N/A speed=1.04x    
frame= 1035
2023-02-06 14:04:01.191007647   fps= 20 q=-1.0 size=N/A time=00:00:54.57 bitrate=N/A speed=1.04x    
frame= 1045 fps= 20 q=-1.0 size=N/A time=00:00:55.08 bitrate=N/A speed=1.04x    
frame= 1056 fps= 20 q=-1.0 size=N/A time=00:00:55.59 bitrate=N/A speed=1.04x    
frame= 1066 fps= 20 q=-1.0 size=N/A time=00:00:56.10 bitrate=N/A speed=1.04x    
frame= 1076 fps= 20 q=-1.0 size=N/A time=00:00:56.61 bitrate=N/A speed=1.04x    
frame= 1086 fps= 20 q=-1.0 size=N/A time=00:00:57.13 bitrate=N/A speed=1.04x    
frame= 1097 fps= 20 q=-1.0 size=N/A time=00:00:57.64 bitrate=N/A speed=1.04x    
frame= 1107 fps= 20 q=-1.0 size=N/A time=00:00:58.15 bitrate=N/A speed=1.04x    
frame= 1117 fps= 20 q=-1.0 size=N/A time=00:00:58.66 bitrate=N/A speed=1.04x    
frame= 1127 fps= 20 q=-1.0 size=N/A time=00:00:59.17 bitrate=N/A speed=1.04x    
frame= 1138 fps= 20 q=-1.0 size=N/A time=00:00:59.69 bitrate=N/A speed=1.04x    
frame= 1148 fps= 20 q=-1.0 size=N/A time=00:01:00.20 bitrate=N/A speed=1.04x    
frame= 1160 fps= 20 q=-1.0 size=N/A time=00:01:00.75 bitrate=N/A speed=1.04x    
frame= 1168 fps= 20 q=-1.0 size=N/A time=00:01:01.21 bitrate=N/A speed=1.04x    
frame= 1179 fps= 20 q=-1.0 size=N/A time=00:01:01.73 bitrate=N/A speed=1.04x    
frame= 1189 fps= 20 q=-1.0 size=N/A time=00:01:02.25 bitrate=N/A speed=1.04x    
frame= 1201 fps= 20 q=-1.0 size=N/A time=00:01:02.80 bitrate=N/A speed=1.04x    
frame= 1211 fps= 20 q=-1.0 size=N/A time=00:01:03.30 bitrate=N/A speed=1.04x    
frame= 1220 fps= 20 q=-1.0 size=N/A time=00:01:03.78 bitrate=N/A speed=1.04x    
frame= 1230 fps= 20 q=-1.0 size=N/A time=00:01:04.29 bitrate=N/A speed=1.04x    
frame= 1240 fps= 20 q=-1.0 size=N/A time=00:01:04.81 bitrate=N/A speed=1.04x    
frame= 1250 fps= 20 q=-1.0 size=N/A time=00:01:05.32 bitrate=N/A speed=1.04x    
frame= 1261 fps= 20 q=-1.0 size=N/A time=00:01:05.83 bitrate=N/A speed=1.03x    
frame= 1271 fps= 20 q=-1.0 size=N/A time=00:01:06.34 bitrate=N/A speed=1.03x    
frame= 1281 fps= 20 q=-1.0 size=N/A time=00:01:06.85 bitrate=N/A speed=1.03x    
frame= 1291 fps= 20 q=-1.0 size=N/A time=00:01:07.37 bitrate=N/A speed=1.03x    
frame= 1302 fps= 20 q=-1.0 size=N/A time=00:01:07.88 bitrate=N/A speed=1.03x    
frame= 1312 fps= 20 q=-1.0 size=N/A time=00:01:08.39 bitrate=N/A speed=1.03x    
frame= 1322 fps= 20 q=-1.0 size=N/A time=00:01:08.90 bitrate=N/A speed=1.03x    
frame= 1332 fps= 20 q=-1.0 size=N/A time=00:01:09.41 bitrate=N/A speed=1.03x    
frame= 1343 fps= 20 q=-1.0 size=N/A time=00:01:09.93 bitrate=N/A speed=1.03x    
frame= 1353 fps= 20 q=-1.0 size=N/A time=00:01:10.44 bitrate=N/A speed=1.03x    
frame= 1363 fps= 20 q=-1.0 size=N/A time=00:01:10.95 bitrate=N/A speed=1.03x    
frame= 1374 fps= 20 q=-1.0 size=N/A time=00:01:11.46 bitrate=N/A speed=1.03x    
frame= 1384 fps= 20 q=-1.0 size=N/A time=00:01:11.97 bitrate=N/A speed=1.03x    
frame= 1394 fps= 20 q=-1.0 size=N/A time=00:01:12.49 bitrate=N/A speed=1.03x    
frame= 1404 fps= 20 q=-1.0 size=N/A time=00:01:13.00 bitrate=N/A speed=1.03x    
frame= 1414 fps= 20 q=-1.0 size=N/A time=00:01:13.51 bitrate=N/A speed=1.03x    
frame= 1425 fps= 20 q=-1.0 size=N/A time=00:01:14.02 bitrate=N/A speed=1.03x    
frame= 1435 fps= 20 q=-1.0 size=N/A time=00:01:14.53 bitrate=N/A speed=1.03x    
frame= 1444 fps= 20 q=-1.0 size=N/A time=00:01:15.05 bitrate=N/A speed=1.03x    
frame= 1454 fps= 20 q=-1.0 size=N/A time=00:01:15.56 bitrate=N/A speed=1.03x    
frame= 1465 fps= 20 q=-1.0 size=N/A time=00:01:16.07 bitrate=N/A speed=1.03x    
frame= 1475 fps= 20 q=-1.0 size=N/A time=00:01:16.58 bitrate=N/A speed=1.03x    
frame= 1485 fps= 20 q=-1.0 size=N/A time=00:01:17.09 bitrate=N/A speed=1.03x    
frame= 1495 fps= 20 q=-1.0 size=N/A time=00:01:17.61 bitrate=N/A speed=1.03x    
frame= 1505 fps= 20 q=-1.0 size=N/A time=00:01:18.12 bitrate=N/A speed=1.03x    
frame= 1516 fps= 20 q=-1.0 size=N/A time=00:01:18.63 bitrate=N/A speed=1.03x    
frame= 1526 fps= 20 q=-1.0 size=N/A time=00:01:19.14 bitrate=N/A speed=1.03x    
frame= 1536 fps= 20 q=-1.0 size=N/A time=00:01:19.65 bitrate=N/A speed=1.03x    
frame= 1547 fps= 20 q=-1.0 size=N/A time=00:01:20.17 bitrate=N/A speed=1.03x    
frame= 1557 fps= 20 q=-1.0 size=N/A time=00:01:20.68 bitrate=N/A speed=1.03x    
frame= 1567 fps= 20 q=-1.0 size=N/A time=00:01:21.19 bitrate=N/A speed=1.03x    
frame= 1577 fps= 20 q=-1.0 size=N/A time=00:01:21.70 bitrate=N/A speed=1.03x    
frame= 1588 fps= 20 q=-1.0 size=N/A time=00:01:22.21 bitrate=N/A speed=1.03x    
frame= 1598 fps= 20 q=-1.0 size=N/A time=00:01:22.73 bitrate=N/A speed=1.03x    
frame= 1608 fps= 20 q=-1.0 size=N/A time=00:01:23.24 bitrate=N/A speed=1.03x    
frame= 1618 fps= 20 q=-1.0 size=N/A time=00:01:23.75 bitrate=N/A speed=1.03x    
frame= 1628 fps= 20 q=-1.0 size=N/A time=00:01:24.26 bitrate=N/A speed=1.03x    
frame= 1639 fps= 20 q=-1.0 size=N/A time=00:01:24.77 bitrate=N/A speed=1.03x    
frame= 1649 fps= 20 q=-1.0 size=N/A time=00:01:25.29 bitrate=N/A speed=1.03x    
frame= 1659 fps= 20 q=-1.0 size=N/A time=00:01:25.80 bitrate=N/A speed=1.03x    
frame= 1671 fps= 20 q=-1.0 size=N/A time=00:01:26.34 bitrate=N/A speed=1.03x    
frame= 1680 fps= 20 q=-1.0 size=N/A time=00:01:26.81 bitrate=N/A speed=1.03x    
frame= 1690 fps= 20 q=-1.0 size=N/A time=00:01:27.33 bitrate=N/A speed=1.03x    
frame= 1700 fps= 20 q=-1.0 size=N/A time=00:01:27.85 bitrate=N/A speed=1.03x    
frame= 1710 fps= 20 q=-1.0 size=N/A time=00:01:28.36 bitrate=N/A speed=1.03x    
frame= 1720 fps= 20 q=-1.0 size=N/A time=00:01:28.87 bitrate=N/A speed=1.03x    
frame= 1731 fps= 20 q=-1.0 size=N/A time=00:01:29.38 bitrate=N/A speed=1.03x    
frame= 1741 fps= 20 q=-1.0 size=N/A time=00:01:29.89 bitrate=N/A speed=1.03x    
frame= 1751 fps= 20 q=-1.0 size=N/A time=00:01:30.41 bitrate=N/A speed=1.02x    
frame= 1762 fps= 20 q=-1.0 size=N/A time=00:01:30.92 bitrate=N/A speed=1.02x    
frame= 1772 fps= 20 q=-1.0 size=N/A time=00:01:31.43 bitrate=N/A speed=1.02x    
frame= 1782 fps= 20 q=-1.0 size=N/A time=00:01:31.94 bitrate=N/A speed=1.02x    
frame= 1792 fps= 20 q=-1.0 size=N/A time=00:01:32.45 bitrate=N/A speed=1.02x    
frame= 1802 fps= 20 q=-1.0 size=N/A time=00:01:32.97 bitrate=N/A speed=1.02x    
frame= 1813 fps= 20 q=-1.0 size=N/A time=00:01:33.48 bitrate=N/A speed=1.02x    
frame= 1823 fps= 20 q=-1.0 size=N/A time=00:01:33.99 bitrate=N/A speed=1.02x    
frame= 1833 fps= 20 q=-1.0 size=N/A time=00:01:34.50 bitrate=N/A speed=1.02x    
frame= 1844 fps= 20 q=-1.0 size=N/A time=00:01:35.01 bitrate=N/A speed=1.02x    
frame= 1854 fps= 20 q=-1.0 size=N/A time=00:01:35.53 bitrate=N/A speed=1.02x    
frame= 1864 fps= 20 q=-1.0 size=N/A time=00:01:36.04 bitrate=N/A speed=1.02x    
frame= 1874 fps= 20 q=-1.0 size=N/A time=00:01:36.55 bitrate=N/A speed=1.02x    
frame= 1884 fps= 20 q=-1.0 size=N/A time=00:01:37.06 bitrate=N/A speed=1.02x    
frame= 1895 fps= 20 q=-1.0 size=N/A time=00:01:37.57 bitrate=N/A speed=1.02x    
frame= 1905 fps= 20 q=-1.0 size=N/A time=00:01:38.09 bitrate=N/A speed=1.02x    
frame= 1915 fps= 20 q=-1.0 size=N/A time=00:01:38.60 bitrate=N/A speed=1.02x    
frame= 1925 fps= 20 q=-1.0 size=N/A time=00:01:39.11 bitrate=N/A speed=1.02x    
frame= 1938 fps= 20 q=-1.0 size=N/A time=00:01:39.69 bitrate=N/A speed=1.02x    
frame= 1948 fps= 20 q=-1.0 size=N/A time=00:01:40.19 bitrate=N/A speed=1.02x    
frame= 1958 fps= 20 q=-1.0 size=N/A time=00:01:40.68 bitrate=N/A speed=1.02x    
frame= 1969 fps= 20 q=-1.0 size=N/A time=00:01:41.24 bitrate=N/A speed=1.02x    
frame= 1979 fps= 20 q=-1.0 size=N/A time=00:01:41.80 bitrate=N/A speed=1.02x    
frame= 1989 fps= 20 q=-1.0 size=N/A time=00:01:42.31 bitrate=N/A speed=1.02x    
frame= 2000 fps= 20 q=-1.0 size=N/A time=00:01:42.79 bitrate=N/A speed=1.02x    
frame= 2010 fps= 20 q=-1.0 size=N/A time=00:01:43.33 bitrate=N/A speed=1.02x    
frame= 2020 fps= 20 q=-1.0 size=N/A time=00:01:43.85 bitrate=N/A speed=1.02x    
frame= 2031 fps= 20 q=-1.0 size=N/A time=00:01:44.36 bitrate=N/A speed=1.02x    
frame= 2041 fps= 20 q=-1.0 size=N/A time=00:01:44.87 bitrate=N/A speed=1.02x    
frame= 2043 fps= 20 q=-1.0 size=N/A time=00:01:45.00 bitrate=N/A speed=1.01x    
frame= 2074 fps= 20 q=-1.0 size=N/A time=00:01:46.49 bitrate=N/A speed=1.02x

Video is working fine, audio however is not

NickM-27 commented 1 year ago

it's because your audio is not AAC, it is in fact PCMU, so your config is not correct. you should have go2rtc configured for that camera as

go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=aac"

Carve commented 1 year ago

it's because your audio is not AAC, it is in fact PCMU, so your config is not correct. you should have go2rtc configured for that camera as
go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://admin:UNOHASMYPW@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://admin:UNOHASMYPW@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=aac"

2023-02-06 14:09:55.231975851  14:09:55.231 INF go2rtc version 1.1.1 linux/amd64
2023-02-06 14:09:55.232074768  14:09:55.232 INF [api] listen addr=:1984
2023-02-06 14:09:55.232177994  14:09:55.232 INF [rtsp] listen addr=:8554
2023-02-06 14:09:55.232289933  14:09:55.232 INF [srtp] listen addr=:8443
2023-02-06 14:09:55.232350435  14:09:55.232 INF [webrtc] listen addr=:8555
2023-02-06 14:09:55.633454253  14:09:55.633 DBG [exec] run url="exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_transport tcp -i rtsp://admin:WEEEEEEEEEEEEEEEEEEE@192.168.2.201:8554/unicast -c copy -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://localhost:8554/9dfa86075241991edf47b7ec0195e5cd"
2023-02-06 14:09:58.193826088  Input #0, rtsp, from 'rtsp://admin:WEEEEEEEEEEEEEEEEEEE@192.168.2.201:8554/unicast':
2023-02-06 14:09:58.193835873    Metadata:
2023-02-06 14:09:58.193840235      title           : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:09:58.193870743      comment         : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:09:58.193873729    Duration: N/A, start: 0.000000, bitrate: N/A
2023-02-06 14:09:58.193914999    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 20 tbr, 90k tbn
2023-02-06 14:09:58.193959398    Stream #0:1: Audio: pcm_s16be, 16000 Hz, mono, s16, 256 kb/s
2023-02-06 14:09:58.238323541  Output #0, rtsp, to 'rtsp://localhost:8554/9dfa86075241991edf47b7ec0195e5cd':
2023-02-06 14:09:58.238327699    Metadata:
2023-02-06 14:09:58.238345911      title           : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:09:58.238349296  14:09:58.238 DBG [exec] run launch=2.604780201s
2023-02-06 14:09:58.238352611      comment         : LIVE555 Streaming Media v2022.04.26
2023-02-06 14:09:58.238355493      encoder         : Lavf59.27.100
2023-02-06 14:09:58.238408146    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 20 tbr, 90k tbn
2023-02-06 14:09:58.238446666    Stream #0:1: Audio: pcm_s16be, 16000 Hz, mono, s16, 256 kb/s
2023-02-06 14:09:58.238459821  Stream mapping:
2023-02-06 14:09:58.238463119    Stream #0:0 -> #0:0 (copy)
2023-02-06 14:09:58.238475766    Stream #0:1 -> #0:1 (copy)
2023-02-06 14:09:58.238479056  Press [q] to stop, [?] for help
2023-02-06 14:09:58.238484446  14:09:58.238 DBG [exec] run url="exec:ffmpeg -hide_banner -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_transport tcp -i rtsp://localhost:8554/ashers_room?audio -vn -c:a aac -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://localhost:8554/49dffcb541d9d19cffc80d3d54b4845c"
2023-02-06 14:09:58.244462473  frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:03.11 bitrate=N/A speed=3.11e+06x    
Input #0, rtsp, from 'rtsp://localhost:8554/ashers_room?audio':
2023-02-06 14:09:58.244462979    Metadata:
2023-02-06 14:09:58.244465496      title           : go2rtc/1.1.1
2023-02-06 14:09:58.244467484    Duration: N/A, bitrate: N/A
2023-02-06 14:09:58.244526607  Codec AVOption flags () specified for input file #0 (rtsp://localhost:8554/ashers_room?audio) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some decoder which was not actually used for any stream.
2023-02-06 14:09:58.244545740  Output #0, rtsp, to 'rtsp://localhost:8554/49dffcb541d9d19cffc80d3d54b4845c':
2023-02-06 14:09:58.244546290  Output file #0 does not contain any stream
2023-02-06 14:09:58.244822800  14:09:58.244 ERR github.com/AlexxIT/go2rtc/cmd/streams/producer.go:52 > error="exec: exit status 1" url=ffmpeg:ashers_room#audio=aac
2023-02-06 14:09:58.736693751  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55203, current: 50820; changing to 55204. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.739676755  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55204, current: 50820; changing to 55205. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.739700090  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55205, current: 50820; changing to 55206. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.868776230  frame=    0 fps=0.0 q=-1.0 size=N/A time=00:00:03.48 bitrate=N/A speed=6.96x    
[rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55206, current: 52901; changing to 55207. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.869015632  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55207, current: 52901; changing to 55208. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.869157493  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55208, current: 52901; changing to 55209. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.990891088  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55209, current: 54982; changing to 55210. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.992788419  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55210, current: 54982; changing to 55211. This may result in incorrect timestamps in the output file.
2023-02-06 14:09:58.992894717  [rtsp @ 0x55bb8ec5e000] Non-monotonous DTS in output stream 0:1; previous: 55211, current: 54982; changing to 55212. This may result in incorrect timestamps in the output file.
2023-02-06 14:10:51.095688278  frame=    9 fps=8.9 q=-1.0 size=N/A time=00:00:03.73 bitrate=N/A speed= 3.7x

NickM-27 commented 1 year ago

you are using ?audio which means only audio is included in the stream which is unusable for frigate's purposes

Carve commented 1 year ago

you are using ?audio which means only audio is included in the stream which is unusable for frigate's purposes

That was because I had:

inputs:
        - path: rtsp://127.0.0.1:8554/ashers_room?video&audio=aac

switching over to:

inputs:
        - path: rtsp://127.0.0.1:8554/ashers_room?video

Removes the error but I don't have audio on the live view via the web UI (which was working before switch the go2rtc to ffmpeg):

NickM-27 commented 1 year ago

you are using ?audio which means only audio is included in the stream which is unusable for frigate's purposes

That was because I had:
inputs:

        - path: rtsp://127.0.0.1:8554/ashers_room?video&audio=aac
switching over to:
inputs:

        - path: rtsp://127.0.0.1:8554/ashers_room?video
Removes the error but I don't have audio on the live view via the web UI (which was working before switch the go2rtc to ffmpeg):

As I said, changing the part under inputs has 0 effect on the live view audio... The only part that affects audio in live view is the go2rtc config.

So, are you trying to view via MSE or WebRTC?

Carve commented 1 year ago

you are using ?audio which means only audio is included in the stream which is unusable for frigate's purposes

That was because I had:
inputs:

        - path: rtsp://127.0.0.1:8554/ashers_room?video&audio=aac
switching over to:
inputs:

        - path: rtsp://127.0.0.1:8554/ashers_room?video
Removes the error but I don't have audio on the live view via the web UI (which was working before switch the go2rtc to ffmpeg):
As I said, changing the part under inputs has 0 effect on the live view audio... The only part that affects audio in live view is the go2rtc config.

So, are you trying to view via MSE or WebRTC?

WebRTC. But the same occurs on MSE. My go2rtc has been unchanged from what you suggested previously;

go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://user:pass@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://user:pass@192.168.2.201:8554/unicast"
      - "ffmpeg:ashers_room#audio=aac"
  webrtc:
    candidates:
      - 192.168.1.9:8555
      - 192.168.1.10:8555
      - 192.168.1.67:8555
      - 192.168.1.68:8555
      - 192.168.1.69:8555
      - stun:8555

NickM-27 commented 1 year ago

Okay, maybe try


go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://user:pass@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://user:pass@192.168.2.201:8554/unicast#video=copy#audio=aac#audio=opus"
  webrtc:
    candidates:
      - 192.168.1.9:8555
      - 192.168.1.10:8555
      - 192.168.1.67:8555
      - 192.168.1.68:8555
      - 192.168.1.69:8555
      - stun:8555

Carve commented 1 year ago

Okay, maybe try

go2rtc:
  log:
    exec: trace
  streams:
    doorbell:
      - rtsp://user:pass@192.168.2.200:554
      - "ffmpeg:doorbell#audio=opus"
    ashers_room:
      - "ffmpeg:rtsp://user:pass@192.168.2.201:8554/unicast#video=copy#audio=aac#audio=opus"
  webrtc:
    candidates:
      - 192.168.1.9:8555
      - 192.168.1.10:8555
      - 192.168.1.67:8555
      - 192.168.1.68:8555
      - 192.168.1.69:8555
      - stun:8555

Thanks, that worked, audio is now back on the live stream side on the web ui. Ram usage has also stayed stable (so far) and I will keep an eye on it and keep you updated.

skupi20 commented 1 year ago

I limit memory for frigate docker and at least my server not hang with out of memory.

In system logs and frigate i saw that the out of memory is happen when i get a lot of lines "Unable to read frames from ffmpeg process." i put only two below, and later exiting capture thread:

[2023-02-07 07:00:31] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2023-02-07 07:00:31] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2023-02-07 07:00:31] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread...

system log:

07:00:31 serwer kernel: 07:00:31 serwer kernel: 07:00:31 serwer kernel: 07:00:31 serwer kernel: 07:00:31 serwer kernel: [ pid ] 07:00:31 serwer kernel: [ 106258] 07:00:31 serwer kernel: [ 106315] 07:00:31 serwer kernel: [ 106317] 07:00:31 serwer kernel: [ 106324] 07:00:31 serwer kernel: [ 106325] 07:00:31 serwer kernel: [ 106326] 07:00:31 serwer kernel: [ 106327] 07:00:31 serwer kernel: [ 106328] 07:00:31 serwer kernel: [ 106329] 07:00:31 serwer kernel: [ 106330] 07:00:31 serwer kernel: [ 106331] 07:00:31 serwer kernel: [ 106340] 07:00:31 serwer kernel: [ 106341] 07:00:31 serwer kernel: 07:00:31 serwer kernel: 07:00:31 serwer kernel: 07:00:31 serwer kernel: [ 106388] 07:00:31 serwer kernel: [ 106393] 07:00:31 serwer kernel: [ 106408] 07:00:31 serwer kernel: [ 106428] 07:00:31 serwer kernel: [ 106429] 07:00:31 serwer kernel: [ 106430] 07:00:31 serwer kernel: [ 106431] 07:00:31 serwer kernel: [ 106434] 07:00:31 serwer kernel: [ 106443] 07:00:31 serwer kernel: [ 106461] 07:00:31 serwer kernel: [ 106488] 07:00:31 serwer kernel: [ 106735] 07:00:31 serwer kernel: [ 106741] 07:00:31 serwer kernel: [ 106742] 07:00:31 serwer kernel: [ 106744] 07:00:31 serwer kernel: [ 106747] 07:00:31 serwer kernel: [ 106748] 07:00:31 serwer kernel: [ 106749] 07:00:31 serwer kernel: [ 106750] 07:00:31 serwer kernel: [ 106751] 07:00:31 serwer kernel: [ 106752] 07:00:31 serwer kernel: [ 106755] 07:00:31 serwer kernel: [ 106761] 07:00:31 serwer kernel: [ 106769] 07:00:31 serwer kernel: [ 106770] 07:00:31 serwer kernel: [ 106773] 07:00:31 serwer kernel: [ 106774] 07:00:31 serwer kernel: [ 483882] 07:00:31 serwer kernel: [ 485141] ffmpeg invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 memory: usage 6291456kB, limit 6291456kB, failcnt 98723 swap: usage 0kB, limit 6291456kB, failcnt 0 Tasks state (memory values in pages): uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name 0 106258 53 17 28672 0 0 s6-svscan 0 106315 53 15 28672 0 0 s6-supervise 0 106317 50 1 24576 0 0 s6-linux-init-s 0 106324 53 16 28672 0 0 s6-supervise 0 106325 53 14 28672 0 0 s6-supervise 0 106326 53 16 28672 0 0 s6-supervise 0 106327 53 15 28672 0 0 s6-supervise 0 106328 53 15 28672 0 0 s6-supervise 0 106329 53 15 28672 0 0 s6-supervise 0 106330 53 16 28672 0 0 s6-supervise 0 106331 53 15 28672 0 0 s6-supervise 0 106340 130 41 28672 0 0 s6-fdholderd 0 106341 47 1 24576 0 0 s6-ipcserverd [ 106378] 65534 106378 69 1 40960 0 0 s6-log [ 106380] 65534 106380 69 1 36864 0 0 s6-log [ 106381] 65534 106381 69 1 36864 0 0 s6-log 0 106388 181799 5001 135168 0 0 go2rtc 0 106393 1324855 116689 2113536 0 0 python3 0 106408 145419 2896 114688 0 0 nginx 0 106428 211189 1514 229376 0 0 nginx 0 106429 211185 1523 229376 0 0 nginx 0 106430 214085 4164 253952 0 0 nginx 0 106431 216315 6554 274432 0 0 nginx 0 106434 211095 1239 225280 0 0 nginx 0 106443 211095 1195 225280 0 0 nginx 0 106461 211119 1461 229376 0 0 nginx 0 106488 211095 1216 225280 0 0 nginx 0 106735 779302 43128 684032 0 0 frigate.logger 0 106741 3851 2654 69632 0 0 python3 0 106742 3266691 270766 4001792 0 0 frigate.detecto 0 106744 884826 45951 802816 0 0 frigate.output 0 106747 943277 50184 880640 0 0 frigate.process 0 106748 31705 4154 167936 0 0 ffmpeg 0 106749 943476 50202 880640 0 0 frigate.process 0 106750 31705 4124 163840 0 0 ffmpeg 0 106751 943575 50470 880640 0 0 frigate.process 0 106752 887826 44944 753664 0 0 frigate.capture 0 106755 884514 45610 774144 0 0 frigate.capture 0 106761 884514 45611 770048 0 0 frigate.capture 0 106769 2504417 45141 700416 0 0 ffmpeg 0 106770 31705 4178 167936 0 0 ffmpeg 0 106773 31705 4163 167936 0 0 ffmpeg 0 106774 32641 5575 176128 0 0 ffmpeg 0 483882 2518137 47134 712704 0 0 ffmpeg 0 485141 3771095 1300023 10752000 0 0 ffmpeg

Of curse all happen with cameras that have problematic connetion or rstp stream, but before watchdog have no problem with restart process without eating whole ram. I saw last night it happen more than 10 times. I switch to direct ffmpeg from frigate and will see what happen.

LostSoulfly commented 1 year ago

I've had this happen to me a few times on beta7 as well. I'll wake up in the morning to none of my cameras working, all 12 cores of my server pegged at 100% and frigate suddenly using 10/10GB of RAM I allocate it.

Config: config.yml.txt

I have since reverted to beta6 and will see if it happens again.

skupi20 commented 1 year ago

Update: On direct ffmpeg from frigate, problematic cameras restart stream without such issue, last two days. I wait till guys update go2rtc, as it was updated today to 1.1.2. But still i think its not normal that watchdog or any other part of frigate kill, exit, close process with memory leak. Especially that before beta 7 it work correctly, some commits near beta 7 start that problem.

I never use memory limit on my dockers, for now i will start do it :) Cause whole server and all services hang. I have oom killer but i saw that memory eating happen so fast that oom killer didn't make it. I saw it worked two times when i were checking htop. But most time it kill, hang whole server. Only limiting memory in docker help to restart process

LostSoulfly commented 1 year ago

Following up, beta6 exhibited the same thing but beta5 has been solid for me for about a week.

nabahr commented 1 year ago

I've finally encountered the problem again tonight. I have changed a few things since I originally encountered this issue though. I am currently running beta8, 0.12.0-27A31E7, and I switched over my wyze cameras (2 v2 and 1 v3) off of the rtsp firmware, updated them all, and added the docker-wyze-bridge in a separate proxmox container to get an rtsp feed from them into frigate.

The issue started around roughly 19:15-19:20. I had a script running to log the output of top every 5 seconds, right at the time issues started I see one ffmpeg process start eating up memory. There is a 4 minute gap from 19:16:59 to 19:20:54 when the last log gets saved then nothing. CPU log at the end of this post.

In the frigate container log, I see this

{"log":"s6-supervise go2rtc: warning: finish script lifetime reached maximum value - sending it a SIGKILL\n","stream":"stderr","time":"2023-02-18T01:17:40.028441362Z"}
{"log":"2023-02-17 19:17:30.184485213  fatal error: concurrent map iteration and map write\n","stream":"stdout","time":"2023-02-18T01:17:41.276569074Z"}

Followed by what looks like a trace from go2rtc, it is pretty long and a little garbled since it's stored in json from the docker container so I won't add it here but I can upload it somewhere to share if you want. I have no recordings after this time and this is when frigate sensors first went unavailable in homeassistant too. Things kind of continued to run for a little bit though, at least there are more logs and homeassistant would occasionally get sensors back for a short bit here and there, up until about 19:47 when everything went unavailable and the lost log line was captured, then nothing until I force stopped the lxc container in proxmox and restarted.

Logs from top, I only include processes that are using any CPU.

top - 19:16:38 up 11 days,  9:19,  0 users,  load average: 1.00, 1.00, 1.04
Tasks:  81 total,   4 running,  77 sleeping,   0 stopped,   0 zombie
%Cpu(s): 36.1 us, 31.1 sy,  0.0 ni, 26.2 id,  6.6 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   8192.0 total,    823.2 free,   2148.1 used,   5220.7 buff/cache
MiB Swap:   1024.0 total,   1004.5 free,     19.5 used.   6043.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627751 root      20   0 1075740 100904   5292 S  60.0   1.2  45:46.10 frigate+
2627764 root      20   0  952524  91060  28252 R  60.0   1.1  58:24.99 ffmpeg
2627530 root      20   0 2842428 486004  43016 S  40.0   5.8 102:38.69 python3
2627742 root      20   0 1083388 141504  14060 R  20.0   1.7  26:56.60 frigate+
2964591 root      20   0  134356  28432  17716 R  20.0   0.3   0:00.03 ffprobe
2627736 root      20   0 1077884 138260  13860 S  13.3   1.6  11:39.28 frigate+
2627716 root      20   0   14392  10448   5172 S   6.7   0.1   8:04.33 python3
2627719 root      20   0 1269480 104976   9264 S   6.7   1.3  43:20.84 frigate+
2627750 root      20   0  951024  89604  27676 S   6.7   1.1  57:36.96 ffmpeg
2627754 root      20   0 1110900 117352   6488 S   6.7   1.4  45:58.76 frigate+

top - 19:16:43 up 11 days,  9:19,  0 users,  load average: 1.08, 1.02, 1.04
Tasks:  80 total,   3 running,  77 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.2 us, 27.4 sy,  0.0 ni, 29.0 id, 19.4 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   8192.0 total,    821.9 free,   3389.6 used,   3980.5 buff/cache
MiB Swap:   1024.0 total,   1004.5 free,     19.5 used.   4802.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627764 root      20   0 2261464   1.3g  28252 R  66.7  16.7  58:28.30 ffmpeg
2627530 root      20   0 2842428 486000  43016 S  20.0   5.8 102:39.06 python3
2964602 root      20   0  134348  27820  17140 R  20.0   0.3   0:00.03 ffprobe
2627719 root      20   0 1269480 104976   9264 S   6.7   1.3  43:21.08 frigate+
2627738 root      20   0 1078884 140084  13864 S   6.7   1.7  39:08.42 frigate+
2627746 root      20   0 1075740 100572   4968 S   6.7   1.2  45:41.53 frigate+
2627767 root      20   0 1045364 108312   6744 S   6.7   1.3  45:24.72 frigate+
2751976 root      20   0  958592  99284  28052 S   6.7   1.2  34:00.11 ffmpeg
2821611 root      20   0  953472 103360  27924 S   6.7   1.2  24:06.80 ffmpeg
2929632 root      20   0  951380  93216  27808 S   6.7   1.1   6:12.89 ffmpeg
2964600 root      20   0    7052   3112   2684 R   6.7   0.0   0:00.01 top

top - 19:16:48 up 11 days,  9:19,  0 users,  load average: 1.16, 1.04, 1.05
Tasks:  81 total,   2 running,  79 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.6 us, 29.2 sy,  0.0 ni, 32.3 id, 12.3 wa,  0.0 hi,  1.5 si,  0.0 st
MiB Mem :   8192.0 total,    819.4 free,   5063.2 used,   2309.4 buff/cache
MiB Swap:   1024.0 total,   1004.5 free,     19.5 used.   3128.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627764 root      20   0 4029792   3.0g  28252 S  66.7  37.8  58:31.77 ffmpeg
2627530 root      20   0 2842428 485092  42104 S  20.0   5.8 102:39.48 python3
2627741 root      20   0 1078760 139312  13864 S   6.7   1.7  16:59.38 frigate+
2627746 root      20   0 1075740 100572   4968 S   6.7   1.2  45:41.72 frigate+
2627750 root      20   0  951024  89604  27676 S   6.7   1.1  57:37.36 ffmpeg
2627767 root      20   0 1045364 108312   6744 S   6.7   1.3  45:24.90 frigate+
2751976 root      20   0  958592  99284  28052 S   6.7   1.2  34:00.26 ffmpeg
2821570 root      20   0  952620 103888  27976 S   6.7   1.2  24:31.43 ffmpeg
2821611 root      20   0  953472 103360  27924 S   6.7   1.2  24:07.00 ffmpeg

top - 19:16:53 up 11 days,  9:19,  0 users,  load average: 1.31, 1.07, 1.06
Tasks:  80 total,   2 running,  78 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.6 us, 33.3 sy,  0.0 ni, 25.4 id, 20.6 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   8192.0 total,    819.2 free,   6538.2 used,    834.6 buff/cache
MiB Swap:   1024.0 total,    955.0 free,     69.0 used.   1653.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627764 root      20   0 5626428   4.5g  16012 R  68.8  56.6  58:35.41 ffmpeg
2627525 root      20   0  867256  36344      4 S  12.5   0.4  75:15.69 go2rtc
2627530 root      20   0 2842428 483496  40572 S  12.5   5.8 102:39.77 python3
2627742 root      20   0 1080348 109864  13276 S  12.5   1.3  26:58.46 frigate+
2627775 root      20   0 1045364  71980   6136 S  12.5   0.9  45:52.64 frigate+
2627735 root      20   0 1078284 107652  13328 S   6.2   1.3  18:06.95 frigate+
2627746 root      20   0 1075740  68940   4460 S   6.2   0.8  45:41.93 frigate+
2627750 root      20   0  951024  78196  16004 S   6.2   0.9  57:37.56 ffmpeg
2627778 root      20   0 1045364  86488   6140 S   6.2   1.0  24:13.76 frigate+
2821570 root      20   0  952620  92004  16092 S   6.2   1.1  24:31.65 ffmpeg
2929632 root      20   0  951380  81568  15896 S   6.2   1.0   6:13.39 ffmpeg

top - 19:16:59 up 11 days,  9:19,  0 users,  load average: 2.32, 1.28, 1.13
Tasks:  80 total,   4 running,  76 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.9 us, 23.9 sy,  0.0 ni, 23.9 id, 32.8 wa,  0.0 hi,  1.5 si,  0.0 st
MiB Mem :   8192.0 total,    819.1 free,   6740.4 used,    632.5 buff/cache
MiB Swap:   1024.0 total,      8.7 free,   1015.3 used.   1451.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627764 root      20   0 6789644   5.6g   5376 R  43.8  69.7  58:38.30 ffmpeg
2627525 root      20   0  867256  16108      4 S  18.8   0.2  75:15.93 go2rtc
2964639 root      20   0  126672  13700  10732 D  18.8   0.2   0:00.03 ffmpeg
2627530 root      20   0 2842428 147176  13452 S   6.2   1.8 102:39.93 python3
2627717 root      20   0 1021556  25040   7048 S   6.2   0.3   9:19.30 frigate+
2627735 root      20   0 1078284  20224   8468 S   6.2   0.2  18:07.00 frigate+
2627742 root      20   0 1080348  43504   8832 S   6.2   0.5  26:58.95 frigate+
2627746 root      20   0 1075740  19708   3352 S   6.2   0.2  45:42.17 frigate+
2627750 root      20   0  951024  35588   7264 S   6.2   0.4  57:37.76 ffmpeg
2627767 root      20   0 1045364  20240   3916 S   6.2   0.2  45:25.35 frigate+
2627775 root      20   0 1045364  20456   3972 S   6.2   0.2  45:52.85 frigate+
2627778 root      20   0 1045364  20588   4024 S   6.2   0.2  24:13.99 frigate+
2751976 root      20   0  958592  37236   7144 S   6.2   0.4  34:00.58 ffmpeg
2821611 root      20   0  953472  49384   6968 S   6.2   0.6  24:07.42 ffmpeg
2929632 root      20   0  951380  71524   5852 S   6.2   0.9   6:13.60 ffmpeg
2964637 root      20   0    7052   3200   2772 R   6.2   0.0   0:00.01 top

top - 19:20:54 up 11 days,  9:23,  0 users,  load average: 52.66, 28.66, 12.25
Tasks:  79 total,   5 running,  73 sleeping,   0 stopped,   1 zombie
%Cpu(s): 26.2 us, 42.6 sy,  0.0 ni,  1.6 id, 27.9 wa,  0.0 hi,  1.6 si,  0.0 st
MiB Mem :   8192.0 total,   1281.8 free,   6736.7 used,    173.4 buff/cache
MiB Swap:   1024.0 total,    173.7 free,    850.3 used.   1455.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2627530 root      20   0 2850624 139732   7068 S  56.2   1.7 102:58.00 python3
2627764 root      20   0 7401196   6.1g   5976 R  56.2  76.8  58:51.70 ffmpeg
2627735 root      20   0 1081324  32524  14280 S  25.0   0.4  18:18.95 frigate+
2627739 root      20   0 1078296  23688  12532 D  12.5   0.3  16:49.00 frigate+
2627742 root      20   0 1083388  47024  11836 S  12.5   0.6  27:11.04 frigate+
2627750 root      20   0  829668  40964  12724 D  12.5   0.5  57:51.84 ffmpeg
2627717 root      20   0       0      0      0 Z   6.2   0.0   9:29.51 frigate+
2627760 root      20   0 1042324  20740   2932 S   6.2   0.2  45:47.55 frigate+
2964646 root      20   0    7052   3240   2812 R   6.2   0.0   0:10.77 top
2964653 root      20   0    2904    884    772 S   6.2   0.0   0:00.01 timeout
2964657 root      20   0    2904    920    812 S   6.2   0.0   0:00.01 timeout
2964659 root      20   0    5632   3056   2768 R   6.2   0.0   0:00.01 intel_g+
2964662 root      20   0    2904    912    804 S   6.2   0.0   0:00.01 timeout

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

TheDescender commented 5 months ago

@nabahr Did you ever find the root cause or maybe even a solution to this problem? I seem to be experiencing the same general issue you are. At random my frigate lxc container will just max itself out on resources. Gonna test systematically turning off my camera streams for a week and see if the problem persists. Just wanted to ask first if you made any headway with this.

Some background info: Running frigate docker container in a standalone lxc in proxmox on a Dell Precision T7810 with 2x E5-2697 v3 cpu's 12 cores assigned (to test) and a PCI coral, so there really should be no reason for this issue to ever occur besides some major stream communication error or FFMPEG instability.

blakeblackshear / frigate