search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.08k stars 1.65k forks source link

[Camera Support]: OOM killed ffmpeg process when rtsp-restream enabled #5924

Closed evilbulgarian closed 1 year ago

evilbulgarian commented 1 year ago

Describe the problem you are having

I am having an oom kill of ffmpeg process when I have preset-rtsp-restrem in the input_args, removing it ffmpeg: inputs:

Version

0.12.0-rc2

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: 10.8.8.1
  user: xxx
  password: xxx

ffmpeg:
  hwaccel_args: preset-vaapi

birdseye:
  enabled: True
  mode: motion

go2rtc:
  streams:
    entryway:
      - rtsp://wyze1.gmg.lan/live
    frontdoor:
      - rtsp://wyze2.gmg.lan/live

record:
  enabled: True
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_objects

cameras:
  frontdoor:
    detect:
      width: 1280
      height: 720
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/frontdoor
          roles:
            - detect
            - record
  entryway:
    detect:
      width: 1280
      height: 720
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/entryway
          roles:
            - detect
            - record

Relevant log output

2023-04-06 19:31:14.585492245  [2023-04-06 19:31:14] watchdog.frontdoor             INFO    : frontdoor exceeded fps limit. Exiting ffmpeg...
2023-04-06 19:31:14.585718067  [2023-04-06 19:31:14] watchdog.frontdoor             INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-06 19:31:20.466185139  172.20.0.2 - - [06/Apr/2023:19:31:20 -0700] "GET /api/stats HTTP/1.1" 200 2527 "-" "HomeAssistant/2023.4.0 aiohttp/3.8.4 Python/3.10" "-"
^[[C2023-04-06 19:31:29.466407224  172.20.0.2 - - [06/Apr/2023:19:31:29 -0700] "GET /api/stats HTTP/1.1" 200 2526 "-" "HomeAssistant/2023.4.0 aiohttp/3.8.4 Python/3.10" "-"
2023-04-06 19:31:30.631212197  19:31:30.631 WRN github.com/AlexxIT/go2rtc/cmd/streams/producer.go:133 > error="read tcp 172.20.0.4:40882->172.18.9.91:554: i/o timeout" url=rtsp://Frontent:Char16ly@wyze1.gmg.lan/live
2023-04-06 19:31:34.595539190  [2023-04-06 19:31:34] watchdog.entryway              INFO    : entryway exceeded fps limit. Exiting ffmpeg...
2023-04-06 19:31:34.595717644  [2023-04-06 19:31:34] watchdog.entryway              INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-06 19:31:38.474349320  172.20.0.2 - - [06/Apr/2023:19:31:38 -0700] "GET /api/stats HTTP/1.1" 200 2530 "-" "HomeAssistant/2023.4.0 aiohttp/3.8.4 Python/3.10" "-"
2023-04-06 19:31:45.833307431  [2023-04-06 19:31:45] frigate.record                 WARNING : Unable to find file from recordings database: /media/frigate/recordings/2023-03-31/02/front_door/44.51.mp4
2023-04-06 19:31:45.962100239  fatal error: concurrent map iteration and map write
2023-04-06 19:31:45.967621354  
2023-04-06 19:31:45.967717053  goroutine 86 [running]:
2023-04-06 19:31:45.980792898  github.com/AlexxIT/go2rtc/pkg/streamer.(*Track).WriteRTP(0xc000290540, 0xc000290570?)
2023-04-06 19:31:45.980799100   github.com/AlexxIT/go2rtc/pkg/streamer/track.go:48 +0x8d
2023-04-06 19:31:45.981739410  github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).Handle(0xc0001fe1e0)
2023-04-06 19:31:45.981744299   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:688 +0x97b
2023-04-06 19:31:45.981745742  github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).Start(0xc0001fe1e0?)
2023-04-06 19:31:45.981751252   github.com/AlexxIT/go2rtc/pkg/rtsp/producer.go:64 +0x94
2023-04-06 19:31:45.982581917  github.com/AlexxIT/go2rtc/cmd/streams.(*Producer).worker(0xc000188360, {0xcc2920?, 0xc0001fe1e0?}, 0x1)
2023-04-06 19:31:45.982586736   github.com/AlexxIT/go2rtc/cmd/streams/producer.go:124 +0x35
2023-04-06 19:31:45.983809575  created by github.com/AlexxIT/go2rtc/cmd/streams.(*Producer).reconnect
2023-04-06 19:31:45.983868355   github.com/AlexxIT/go2rtc/cmd/streams/producer.go:180 +0x42c
2023-04-06 19:31:45.983870740  
2023-04-06 19:31:45.984754214  goroutine 1 [chan receive]:
2023-04-06 19:31:45.988393145  main.main()
2023-04-06 19:31:45.988398415   github.com/AlexxIT/go2rtc/main.go:61 +0x133
2023-04-06 19:31:45.995873106  
2023-04-06 19:31:45.995878356  goroutine 6 [IO wait]:
2023-04-06 19:31:45.995894977  internal/poll.runtime_pollWait(0x7f3724802d28, 0x72)
2023-04-06 19:31:45.995896570   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:45.996069003  internal/poll.(*pollDesc).wait(0xc0000b3680?, 0x6?, 0x0)
2023-04-06 19:31:45.996081867   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:45.996789091  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:45.996810040   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:45.996811994  internal/poll.(*FD).Accept(0xc0000b3680)
2023-04-06 19:31:45.996813727   internal/poll/fd_unix.go:614 +0x234
2023-04-06 19:31:45.996907092  net.(*netFD).accept(0xc0000b3680)
2023-04-06 19:31:45.996915337   net/fd_unix.go:172 +0x35
2023-04-06 19:31:45.997026876  net.(*TCPListener).accept(0xc0000108d0)
2023-04-06 19:31:45.997029030   net/tcpsock_posix.go:142 +0x28
2023-04-06 19:31:45.997037305  net.(*TCPListener).Accept(0xc0000108d0)
2023-04-06 19:31:45.997040862   net/tcpsock.go:288 +0x3d
2023-04-06 19:31:45.997465477  net/http.(*Server).Serve(0xc0001fe000, {0xcc0e78, 0xc0000108d0})
2023-04-06 19:31:45.997486226   net/http/server.go:3070 +0x385
2023-04-06 19:31:45.997487368  github.com/AlexxIT/go2rtc/cmd/api.Init.func1()
2023-04-06 19:31:45.997488290   github.com/AlexxIT/go2rtc/cmd/api/api.go:73 +0x32
2023-04-06 19:31:45.997613494  created by github.com/AlexxIT/go2rtc/cmd/api.Init
2023-04-06 19:31:45.997621800   github.com/AlexxIT/go2rtc/cmd/api/api.go:72 +0x47d
2023-04-06 19:31:45.997623232  
2023-04-06 19:31:45.997637719  goroutine 7 [IO wait]:
2023-04-06 19:31:45.997660322  internal/poll.runtime_pollWait(0x7f3724802c38, 0x72)
2023-04-06 19:31:45.997671603   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:45.997692782  internal/poll.(*pollDesc).wait(0xc0000b3800?, 0x6?, 0x0)
2023-04-06 19:31:45.997694706   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:45.997708792  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:45.997710315   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:45.997717509  internal/poll.(*FD).Accept(0xc0000b3800)
2023-04-06 19:31:45.997718761   internal/poll/fd_unix.go:614 +0x234
2023-04-06 19:31:45.997724732  net.(*netFD).accept(0xc0000b3800)
2023-04-06 19:31:45.997726045   net/fd_unix.go:172 +0x35
2023-04-06 19:31:45.997732046  net.(*TCPListener).accept(0xc000010a68)
2023-04-06 19:31:45.997738508   net/tcpsock_posix.go:142 +0x28
2023-04-06 19:31:45.997743618  net.(*TCPListener).Accept(0xc000010a68)
2023-04-06 19:31:45.997748827   net/tcpsock.go:288 +0x3d
2023-04-06 19:31:45.997750450  github.com/AlexxIT/go2rtc/cmd/rtsp.Init.func1()
2023-04-06 19:31:45.997756181   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:62 +0x58
2023-04-06 19:31:45.997761912  created by github.com/AlexxIT/go2rtc/cmd/rtsp.Init
2023-04-06 19:31:45.997767582   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:60 +0x413
2023-04-06 19:31:45.997913025  
2023-04-06 19:31:45.997915780  goroutine 8 [IO wait]:
2023-04-06 19:31:45.997931820  internal/poll.runtime_pollWait(0x7f3724802b48, 0x72)
2023-04-06 19:31:45.997934635   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:45.997936228  internal/poll.(*pollDesc).wait(0xc0000b3a80?, 0xc000126000?, 0x0)
2023-04-06 19:31:45.997937821   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:45.998048658  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:45.998050782   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:45.998159035  internal/poll.(*FD).ReadFromInet6(0xc0000b3a80, {0xc000126000, 0x800, 0x800}, 0x7f3724ca5fff?)
2023-04-06 19:31:45.998165728   internal/poll/fd_unix.go:277 +0x1e5
2023-04-06 19:31:45.998266146  net.(*netFD).readFromInet6(0xc0000b3a80, {0xc000126000?, 0x0?, 0x417c0b?}, 0x4?)
2023-04-06 19:31:45.998276996   net/fd_posix.go:72 +0x29
2023-04-06 19:31:45.998285161  net.(*UDPConn).readFrom(0xc000100000?, {0xc000126000?, 0x0?, 0x1010000000000?}, 0xc0001160f0)
2023-04-06 19:31:45.998286554   net/udpsock_posix.go:59 +0x85
2023-04-06 19:31:45.998295611  net.(*UDPConn).readFromUDP(0xc000012970, {0xc000126000?, 0x44f352?, 0x800?}, 0xab98a0?)
2023-04-06 19:31:45.998302103   net/udpsock.go:149 +0x31
2023-04-06 19:31:45.998309196  net.(*UDPConn).ReadFrom(0x18?, {0xc000126000, 0x800, 0x800})
2023-04-06 19:31:45.998310459   net/udpsock.go:158 +0x50
2023-04-06 19:31:45.998547152  github.com/AlexxIT/go2rtc/pkg/srtp.(*Server).Serve(0xc000124000, {0xcc4e60?, 0xc000012970})
2023-04-06 19:31:45.998549967   github.com/AlexxIT/go2rtc/pkg/srtp/server.go:41 +0x82
2023-04-06 19:31:45.998668008  github.com/AlexxIT/go2rtc/cmd/srtp.Init.func1()
2023-04-06 19:31:45.998670492   github.com/AlexxIT/go2rtc/cmd/srtp/srtp.go:39 +0x85
2023-04-06 19:31:45.998671574  created by github.com/AlexxIT/go2rtc/cmd/srtp.Init
2023-04-06 19:31:45.998673268   github.com/AlexxIT/go2rtc/cmd/srtp/srtp.go:37 +0x28f
2023-04-06 19:31:46.003519739  
2023-04-06 19:31:46.003525590  goroutine 9 [IO wait]:
2023-04-06 19:31:46.003526913  internal/poll.runtime_pollWait(0x7f3724802a58, 0x72)
2023-04-06 19:31:46.003527835   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:46.003537974  internal/poll.(*pollDesc).wait(0xc0000b3c80?, 0xc000128000?, 0x0)
2023-04-06 19:31:46.003541691   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:46.003545097  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:46.003580744   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:46.003584681  internal/poll.(*FD).ReadFromInet6(0xc0000b3c80, {0xc000128000, 0x2000, 0x2000}, 0x7f3724ca5fff?)
2023-04-06 19:31:46.003588398   internal/poll/fd_unix.go:277 +0x1e5
2023-04-06 19:31:46.003591153  net.(*netFD).readFromInet6(0xc0000b3c80, {0xc000128000?, 0x0?, 0x0?}, 0x1?)
2023-04-06 19:31:46.003594049   net/fd_posix.go:72 +0x29
2023-04-06 19:31:46.003596653  net.(*UDPConn).readFrom(0xc000100000?, {0xc000128000?, 0x0?, 0x1010000000000?}, 0xc000116120)
2023-04-06 19:31:46.003632941   net/udpsock_posix.go:59 +0x85
2023-04-06 19:31:46.003635667  net.(*UDPConn).readFromUDP(0xc0000129e0, {0xc000128000?, 0x44f352?, 0x2000?}, 0xab98a0?)
2023-04-06 19:31:46.003656185   net/udpsock.go:149 +0x31
2023-04-06 19:31:46.003657768  net.(*UDPConn).ReadFrom(0x0?, {0xc000128000, 0x2000, 0x2000})
2023-04-06 19:31:46.003658900   net/udpsock.go:158 +0x50
2023-04-06 19:31:46.007714061  github.com/pion/ice/v2.(*UDPMuxDefault).connWorker(0xc0000b3d00)
2023-04-06 19:31:46.007719711   github.com/pion/ice/v2@v2.2.6/udp_mux.go:239 +0xdb
2023-04-06 19:31:46.007721605  created by github.com/pion/ice/v2.NewUDPMuxDefault
2023-04-06 19:31:46.007737765   github.com/pion/ice/v2@v2.2.6/udp_mux.go:68 +0x1e5
2023-04-06 19:31:46.007738637  
2023-04-06 19:31:46.007739438  goroutine 10 [IO wait]:
2023-04-06 19:31:46.007740380  internal/poll.runtime_pollWait(0x7f3724802968, 0x72)
2023-04-06 19:31:46.007741292   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:46.007742985  internal/poll.(*pollDesc).wait(0xc0000b3e00?, 0x0?, 0x0)
2023-04-06 19:31:46.007749567   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:46.007750780  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:46.007751902   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:46.007757893  internal/poll.(*FD).Accept(0xc0000b3e00)
2023-04-06 19:31:46.007759135   internal/poll/fd_unix.go:614 +0x234
2023-04-06 19:31:46.007760207  net.(*netFD).accept(0xc0000b3e00)
2023-04-06 19:31:46.007765868   net/fd_unix.go:172 +0x35
2023-04-06 19:31:46.007767020  net.(*TCPListener).accept(0xc000010c48)
2023-04-06 19:31:46.007772460   net/tcpsock_posix.go:142 +0x28
2023-04-06 19:31:46.007773672  net.(*TCPListener).Accept(0xc000010c48)
2023-04-06 19:31:46.007779944   net/tcpsock.go:288 +0x3d
2023-04-06 19:31:46.007781187  github.com/pion/ice/v2.(*TCPMuxDefault).start(0xc0001c4d40)
2023-04-06 19:31:46.007782359   github.com/pion/ice/v2@v2.2.6/tcp_mux.go:91 +0xdd
2023-04-06 19:31:46.007788059  github.com/pion/ice/v2.NewTCPMuxDefault.func1()
2023-04-06 19:31:46.007789242   github.com/pion/ice/v2@v2.2.6/tcp_mux.go:82 +0x5a
2023-04-06 19:31:46.007790474  created by github.com/pion/ice/v2.NewTCPMuxDefault
2023-04-06 19:31:46.007796104   github.com/pion/ice/v2@v2.2.6/tcp_mux.go:80 +0x1a5
2023-04-06 19:31:46.008988536  
2023-04-06 19:31:46.008993616  goroutine 17 [syscall]:
2023-04-06 19:31:46.008994878  os/signal.signal_recv()
2023-04-06 19:31:46.008996251   runtime/sigqueue.go:152 +0x2f
2023-04-06 19:31:46.008997373  os/signal.loop()
2023-04-06 19:31:46.009013764   os/signal/signal_unix.go:23 +0x19
2023-04-06 19:31:46.009015026  created by os/signal.Notify.func1.1
2023-04-06 19:31:46.013969250   os/signal/signal.go:151 +0x2a
2023-04-06 19:31:46.013974740  
2023-04-06 19:31:46.014025775  goroutine 33 [runnable]:
2023-04-06 19:31:46.016642673  github.com/AlexxIT/go2rtc/cmd/streams.(*Producer).stop(0xc000188360)
2023-04-06 19:31:46.016647533   github.com/AlexxIT/go2rtc/cmd/streams/producer.go:200 +0x127
2023-04-06 19:31:46.016649226  github.com/AlexxIT/go2rtc/cmd/streams.(*Stream).stopProducers(0xc0001c4580)
2023-04-06 19:31:46.016650678   github.com/AlexxIT/go2rtc/cmd/streams/stream.go:177 +0xb2
2023-04-06 19:31:46.016652312  github.com/AlexxIT/go2rtc/cmd/streams.(*Stream).RemoveConsumer(0xc0001c4580, {0xcc01b8?, 0xc000294000})
2023-04-06 19:31:46.016653724   github.com/AlexxIT/go2rtc/cmd/streams/stream.go:147 +0x13a
2023-04-06 19:31:46.017586771  github.com/AlexxIT/go2rtc/cmd/rtsp.tcpHandler.func1.1()
2023-04-06 19:31:46.017591940   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:191 +0x28
2023-04-06 19:31:46.017593123  github.com/AlexxIT/go2rtc/cmd/rtsp.tcpHandler(0xc000294000)
2023-04-06 19:31:46.017594235   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:240 +0x323
2023-04-06 19:31:46.017611066  created by github.com/AlexxIT/go2rtc/cmd/rtsp.Init.func1
2023-04-06 19:31:46.017612168   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:72 +0x48
2023-04-06 19:31:46.017612920  
2023-04-06 19:31:46.017613751  goroutine 34 [IO wait]:
2023-04-06 19:31:46.017614713  internal/poll.runtime_pollWait(0x7f3724802788, 0x72)
2023-04-06 19:31:46.017615625   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:46.017629310  internal/poll.(*pollDesc).wait(0xc00028e080?, 0xc000297000?, 0x0)
2023-04-06 19:31:46.017630312   internal/poll/fd_poll_runtime.go:84 +0x32
2023-04-06 19:31:46.018766840  internal/poll.(*pollDesc).waitRead(...)
2023-04-06 19:31:46.018771949   internal/poll/fd_poll_runtime.go:89
2023-04-06 19:31:46.018773572  internal/poll.(*FD).Read(0xc00028e080, {0xc000297000, 0x1000, 0x1000})
2023-04-06 19:31:46.018774955   internal/poll/fd_unix.go:167 +0x25a
2023-04-06 19:31:46.018789873  net.(*netFD).Read(0xc00028e080, {0xc000297000?, 0x0?, 0x4d29e0?})
2023-04-06 19:31:46.018791175   net/fd_posix.go:55 +0x29
2023-04-06 19:31:46.018792648  net.(*conn).Read(0xc000292008, {0xc000297000?, 0xc00028e080?, 0x0?})
2023-04-06 19:31:46.018793900   net/net.go:183 +0x45
2023-04-06 19:31:46.023067430  bufio.(*Reader).fill(0xc0002820c0)
2023-04-06 19:31:46.023072669   bufio/bufio.go:106 +0xff
2023-04-06 19:31:46.027952033  bufio.(*Reader).Peek(0xc0002820c0, 0x4)
2023-04-06 19:31:46.027957683   bufio/bufio.go:144 +0x5d
2023-04-06 19:31:46.027959777  github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).Handle(0xc0002940f0)
2023-04-06 19:31:46.027961721   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:598 +0x3cf
2023-04-06 19:31:46.027964316  github.com/AlexxIT/go2rtc/cmd/rtsp.tcpHandler(0xc0002940f0)
2023-04-06 19:31:46.027966530   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:236 +0x2a7
2023-04-06 19:31:46.027968624  created by github.com/AlexxIT/go2rtc/cmd/rtsp.Init.func1
2023-04-06 19:31:46.027970317   github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:72 +0x48
2023-04-06 19:31:46.027971740  
2023-04-06 19:31:46.027973693  goroutine 87 [sleep]:
2023-04-06 19:31:46.027976108  time.Sleep(0x5d21dba00)
2023-04-06 19:31:46.027977721   runtime/time.go:195 +0x135
2023-04-06 19:31:46.030542512  github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).keepalive(0xc0001fe1e0)
2023-04-06 19:31:46.030547441   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:713 +0x70
2023-04-06 19:31:46.030548743  created by github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).Handle
2023-04-06 19:31:46.030549825   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:562 +0x23c
2023-04-06 19:31:46.030550707  
2023-04-06 19:31:46.030551569  goroutine 98 [sleep]:
2023-04-06 19:31:46.030552390  time.Sleep(0x5d21dba00)
2023-04-06 19:31:46.030553252   runtime/time.go:195 +0x135
2023-04-06 19:31:46.030554244  github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).keepalive(0xc0001fe2d0)
2023-04-06 19:31:46.030555235   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:713 +0x70
2023-04-06 19:31:46.030556227  created by github.com/AlexxIT/go2rtc/pkg/rtsp.(*Conn).Handle
2023-04-06 19:31:46.030557159   github.com/AlexxIT/go2rtc/pkg/rtsp/client.go:562 +0x23c
2023-04-06 19:31:46.030557910  
2023-04-06 19:31:46.030558712  goroutine 97 [IO wait]:
2023-04-06 19:31:46.030559644  internal/poll.runtime_pollWait(0x7f3724802698, 0x72)
2023-04-06 19:31:46.030560505   runtime/netpoll.go:305 +0x89
2023-04-06 19:31:46.030561477  internal/poll

FFprobe output from your camera

no output

Frigate stats

No response

Operating system

Other Linux

Install method

Docker CLI

Coral version

USB

Network connection

Wired

Camera make and model

wyze

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

If you're referring to the log that you put it's not an OOM it's a concurrent access error.

Not sure why preset-rtsp-restream would cause that for you but we've not had this reported by anyone else. In any case, go2rtc should be restarted by frigate after this happens.

johnvprcam24573455623 commented 1 year ago

I have the exact same issue:

Apr 23 03:48:16 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,mems_allowed=0,oom_memcg=/docker/f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,task_memcg=/docker/f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,task=ffmpeg,pid=20642,uid=0 Apr 23 03:48:16 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 20642 (ffmpeg) total-vm:1227048kB, anon-rss:311204kB, file-rss:74416kB, shmem-rss:0kB, UID:0 pgtables:1104kB oom_score_adj:0

I have the same config and different cameras and also 4 Wize

NickM-27 commented 1 year ago

@johnvprcam24573455623 please make your own issue, like I pointed to above: the issue OP is pointing to does not have anything to do with OOM. So you don't have the same issue

also as an extra note, wyze cams are known to cause problems, so that's probably why

johnvprcam24573455623 commented 1 year ago

Hi, Nick,

the error is a oom-kill: kernel: oom-kill

evilbulgarian commented 1 year ago

I do have oom issue, I just pasted the relevant frigate logs during the oom kill. Here is the oom as well:

[98361.467531] database invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[98361.467538] CPU: 3 PID: 19213 Comm: database Not tainted 6.3.0 #90
[98361.467541] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[98361.467543] Call Trace:
[98361.467545]  <TASK>
[98361.467546]  dump_header+0x51/0x270
[98361.467550]  oom_kill_process+0x175/0x410
[98361.467553]  out_of_memory+0x39d/0x580
[98361.467555]  __alloc_pages_slowpath+0x935/0xef0
[98361.467558]  __alloc_pages+0x212/0x2b0
[98361.467561]  __filemap_get_folio+0x2d8/0x490
[98361.467563]  filemap_fault+0x23e/0x510
[98361.467566]  __do_fault+0x99/0x170
[98361.467568]  do_fault+0x173/0x7e0
[98361.467570]  handle_mm_fault+0xa87/0x1270
[98361.467572]  exc_page_fault+0x2ab/0x450
[98361.467576]  asm_exc_page_fault+0x26/0x30
[98361.467579] RIP: 0033:0x55a5e23f6f70
[98361.467584] Code: Unable to access opcode bytes at 0x55a5e23f6f46.
[98361.467585] RSP: 002b:00007fa769a01ea8 EFLAGS: 00010206
[98361.467588] RAX: 0000000000000000 RBX: 000055a5e264a7b8 RCX: 000055a5e264a788
[98361.467589] RDX: 000055a5e264a7b8 RSI: 000000000000005c RDI: 0000000000000006
[98361.467591] RBP: 00007fa769a01ef0 R08: 00007fa769a01e60 R09: 0000000000000000
[98361.467592] R10: 0000000000000000 R11: 0000000000000000 R12: 000055a5e264a788
[98361.467593] R13: 0000000064485af8 R14: 00007fa769a01eb0 R15: 0000000064485b03
[98361.467595]  </TASK>
[98361.467596] Mem-Info:
[98361.467597] active_anon:1774692 inactive_anon:2032936 isolated_anon:0
                active_file:3738 inactive_file:2371 isolated_file:0
                unevictable:1987 dirty:120 writeback:0
                slab_reclaimable:23669 slab_unreclaimable:32707
                mapped:1759 shmem:819 pagetables:20303
                sec_pagetables:0 bounce:0
                kernel_misc_reclaimable:0
                free:33229 free_pcp:396 free_cma:0
[98361.467602] Node 0 active_anon:7098768kB inactive_anon:8131744kB active_file:15472kB inactive_file:8964kB unevictable:7948kB isolated(anon):0kB isolated(file):0kB mapped:7036kB dirty:480kB writeback:0kB shmem:3276kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:17344kB pagetables:81212kB sec_pagetables:0kB all_unreclaimable? no
[98361.467606] DMA free:15360kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[98361.467611] lowmem_reserve[]: 0 3443 15661 15661
[98361.467614] DMA32 free:63328kB boost:0kB min:14844kB low:18552kB high:22260kB reserved_highatomic:0KB active_anon:3302004kB inactive_anon:125424kB active_file:0kB inactive_file:1916kB unevictable:0kB writepending:16kB present:3596572kB managed:3531028kB mlocked:0kB bounce:0kB free_pcp:512kB local_pcp:0kB free_cma:0kB
[98361.467618] lowmem_reserve[]: 0 0 12217 12217
[98361.467621] Normal free:54228kB boost:0kB min:52672kB low:65840kB high:79008kB reserved_highatomic:2048KB active_anon:3795288kB inactive_anon:8007796kB active_file:348kB inactive_file:23008kB unevictable:7948kB writepending:464kB present:12828672kB managed:12513636kB mlocked:7948kB bounce:0kB free_pcp:1072kB local_pcp:0kB free_cma:0kB
[98361.467625] lowmem_reserve[]: 0 0 0 0
[98361.467627] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
[98361.467636] DMA32: 1035*4kB (UME) 752*8kB (UME) 458*16kB (UME) 291*32kB (UE) 94*64kB (UE) 37*128kB (UME) 13*256kB (UE) 28*512kB (UME) 6*1024kB (UM) 1*2048kB (M) 0*4096kB = 63404kB
[98361.467647] Normal: 1780*4kB (UE) 1218*8kB (UME) 658*16kB (UME) 421*32kB (UME) 110*64kB (UME) 50*128kB (UE) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54560kB
[98361.467657] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[98361.467659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[98361.467660] 14039 total pagecache pages
[98361.467661] 5379 pages in swap cache
[98361.467662] Free swap  = 12kB
[98361.467662] Total swap = 16777208kB
[98361.467663] 4110310 pages RAM
[98361.467664] 0 pages HighMem/MovableOnly
[98361.467664] 95304 pages reserved
[98361.467665] 0 pages hwpoisoned
[98361.467665] Tasks state (memory values in pages):
[98361.467666] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[98361.467671] [   1660]     0  1660     1726      496    57344       64         -1000 sshd
[98361.467674] [   1721]     0  1721     4821      480    65536      384             0 systemd-udevd
[98361.467678] [   3451]     0  3451      816      550    45056       96             0 dhcpcd
[98361.467681] [   3926]   998  3926     1228      416    49152       64             0 dbus-daemon
[98361.467683] [   4035]    61  4035     1224      480    45056      224             0 avahi-daemon
[98361.467685] [   4036]    61  4036     1075      352    45056       32             0 avahi-daemon
[98361.467687] [   4069]     0  4069    54566      439    73728      416             0 rsyslogd
[98361.467690] [   4103]     0  4103     1690      387    49152       32             0 crond
[98361.467692] [   4135]   997  4135     2758     1139    61440      256             0 stubby
[98361.467694] [   4167]   101  4167     1520      452    53248        0             0 dnsmasq
[98361.467696] [   4199]   102  4199     5034      256    65536     2432             0 ddclient - slee
[98361.467698] [   4230]     0  4230   468424     1011   372736     4503             0 containerd
[98361.467700] [   4304]   482  4304     1847     1152    53248      128             0 mosquitto
[98361.467702] [   4397]    60  4397   525592     1640   602112    37152             0 mysqld
[98361.467704] [   6020]     0  6020     4648      266    57344      224             0 nginx
[98361.467706] [   6021]   103  6021     4919     1066    61440      352             0 nginx
[98361.467708] [   6182]   123  6182    18347     1933    57344        0             0 ntpd
[98361.467710] [   6284]     0  6284     5083      384    86016     4384             0 pgld
[98361.467712] [   6328]   215  6328      627      448    40960        0             0 radvd
[98361.467714] [   6329]     0  6329      627      288    40960        0             0 radvd
[98361.467716] [   6359]     0  6359     2499      416    61440       96             0 rsync
[98361.467718] [   6392]     0  6392    18763      522   135168      544             0 smbd
[98361.467720] [   6396]     0  6396    18341      433   131072      576             0 smbd-notifyd
[98361.467722] [   6397]     0  6397    18339      401   114688      576             0 cleanupd
[98361.467724] [   6400]     0  6400    16335      548   102400      416             0 nmbd
[98361.467726] [   6443]   104  6443   515443     2142   311296     1477             0 zabbix_agent2
[98361.467728] [   6484]     0  6484     1447      480    45056        0             0 agetty
[98361.467730] [   6485]     0  6485     1447      512    49152        0             0 agetty
[98361.467732] [   6486]     0  6486     1447      448    49152        0             0 agetty
[98361.467734] [   6487]     0  6487     1447      480    45056       32             0 agetty
[98361.467736] [   6488]     0  6488     1447      448    45056       32             0 agetty
[98361.467738] [   6489]     0  6489     1447      512    45056        0             0 agetty
[98361.467739] [   6767]     0  6767     9264      448    73728       96             0 elogind-daemon
[98361.467742] [   7431]  1000  7431     3836      416    69632     1504             0 tmux: server
[98361.467744] [   7432]  1000  7432     2746      664    57344      672             0 zsh
[98361.467746] [   7492]  1000  7492     2789      562    61440      736             0 zsh
[98361.467748] [   7546]  1000  7546     2710      604    61440      640             0 zsh
[98361.467750] [   7604]  1000  7604    79772     3489   393216    36192             0 rtorrent main
[98361.467752] [  21487] 65534 21487     2798      621    69632      161             0 rsync
[98361.467754] [  10922]  1000 10922     1447      352    49152        0             0 tail
[98361.467757] [  17450]     0 17450   623039     1031   512000     7103             0 dockerd
[98361.467759] [  17679]     0 17679   288815      352   126976      160             0 docker-proxy
[98361.467761] [  17692]     0 17692   288815      352   139264      160             0 docker-proxy
[98361.467763] [  17705]     0 17705   270446      320   126976      160             0 docker-proxy
[98361.467765] [  17723]     0 17723   307376      320   147456      192             0 docker-proxy
[98361.467767] [  17751]     0 17751   180179      520   118784      576             1 containerd-shim
[98361.467768] [  17775]     0 17775   180243      538   122880      576             1 containerd-shim
[98361.467770] [  17804]     0 17804   180179      552   110592      512             1 containerd-shim
[98361.467772] [  17831]     0 17831       53        0    28672        0             0 s6-svscan
[98361.467774] [  17832]     0 17832      975       64    49152       64             0 entrypoint-coll
[98361.467777] [  17833]     0 17833      255       32    28672        0             0 docker-init
[98361.467779] [  17917]     0 17917   270446      352   126976      160             0 docker-proxy
[98361.467781] [  17929]     0 17929   344242      320   163840      192             0 docker-proxy
[98361.467783] [  17943]     0 17943   270446      320   131072      160             0 docker-proxy
[98361.467785] [  17953]     0 17953   288879      320   135168      160             0 docker-proxy
[98361.467786] [  17968]     0 17968   270382      320   126976      160             0 docker-proxy
[98361.467788] [  17985]     0 17985   180243      602   114688      480             1 containerd-shim
[98361.467790] [  18030]     0 18030   180179      521   118784      509             1 containerd-shim
[98361.467792] [  18079]     0 18079   307376      352   151552      192             0 docker-proxy
[98361.467794] [  18102]     0 18102   307376      352   143360      416             0 docker-proxy
[98361.467796] [  18127]     0 18127     1551       32    53248       96             0 su
[98361.467798] [  18131]     0 18131      937       96    40960       32             0 cron
[98361.467800] [  18137]  1000 18137    45069      315   434176    38464             0 flexget
[98361.467802] [  18139]     0 18139   179213      126   122880     1440             0 watchtower
[98361.467804] [  18155]     0 18155   288879      320   135168      192             0 docker-proxy
[98361.467806] [  18195]     0 18195      204        0    45056        0             0 tini
[98361.467808] [  18196]     0 18196    87313      716  3305472    14509             0 node
[98361.467810] [  18258]     0 18258   270446      352   131072      192             0 docker-proxy
[98361.467812] [  18265]     0 18265       53        0    28672        0             0 s6-supervise
[98361.467814] [  18266]     0 18266       50        0    24576        0             0 s6-linux-init-s
[98361.467816] [  18274]     0 18274       53        0    28672        0             0 s6-supervise
[98361.467818] [  18275]     0 18275       53        0    28672        0             0 s6-supervise
[98361.467819] [  18276]     0 18276       53        0    28672        0             0 s6-supervise
[98361.467821] [  18277]     0 18277       53        0    28672        0             0 s6-supervise
[98361.467823] [  18278]     0 18278       53        0    28672        0             0 s6-supervise
[98361.467825] [  18279]     0 18279       53        0    28672        0             0 s6-supervise
[98361.467827] [  18280]     0 18280       53        0    28672        0             0 s6-supervise
[98361.467829] [  18281]     0 18281       53        0    28672        0             0 s6-supervise
[98361.467830] [  18282]     0 18282       53        0    24576        0             0 s6-supervise
[98361.467832] [  18291]     0 18291       47        0    24576        0             0 s6-ipcserverd
[98361.467834] [  18292]     0 18292      131        0    28672        0             0 s6-fdholderd
[98361.467836] [  18336] 65534 18336       69        0    36864        0             0 s6-log
[98361.467838] [  18337] 65534 18337       69        0    40960        0             0 s6-log
[98361.467840] [  18338] 65534 18338       69       32    40960        0             0 s6-log
[98361.467842] [  18345]     0 18345   181793     1993   139264     3263             0 go2rtc
[98361.467844] [  18361]     0 18361      973       32    45056       32             0 bash
[98361.467846] [  18366]     0 18366   773263     3344   933888    31744             0 python3
[98361.467849] [  18378]     0 18378   145420      192   110592      448             0 nginx
[98361.467851] [  18394]     0 18394   211097      140   221184      608             0 nginx
[98361.467852] [  18395]     0 18395   211097      108   221184      608             0 nginx
[98361.467854] [  18396]     0 18396   211097      108   221184      608             0 nginx
[98361.467856] [  18401]     0 18401   211097      172   221184      576             0 nginx
[98361.467858] [  18413]     0 18413   211097      172   221184      608             0 nginx
[98361.467860] [  18442]     0 18442   211097      108   221184      608             0 nginx
[98361.467862] [  18464]     0 18464   211097      108   221184      640             0 nginx
[98361.467864] [  18465]     0 18465   211097       76   221184      640             0 nginx
[98361.467866] [  18652]     0 18652   180179      644   118784      576             1 containerd-shim
[98361.467868] [  18718]     0 18718       52        0    32768        0             0 s6-svscan
[98361.467870] [  18867]     0 18867       53        0    32768        0             0 s6-supervise
[98361.467871] [  18868]     0 18868       50        0    28672        0             0 s6-linux-init-s
[98361.467873] [  18890]     0 18890       53        0    32768        0             0 s6-supervise
[98361.467875] [  18891]     0 18891       53        0    32768        0             0 s6-supervise
[98361.467877] [  18892]     0 18892       53        0    32768        0             0 s6-supervise
[98361.467879] [  18893]     0 18893       53        0    32768        0             0 s6-supervise
[98361.467881] [  18894]     0 18894       53        0    32768        0             0 s6-supervise
[98361.467882] [  18895]     0 18895       53        0    32768        0             0 s6-supervise
[98361.467884] [  18896]     0 18896       53        0    32768        0             0 s6-supervise
[98361.467886] [  18903]     0 18903       47        0    28672        0             0 s6-ipcserverd
[98361.467888] [  18921]     0 18921      975       96    45056       64             0 bash
[98361.467890] [  18927]     0 18927      937       64    45056       32             0 cron
[98361.467892] [  19084]     0 19084      975       96    45056       64             0 bash
[98361.467894] [  19090]     0 19090      975       64    36864       64             0 bash
[98361.467896] [  19115]    33 19115     1112        0    40960      160             0 lighttpd
[98361.467898] [  19125]   999 19125      975       96    53248       64             0 bash
[98361.467900] [  19126]   999 19126    79635      271   159744      864             0 pihole-FTL
[98361.467902] [  19148]    33 19148    49474       96   139264     1056             0 php-cgi
[98361.467904] [  19150]    33 19150    49474       36   106496     1056             0 php-cgi
[98361.467906] [  19152]    33 19152    49474       36   106496     1056             0 php-cgi
[98361.467908] [  19153]    33 19153    49474       36   106496     1056             0 php-cgi
[98361.467910] [  19154]    33 19154    49474       32   106496     1056             0 php-cgi
[98361.467911] [  19170]     0 19170   213767      495   368640    18016             0 frigate.logger
[98361.467913] [  19176]     0 19176     3564      160    61440     1216             0 python3
[98361.467915] [  19179]     0 19179   300073      731   487424    19168             0 frigate.output
[98361.467917] [  19185]     0 19185    31705       96   176128      704             0 ffmpeg
[98361.467919] [  19186]     0 19186   375385     1036   528384    20608             0 frigate.process
[98361.467921] [  19187]     0 19187   374785      739   524288    20416             0 frigate.process
[98361.467923] [  19188]     0 19188    31705       96   167936      736             0 ffmpeg
[98361.467925] [  19189]     0 19189   321482     1502   434176    18496             0 frigate.capture
[98361.467927] [  19193]     0 19193   318106      556   450560    19808             0 frigate.capture
[98361.467929] [  19197]     0 19197   237312     2768   622592     7072             0 ffmpeg
[98361.467931] [  16209]     0 16209  7934561  3765380 61620224  3849504             0 ffmpeg
[98361.467933] [  17368]  1000 17368     2787      589    61440      672             0 zsh
[98361.467936] [  22347]     0 22347   362803      384   176128      256             0 docker-proxy
[98361.467938] [  22361]     0 22361   180179      618   118784      512             1 containerd-shim
[98361.467939] [  22381]     0 22381       53        0    28672        0             0 s6-svscan
[98361.467941] [  22468]     0 22468       53        0    28672        0             0 s6-supervise
[98361.467943] [  22470]     0 22470       50        0    24576        0             0 s6-linux-init-s
[98361.467945] [  22477]     0 22477       53        0    28672        0             0 s6-supervise
[98361.467947] [  22478]     0 22478       53        0    28672        0             0 s6-supervise
[98361.467949] [  22486]     0 22486       47        0    24576        0             0 s6-ipcserverd
[98361.467951] [  22517]     0 22517       53        0    24576        0             0 s6-supervise
[98361.467953] [  22519]     0 22519   368073    10635  1032192    49888             0 python3
[98361.467955] [  23807]  1000 23807   271172      256   131072      384             0 idalv-rest
[98361.467957] [  23859]  1000 23859     1881      416    49152       64             0 kodi-standalone
[98361.467959] [  23861]  1000 23861     1914      416    57344       96             0 kodi
[98361.467961] [  23866]  1000 23866   628248     4604  1282048    49376             0 kodi-gbm
[98361.467963] [  25950]     0 25950   731283     1129   692224    36457             0 frigate.detecto
[98361.467965] [  28490]     0 28490      622       32    40960        0             0 sleep
[98361.467967] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=0020a45490e1cb39c279c9a9ead8996ebac7867643743bfaa4d486d78b373d78,mems_allowed=0,global_oom,task_memcg=/docker/ea829defadeb33e4dd8a3134f460e276a863a86bd23b5b277e20afff4319ca28,task=ffmpeg,pid=16209,uid=0
[98361.468001] Out of memory: Killed process 16209 (ffmpeg) total-vm:31738244kB, anon-rss:15061264kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:60176kB oom_score_adj:0

@johnvprcam24573455623 please make your own issue, like I pointed to above: the issue OP is pointing to does not have anything to do with OOM. So you don't have the same issue

also as an extra note, wyze cams are known to cause problems, so that's probably why

NickM-27 commented 1 year ago

Those logs were from go2rtc and not frigate, and they did not indicate an oom so there was nothing pointing to that.

We'd need to show an htop output or something similar showing which process in frigate is using the memory

evilbulgarian commented 1 year ago

Its the ffmpeg process which is in my output above: [98361.467666] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [98361.467931] [ 16209] 0 16209 7934561 3765380 61620224 3849504 0 ffmpeg

NickM-27 commented 1 year ago

That doesn't give useful info though, we need to know which ffmpeg command is it because there's many different types of ffmpeg processes throughout frigate

johnvprcam24573455623 commented 1 year ago

Same here and it also crashed my BOX:

Memory cgroup out of memory: Killed process 6126 (ffmpeg) total-vm:3433428kB, anon-rss:2454116kB, file-rss:64204kB, shmem-rss:0kB, UID:0 pgtables:5308kB oom_score_adj:0

Can we activate any additional logs that can help throubleshoot?

This is the complete logs of the crash

Apr 28 09:17:19 UNRAIDSRV kernel: CPU: 7 PID: 30278 Comm: frigate.process Tainted: G           O      5.19.17-Unraid #2
Apr 28 09:17:19 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:17:19 UNRAIDSRV kernel: Call Trace:
Apr 28 09:17:19 UNRAIDSRV kernel: <TASK>
Apr 28 09:17:19 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:17:19 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:17:19 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:17:19 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:17:19 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:17:19 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:17:19 UNRAIDSRV kernel: ? select_task_rq_fair+0xb38/0xb4e
Apr 28 09:17:19 UNRAIDSRV kernel: obj_cgroup_charge_pages+0x28/0x4d
Apr 28 09:17:19 UNRAIDSRV kernel: obj_cgroup_charge+0x70/0xd7
Apr 28 09:17:19 UNRAIDSRV kernel: slab_pre_alloc_hook.constprop.0+0x8e/0x14b
Apr 28 09:17:19 UNRAIDSRV kernel: ? __alloc_file+0x26/0x93
Apr 28 09:17:19 UNRAIDSRV kernel: kmem_cache_alloc+0x48/0x143
Apr 28 09:17:19 UNRAIDSRV kernel: ? rcuwait_wake_up+0x2d/0x38
Apr 28 09:17:19 UNRAIDSRV kernel: __alloc_file+0x26/0x93
Apr 28 09:17:19 UNRAIDSRV kernel: alloc_empty_file+0x7f/0xd5
Apr 28 09:17:19 UNRAIDSRV kernel: path_openat+0x4b/0xaa9
Apr 28 09:17:19 UNRAIDSRV kernel: ? current_time+0x5a/0x7a
Apr 28 09:17:19 UNRAIDSRV kernel: ? atime_needs_update+0x9c/0xf7
Apr 28 09:17:19 UNRAIDSRV kernel: do_filp_open+0x55/0xb8
Apr 28 09:17:19 UNRAIDSRV kernel: ? pipe_read+0x30a/0x33e
Apr 28 09:17:19 UNRAIDSRV kernel: ? __handle_mm_fault+0xa6d/0xc7d
Apr 28 09:17:19 UNRAIDSRV kernel: ? slab_post_alloc_hook+0x4d/0x15e
Apr 28 09:17:19 UNRAIDSRV kernel: ? getname_flags+0x29/0x152
Apr 28 09:17:19 UNRAIDSRV kernel: ? kmem_cache_alloc+0x11a/0x143
Apr 28 09:17:19 UNRAIDSRV kernel: do_sys_openat2+0x6c/0xd9
Apr 28 09:17:19 UNRAIDSRV kernel: do_sys_open+0x3a/0x5a
Apr 28 09:17:19 UNRAIDSRV kernel: do_syscall_64+0x6b/0x81
Apr 28 09:17:19 UNRAIDSRV kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Apr 28 09:17:19 UNRAIDSRV kernel: RIP: 0033:0x154cb2b97e41
Apr 28 09:17:19 UNRAIDSRV kernel: Code: 44 24 18 31 c0 41 83 e2 40 75 3e 89 f0 25 00 00 41 00 3d 00 00 41 00 74 30 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 3f 48 8b 54 24 18 64 48 2b 14 25 28 00 00 00
Apr 28 09:17:19 UNRAIDSRV kernel: RSP: 002b:00007ffc8ac70110 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
Apr 28 09:17:19 UNRAIDSRV kernel: RAX: ffffffffffffffda RBX: 00007ffc8ac70170 RCX: 0000154cb2b97e41
Apr 28 09:17:19 UNRAIDSRV kernel: RDX: 00000000000a0002 RSI: 00007ffc8ac70170 RDI: 00000000ffffff9c
Apr 28 09:17:19 UNRAIDSRV kernel: RBP: 0000000000000180 R08: 0000000000000000 R09: 0000000000000000
Apr 28 09:17:19 UNRAIDSRV kernel: R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000002
Apr 28 09:17:19 UNRAIDSRV kernel: R13: 0000154c800916f0 R14: 0000154c80091720 R15: 00000000009636f0
Apr 28 09:17:19 UNRAIDSRV kernel: </TASK>
Apr 28 09:17:19 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 81395
Apr 28 09:17:19 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:17:19 UNRAIDSRV kernel: kmem: usage 36076kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:17:19 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:17:19 UNRAIDSRV kernel: anon 3157348352
Apr 28 09:17:19 UNRAIDSRV kernel: file 26935296
Apr 28 09:17:19 UNRAIDSRV kernel: kernel 36941824
Apr 28 09:17:19 UNRAIDSRV kernel: kernel_stack 7766016
Apr 28 09:17:19 UNRAIDSRV kernel: pagetables 18161664
Apr 28 09:17:19 UNRAIDSRV kernel: percpu 10064
Apr 28 09:17:19 UNRAIDSRV kernel: sock 0
Apr 28 09:17:19 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:17:19 UNRAIDSRV kernel: shmem 26791936
Apr 28 09:17:19 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:17:19 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:17:19 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:17:19 UNRAIDSRV kernel: swapcached 0
Apr 28 09:17:19 UNRAIDSRV kernel: anon_thp 312475648
Apr 28 09:17:19 UNRAIDSRV kernel: file_thp 0
Apr 28 09:17:19 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:17:19 UNRAIDSRV kernel: inactive_anon 3174850560
Apr 28 09:17:19 UNRAIDSRV kernel: active_anon 9289728
Apr 28 09:17:19 UNRAIDSRV kernel: inactive_file 143360
Apr 28 09:17:19 UNRAIDSRV kernel: active_file 0
Apr 28 09:17:19 UNRAIDSRV kernel: unevictable 0
Apr 28 09:17:19 UNRAIDSRV kernel: slab_reclaimable 956032
Apr 28 09:17:19 UNRAIDSRV kernel: slab_unreclaimable 8774296
Apr 28 09:17:19 UNRAIDSRV kernel: slab 9730328
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_refault_file 225956
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_activate_file 19929
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_restore_file 3925
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:17:19 UNRAIDSRV kernel: pgfault 369855614
Apr 28 09:17:19 UNRAIDSRV kernel: pgmajfault 1336
Apr 28 09:17:19 UNRAIDSRV kernel: pgrefill 64608
Apr 28 09:17:19 UNRAIDSRV kernel: pgscan 554608
Apr 28 09:17:19 UNRAIDSRV kernel: pgsteal 261286
Apr 28 09:17:19 UNRAIDSRV kernel: pgactivate 106265120
Apr 28 09:17:19 UNRAIDSRV kernel: pgdeactivate 64212
Apr 28 09:17:19 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:17:19 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:17:19 UNRAIDSRV kernel: thp_fault_alloc 17074
Apr 28 09:17:19 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:17:19 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:17:19 UNRAIDSRV kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:17:19 UNRAIDSRV kernel: [  29675]     0 29675       53       16    24576        0             0 s6-svscan
Apr 28 09:17:19 UNRAIDSRV kernel: [  29747]     0 29747       53       14    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29748]     0 29748       50        1    24576        0             0 s6-linux-init-s
Apr 28 09:17:19 UNRAIDSRV kernel: [  29756]     0 29756       53       16    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29757]     0 29757       53       15    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29758]     0 29758       53       16    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29759]     0 29759       53       15    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29760]     0 29760       53       15    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29761]     0 29761       53       15    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29762]     0 29762       53       16    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29763]     0 29763       53       16    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29764]     0 29764       53       16    24576        0             0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [  29773]     0 29773       47        1    24576        0             0 s6-ipcserverd
Apr 28 09:17:19 UNRAIDSRV kernel: [  29774]     0 29774      131       41    24576        0             0 s6-fdholderd
Apr 28 09:17:19 UNRAIDSRV kernel: [  29812] 65534 29812       69       27    36864        0             0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [  29813] 65534 29813       69       28    36864        0             0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [  29814] 65534 29814       70       27    36864        0             0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [  29821]     0 29821   181857     6962   143360        0             0 go2rtc
Apr 28 09:17:19 UNRAIDSRV kernel: [  29830]     0 29830      973      730    40960        0             0 bash
Apr 28 09:17:19 UNRAIDSRV kernel: [  29833]     0 29833   618176    44055   905216        0             0 python3
Apr 28 09:17:19 UNRAIDSRV kernel: [  29849]     0 29849   145420     3479   102400        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29866]     0 29866   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29867]     0 29867   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29868]     0 29868   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29875]     0 29875   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29897]     0 29897   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29932]     0 29932   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29962]     0 29962   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  29970]     0 29970   161947     1584   212992        0             0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [  30266]     0 30266   212817    19824   376832        0             0 frigate.logger
Apr 28 09:17:19 UNRAIDSRV kernel: [  30272]     0 30272     3564     2647    65536        0             0 python3
Apr 28 09:17:19 UNRAIDSRV kernel: [  30273]     0 30273   301653    26449   479232        0             0 frigate.detecto
Apr 28 09:17:19 UNRAIDSRV kernel: [  30275]     0 30275   349154    24376   524288        0             0 frigate.output
Apr 28 09:17:19 UNRAIDSRV kernel: [  30278]     0 30278   366610    25271   536576        0             0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [  30279]     0 30279    31705     5386   172032        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30280]     0 30280   374564    33602   602112        0             0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [  30284]     0 30284   371525    30250   573440        0             0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [  30285]     0 30285    31705     4897   163840        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30286]     0 30286   369682    28568   565248        0             0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [  30287]     0 30287    31705     4486   163840        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30288]     0 30288   316513    23315   495616        0             0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [  30289]     0 30289   316552    22757   475136        0             0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [  30292]     0 30292   324793    23723   466944        0             0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [  30293]     0 30293    31705     5347   163840        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30297]     0 30297   317199    25091   487424        0             0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [  30298]     0 30298   257592    29999   622592        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30303]     0 30303    31705     5371   163840        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30304]     0 30304   320153    23173   462848        0             0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [  30310]     0 30310   317199    22945   483328        0             0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [  30312]     0 30312   308966    31568   647168        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  30315]     0 30315    31705     5881   167936        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  27058]     0 27058   872202   627124  5423104        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [  31302]     0 31302   306464    26832   626688        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [   3524]     0  3524    32871     6892   196608        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [   4124]     0  4124      622      213    40960        0             0 sleep
Apr 28 09:17:19 UNRAIDSRV kernel: [   4506]     0  4506    31668     3866   163840        0             0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [   4514]     0  4514    31645     2414   147456        0             0 ffprobe
Apr 28 09:17:19 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=27058,uid=0
Apr 28 09:17:19 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 27058 (ffmpeg) total-vm:3488808kB, anon-rss:2443960kB, file-rss:64536kB, shmem-rss:0kB, UID:0 pgtables:5296kB oom_score_adj:0
Apr 28 09:18:46 UNRAIDSRV kernel: ffmpeg invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Apr 28 09:18:46 UNRAIDSRV kernel: CPU: 0 PID: 4705 Comm: ffmpeg Tainted: G           O      5.19.17-Unraid #2
Apr 28 09:18:46 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:18:46 UNRAIDSRV kernel: Call Trace:
Apr 28 09:18:46 UNRAIDSRV kernel: <TASK>
Apr 28 09:18:46 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:18:46 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:18:46 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:18:46 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:18:46 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:18:46 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:18:46 UNRAIDSRV kernel: charge_memcg+0x29/0x71
Apr 28 09:18:46 UNRAIDSRV kernel: __mem_cgroup_charge+0x29/0x41
Apr 28 09:18:46 UNRAIDSRV kernel: __handle_mm_fault+0x813/0xc7d
Apr 28 09:18:46 UNRAIDSRV kernel: ? __default_send_IPI_dest_field+0x32/0x50
Apr 28 09:18:46 UNRAIDSRV kernel: handle_mm_fault+0x113/0x1d7
Apr 28 09:18:46 UNRAIDSRV kernel: do_user_addr_fault+0x36a/0x514
Apr 28 09:18:46 UNRAIDSRV kernel: exc_page_fault+0xfc/0x11e
Apr 28 09:18:46 UNRAIDSRV kernel: asm_exc_page_fault+0x22/0x30
Apr 28 09:18:46 UNRAIDSRV kernel: RIP: 0033:0x15042db317f7
Apr 28 09:18:46 UNRAIDSRV kernel: Code: 48 ff c7 48 01 fe 48 8d 54 11 80 0f 1f 84 00 00 00 00 00 c5 fe 6f 0e c5 fe 6f 56 20 c5 fe 6f 5e 40 c5 fe 6f 66 60 48 83 ee 80 <c5> fd 7f 0f c5 fd 7f 57 20 c5 fd 7f 5f 40 c5 fd 7f 67 60 48 83 ef
Apr 28 09:18:46 UNRAIDSRV kernel: RSP: 002b:00007ffdda03ea68 EFLAGS: 00010207
Apr 28 09:18:46 UNRAIDSRV kernel: RAX: 000055772af8f730 RBX: 000015042bfe7800 RCX: 000055772af8f730
Apr 28 09:18:46 UNRAIDSRV kernel: RDX: 000055772af8fe30 RSI: 000015042bfe7090 RDI: 000055772af8f740
Apr 28 09:18:46 UNRAIDSRV kernel: RBP: 0000000000000780 R08: 0000000000000000 R09: 0000000000000000
Apr 28 09:18:46 UNRAIDSRV kernel: R10: 0000000000000000 R11: 000000000000021b R12: 0000000000000800
Apr 28 09:18:46 UNRAIDSRV kernel: R13: 000000000000021c R14: 0000000000000f00 R15: 00000000000001eb
Apr 28 09:18:46 UNRAIDSRV kernel: </TASK>
Apr 28 09:18:46 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 92888
Apr 28 09:18:46 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:18:46 UNRAIDSRV kernel: kmem: usage 35904kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:18:46 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:18:46 UNRAIDSRV kernel: anon 3159777280
Apr 28 09:18:46 UNRAIDSRV kernel: file 24682496
Apr 28 09:18:46 UNRAIDSRV kernel: kernel 36765696
Apr 28 09:18:46 UNRAIDSRV kernel: kernel_stack 7667712
Apr 28 09:18:46 UNRAIDSRV kernel: pagetables 17879040
Apr 28 09:18:46 UNRAIDSRV kernel: percpu 10064
Apr 28 09:18:46 UNRAIDSRV kernel: sock 0
Apr 28 09:18:46 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:18:46 UNRAIDSRV kernel: shmem 24584192
Apr 28 09:18:46 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:18:46 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:18:46 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:18:46 UNRAIDSRV kernel: swapcached 0
Apr 28 09:18:46 UNRAIDSRV kernel: anon_thp 784334848
Apr 28 09:18:46 UNRAIDSRV kernel: file_thp 0
Apr 28 09:18:46 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:18:46 UNRAIDSRV kernel: inactive_anon 3175759872
Apr 28 09:18:46 UNRAIDSRV kernel: active_anon 8585216
Apr 28 09:18:46 UNRAIDSRV kernel: inactive_file 45056
Apr 28 09:18:46 UNRAIDSRV kernel: active_file 0
Apr 28 09:18:46 UNRAIDSRV kernel: unevictable 0
Apr 28 09:18:46 UNRAIDSRV kernel: slab_reclaimable 1285840
Apr 28 09:18:46 UNRAIDSRV kernel: slab_unreclaimable 8689776
Apr 28 09:18:46 UNRAIDSRV kernel: slab 9975616
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_refault_file 254111
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_activate_file 22716
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_restore_file 3925
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:18:46 UNRAIDSRV kernel: pgfault 372540282
Apr 28 09:18:46 UNRAIDSRV kernel: pgmajfault 1453
Apr 28 09:18:46 UNRAIDSRV kernel: pgrefill 71669
Apr 28 09:18:46 UNRAIDSRV kernel: pgscan 588191
Apr 28 09:18:46 UNRAIDSRV kernel: pgsteal 290931
Apr 28 09:18:46 UNRAIDSRV kernel: pgactivate 106916000
Apr 28 09:18:46 UNRAIDSRV kernel: pgdeactivate 71272
Apr 28 09:18:46 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:18:46 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:18:46 UNRAIDSRV kernel: thp_fault_alloc 17469
Apr 28 09:18:46 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:18:46 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:18:46 UNRAIDSRV kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:18:46 UNRAIDSRV kernel: [  29675]     0 29675       53       16    24576        0             0 s6-svscan
Apr 28 09:18:46 UNRAIDSRV kernel: [  29747]     0 29747       53       14    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29748]     0 29748       50        1    24576        0             0 s6-linux-init-s
Apr 28 09:18:46 UNRAIDSRV kernel: [  29756]     0 29756       53       16    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29757]     0 29757       53       15    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29758]     0 29758       53       16    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29759]     0 29759       53       15    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29760]     0 29760       53       15    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29761]     0 29761       53       15    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29762]     0 29762       53       16    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29763]     0 29763       53       16    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29764]     0 29764       53       16    24576        0             0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [  29773]     0 29773       47        1    24576        0             0 s6-ipcserverd
Apr 28 09:18:46 UNRAIDSRV kernel: [  29774]     0 29774      131       41    24576        0             0 s6-fdholderd
Apr 28 09:18:46 UNRAIDSRV kernel: [  29812] 65534 29812       69       27    36864        0             0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [  29813] 65534 29813       69       28    36864        0             0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [  29814] 65534 29814       70       27    36864        0             0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [  29821]     0 29821   181857     7116   143360        0             0 go2rtc
Apr 28 09:18:46 UNRAIDSRV kernel: [  29830]     0 29830      973      729    40960        0             0 bash
Apr 28 09:18:46 UNRAIDSRV kernel: [  29833]     0 29833   618176    44055   905216        0             0 python3
Apr 28 09:18:46 UNRAIDSRV kernel: [  29849]     0 29849   145420     3477   102400        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29866]     0 29866   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29867]     0 29867   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29868]     0 29868   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29875]     0 29875   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29897]     0 29897   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29932]     0 29932   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29962]     0 29962   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  29970]     0 29970   161947     1582   212992        0             0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [  30266]     0 30266   212817    19820   376832        0             0 frigate.logger
Apr 28 09:18:46 UNRAIDSRV kernel: [  30272]     0 30272     3564     2644    65536        0             0 python3
Apr 28 09:18:46 UNRAIDSRV kernel: [  30273]     0 30273   301653    26410   479232        0             0 frigate.detecto
Apr 28 09:18:46 UNRAIDSRV kernel: [  30275]     0 30275   349154    24375   524288        0             0 frigate.output
Apr 28 09:18:46 UNRAIDSRV kernel: [  30278]     0 30278   366610    25268   536576        0             0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [  30279]     0 30279    31705     5380   172032        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30280]     0 30280   374564    33600   602112        0             0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [  30284]     0 30284   371525    30248   573440        0             0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [  30285]     0 30285    31705     4891   163840        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30286]     0 30286   369682    28567   565248        0             0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [  30287]     0 30287    31705     4480   163840        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30288]     0 30288   316513    23312   495616        0             0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [  30289]     0 30289   316552    22754   475136        0             0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [  30292]     0 30292   324793    23720   466944        0             0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [  30293]     0 30293    31705     5341   163840        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30297]     0 30297   317199    25089   487424        0             0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [  30298]     0 30298   257592    29958   622592        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30303]     0 30303    31705     5365   163840        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30304]     0 30304   320153    23170   462848        0             0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [  30310]     0 30310   316439    22944   483328        0             0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [  30312]     0 30312   307946    31652   647168        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  30315]     0 30315    31705     5875   167936        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [  31302]     0 31302   306464    26822   626688        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [   3524]     0  3524    32871     6940   196608        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [   4705]     0  4705   824549   629241  5427200        0             0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [   5670]     0  5670      622      216    40960        0             0 sleep
Apr 28 09:18:46 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=4705,uid=0
Apr 28 09:18:46 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 4705 (ffmpeg) total-vm:3298196kB, anon-rss:2452320kB, file-rss:64644kB, shmem-rss:0kB, UID:0 pgtables:5300kB oom_score_adj:0
Apr 28 09:20:24 UNRAIDSRV kernel: ffmpeg invoked oom-killer: gfp_mask=0x8c40(GFP_NOFS|__GFP_NOFAIL), order=0, oom_score_adj=0
Apr 28 09:20:24 UNRAIDSRV kernel: CPU: 6 PID: 6126 Comm: ffmpeg Tainted: G           O      5.19.17-Unraid #2
Apr 28 09:20:24 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:20:24 UNRAIDSRV kernel: Call Trace:
Apr 28 09:20:24 UNRAIDSRV kernel: <TASK>
Apr 28 09:20:24 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:20:24 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:20:24 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:20:24 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:20:24 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:20:24 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:20:24 UNRAIDSRV kernel: ? get_page_from_freelist+0x6ff/0x82d
Apr 28 09:20:24 UNRAIDSRV kernel: charge_memcg+0x29/0x71
Apr 28 09:20:24 UNRAIDSRV kernel: __mem_cgroup_charge+0x29/0x41
Apr 28 09:20:24 UNRAIDSRV kernel: __filemap_add_folio+0xb9/0x34b
Apr 28 09:20:24 UNRAIDSRV kernel: ? lruvec_page_state+0x43/0x43
Apr 28 09:20:24 UNRAIDSRV kernel: filemap_add_folio+0x37/0x91
Apr 28 09:20:24 UNRAIDSRV kernel: __filemap_get_folio+0x1a4/0x1ff
Apr 28 09:20:24 UNRAIDSRV kernel: pagecache_get_page+0x13/0x8c
Apr 28 09:20:24 UNRAIDSRV kernel: alloc_extent_buffer+0x12d/0x38b
Apr 28 09:20:24 UNRAIDSRV kernel: ? bio_add_page+0x5a/0x80
Apr 28 09:20:24 UNRAIDSRV kernel: read_tree_block+0x21/0x7f
Apr 28 09:20:24 UNRAIDSRV kernel: read_block_for_search+0x200/0x27d
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_search_slot+0x6f7/0x7c5
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_lookup_csum+0x5b/0xfd
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_lookup_bio_sums+0x1f4/0x4a2
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_submit_data_bio+0x102/0x18b
Apr 28 09:20:24 UNRAIDSRV kernel: read_pages+0x4a/0xe9
Apr 28 09:20:24 UNRAIDSRV kernel: page_cache_ra_unbounded+0x10c/0x13f
Apr 28 09:20:24 UNRAIDSRV kernel: filemap_fault+0x2e7/0x524
Apr 28 09:20:24 UNRAIDSRV kernel: __do_fault+0x30/0x6e
Apr 28 09:20:24 UNRAIDSRV kernel: __handle_mm_fault+0x9a5/0xc7d
Apr 28 09:20:24 UNRAIDSRV kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Apr 28 09:20:24 UNRAIDSRV kernel: handle_mm_fault+0x113/0x1d7
Apr 28 09:20:24 UNRAIDSRV kernel: do_user_addr_fault+0x36a/0x514
Apr 28 09:20:24 UNRAIDSRV kernel: exc_page_fault+0xfc/0x11e
Apr 28 09:20:24 UNRAIDSRV kernel: asm_exc_page_fault+0x22/0x30
Apr 28 09:20:24 UNRAIDSRV kernel: RIP: 0033:0x1505b00e22dc
Apr 28 09:20:24 UNRAIDSRV kernel: Code: 8a e3 dc ff 31 d2 83 78 44 03 0f 84 ee 04 00 00 41 0f b6 87 08 05 00 00 c1 e2 05 b9 72 00 00 00 48 89 ee 4d 8d 8f e8 00 00 00 <f3> 0f 10 05 60 29 4a 00 83 e0 df 4c 89 cf 09 d0 49 8d 97 ec 04 00
Apr 28 09:20:24 UNRAIDSRV kernel: RSP: 002b:00007fff99f13b00 EFLAGS: 00010256
Apr 28 09:20:24 UNRAIDSRV kernel: RAX: 0000000000000000 RBX: 000055bc1b1e1050 RCX: 0000000000000072
Apr 28 09:20:24 UNRAIDSRV kernel: RDX: 0000000000000000 RSI: 00007fff99f13de0 RDI: 0000000000000031
Apr 28 09:20:24 UNRAIDSRV kernel: RBP: 00007fff99f13de0 R08: 0000000000000007 R09: 000055bc8e791228
Apr 28 09:20:24 UNRAIDSRV kernel: R10: 70ace78c2583f88f R11: 0000000000000000 R12: 0000000000000000
Apr 28 09:20:24 UNRAIDSRV kernel: R13: 0000000000000000 R14: 00007fff99f13c90 R15: 000055bc8e791140
Apr 28 09:20:24 UNRAIDSRV kernel: </TASK>
Apr 28 09:20:24 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 104946
Apr 28 09:20:24 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:20:24 UNRAIDSRV kernel: kmem: usage 35964kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:20:24 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:20:24 UNRAIDSRV kernel: anon 3162849280
Apr 28 09:20:24 UNRAIDSRV kernel: file 21549056
Apr 28 09:20:24 UNRAIDSRV kernel: kernel 36827136
Apr 28 09:20:24 UNRAIDSRV kernel: kernel_stack 7700480
Apr 28 09:20:24 UNRAIDSRV kernel: pagetables 17899520
Apr 28 09:20:24 UNRAIDSRV kernel: percpu 10064
Apr 28 09:20:24 UNRAIDSRV kernel: sock 0
Apr 28 09:20:24 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:20:24 UNRAIDSRV kernel: shmem 21434368
Apr 28 09:20:24 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:20:24 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:20:24 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:20:24 UNRAIDSRV kernel: swapcached 0
Apr 28 09:20:24 UNRAIDSRV kernel: anon_thp 809500672
Apr 28 09:20:24 UNRAIDSRV kernel: file_thp 0
Apr 28 09:20:24 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:20:24 UNRAIDSRV kernel: inactive_anon 3175710720
Apr 28 09:20:24 UNRAIDSRV kernel: active_anon 8548352
Apr 28 09:20:24 UNRAIDSRV kernel: inactive_file 114688
Apr 28 09:20:24 UNRAIDSRV kernel: active_file 0
Apr 28 09:20:24 UNRAIDSRV kernel: unevictable 0
Apr 28 09:20:24 UNRAIDSRV kernel: slab_reclaimable 1274104
Apr 28 09:20:24 UNRAIDSRV kernel: slab_unreclaimable 8701640
Apr 28 09:20:24 UNRAIDSRV kernel: slab 9975744
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_refault_file 284955
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_activate_file 23324
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_restore_file 3963
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:20:24 UNRAIDSRV kernel: pgfault 375927252
Apr 28 09:20:24 UNRAIDSRV kernel: pgmajfault 1573
Apr 28 09:20:24 UNRAIDSRV kernel: pgrefill 76636
Apr 28 09:20:24 UNRAIDSRV kernel: pgscan 621258
Apr 28 09:20:24 UNRAIDSRV kernel: pgsteal 321885
Apr 28 09:20:24 UNRAIDSRV kernel: pgactivate 107798065
Apr 28 09:20:24 UNRAIDSRV kernel: pgdeactivate 76233
Apr 28 09:20:24 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:20:24 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:20:24 UNRAIDSRV kernel: thp_fault_alloc 17912
Apr 28 09:20:24 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:20:24 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:20:24 UNRAIDSRV kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:20:24 UNRAIDSRV kernel: [  29675]     0 29675       53       16    24576        0             0 s6-svscan
Apr 28 09:20:24 UNRAIDSRV kernel: [  29747]     0 29747       53       14    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29748]     0 29748       50        1    24576        0             0 s6-linux-init-s
Apr 28 09:20:24 UNRAIDSRV kernel: [  29756]     0 29756       53       16    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29757]     0 29757       53       15    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29758]     0 29758       53       16    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29759]     0 29759       53       15    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29760]     0 29760       53       15    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29761]     0 29761       53       15    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29762]     0 29762       53       16    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29763]     0 29763       53       16    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29764]     0 29764       53       16    24576        0             0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [  29773]     0 29773       47        1    24576        0             0 s6-ipcserverd
Apr 28 09:20:24 UNRAIDSRV kernel: [  29774]     0 29774      131       41    24576        0             0 s6-fdholderd
Apr 28 09:20:24 UNRAIDSRV kernel: [  29812] 65534 29812       69       27    36864        0             0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [  29813] 65534 29813       69       28    36864        0             0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [  29814] 65534 29814       70       27    36864        0             0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [  29821]     0 29821   181857     7306   143360        0             0 go2rtc
Apr 28 09:20:24 UNRAIDSRV kernel: [  29830]     0 29830      973      729    40960        0             0 bash
Apr 28 09:20:24 UNRAIDSRV kernel: [  29833]     0 29833   618176    44057   905216        0             0 python3
Apr 28 09:20:24 UNRAIDSRV kernel: [  29849]     0 29849   145420     3477   102400        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29866]     0 29866   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29867]     0 29867   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29868]     0 29868   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29875]     0 29875   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29897]     0 29897   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29932]     0 29932   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29962]     0 29962   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  29970]     0 29970   161947     1582   212992        0             0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [  30266]     0 30266   212817    19818   376832        0             0 frigate.logger
Apr 28 09:20:24 UNRAIDSRV kernel: [  30272]     0 30272     3564     2643    65536        0             0 python3
Apr 28 09:20:24 UNRAIDSRV kernel: [  30273]     0 30273   301653    26373   479232        0             0 frigate.detecto
Apr 28 09:20:24 UNRAIDSRV kernel: [  30275]     0 30275   349154    24375   524288        0             0 frigate.output
Apr 28 09:20:24 UNRAIDSRV kernel: [  30278]     0 30278   366610    25268   536576        0             0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [  30279]     0 30279    31705     5339   172032        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30280]     0 30280   374628    33601   606208        0             0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [  30284]     0 30284   371525    30248   573440        0             0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [  30285]     0 30285    31705     4858   163840        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30286]     0 30286   369682    28567   565248        0             0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [  30287]     0 30287    31705     4444   163840        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30288]     0 30288   316513    23311   495616        0             0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [  30289]     0 30289   316552    22753   475136        0             0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [  30292]     0 30292   324793    23719   466944        0             0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [  30293]     0 30293    31705     5305   163840        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30297]     0 30297   317199    25089   487424        0             0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [  30298]     0 30298   257592    29886   622592        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30303]     0 30303    31705     5332   163840        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30304]     0 30304   320153    23169   462848        0             0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [  30310]     0 30310   316439    22944   483328        0             0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [  30312]     0 30312   307946    31582   647168        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  30315]     0 30315    31705     5843   167936        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [  31302]     0 31302   306976    26775   634880        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [   3524]     0  3524    32871     6863   196608        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [   6126]     0  6126   858357   629580  5435392        0             0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [   7102]     0  7102      622      213    40960        0             0 sleep
Apr 28 09:20:24 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=6126,uid=0
Apr 28 09:20:24 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 6126 (ffmpeg) total-vm:3433428kB, anon-rss:2454116kB, file-rss:64204kB, shmem-rss:0kB, UID:0 pgtables:5308kB oom_score_adj:0
blakeblackshear commented 1 year ago

We need to try reducing the configs to the bare minimum and see where this happens when adding things back.

Just use one camera. Remove the hwaccel args. Remove the go2rtc config.

If this works without any oom issues, then add things back one at a time in this order:

  1. Add back hwaccel args
  2. Remove hwaccel args and add back go2rtc
  3. Add hwaccel args back again
  4. Add second camera

Let's see where the oom errors come up.

@johnvprcam24573455623 unless your config is identical to the one on this issue, I have no idea if these steps are relevant for you.

hibby50 commented 1 year ago

I am also having this issue. It started right after enabling the hwaccel and seems to happen randomly. Let me know if there is anything I can do to help debug.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

evilbulgarian commented 1 year ago

So some more observations:

NickM-27 commented 1 year ago

yes, this is a known issue with the currently included go2rtc 1.2.0. You can update the internal go2rtc to a newer version which many users have said fixes the issue https://docs.frigate.video/configuration/advanced#custom-go2rtc-version

hibby50 commented 1 year ago

Thank you both. I can confirm 2 of my cameras are pieces of crap (ezviz) on WiFi and have constant ffmpeg errors with and without go2rtc. For now I have just set a memory limit for my pod to prevent it from effecting my whole system. I will probably hold out until go2rtc is updated officially or in an RC.

johnvprcam24573455623 commented 1 year ago

I am wondering if we have an ETA for the latest go2rtc update in Frigate?

Thanks!

NickM-27 commented 1 year ago

It has already been updated in the dev branch, will be part of 0.13

evilbulgarian commented 1 year ago

Here are some more logs after upgrade to latest dev container image: dev-7e5abe7

2023-06-13 14:20:32.141309371 [2023-06-13 14:20:32] watchdog.front_door ERROR : Ffmpeg process crashed unexpectedly for front_door. 2023-06-13 14:20:32.141354140 [2023-06-13 14:20:32] watchdog.front_door ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2023-06-13 14:20:32.141423074 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [rtsp @ 0x55e1df52e100] RTP: PT=60: bad cseq 9055 expected=9e97 2023-06-13 14:20:32.141788835 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [segment @ 0x55e1df541640] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly 2023-06-13 14:20:32.141816981 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [out_1_0 @ 0x55e1df850480] 100 buffers queued in out_1_0, something may be wrong. 2023-06-13 14:20:32.141862378 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [out_1_0 @ 0x55e1df850480] 1000 buffers queued in out_1_0, something may be wrong.

NickM-27 commented 1 year ago

to be clear I was saying the OOM is fixed, the actual ffmpeg issues will come down to your camera

evilbulgarian commented 1 year ago

to be clear I was saying the OOM is fixed, the actual ffmpeg issues will come down to your camera

roger thank you!

What is weird is that using mpv the camera playback is perfectly smooth and fine..

❯ mpv -v rtsp://admin:xxx@spycam1.denof.sin/live
[cplayer] Command line options: '-v' 'rtsp://admin:xxx@spycam1.denof.sin/live'
[cplayer] mpv 0.35.1 Copyright © 2000-2023 mpv/MPlayer/mplayer2 projects
[cplayer]  built on UNKNOWN
[cplayer] FFmpeg library versions:
[cplayer]    libavutil       56.70.100
[cplayer]    libavcodec      58.134.100
[cplayer]    libavformat     58.76.100
[cplayer]    libswscale      5.9.100
[cplayer]    libavfilter     7.110.100
[cplayer]    libswresample   3.9.100
[cplayer] FFmpeg version: 4.4.4
[cplayer] 
[cplayer] Configuration: meson configure build -Dprefix=/usr -Dbuildtype=plain -Doptimization=plain
[cplayer] List of enabled features: alsa cdda cplayer cplugins dmabuf-interop-gl dmabuf-interop-pl dmabuf-wayland drm drm-is-kms dvdnav egl egl-drm egl-helpers egl-x11 ffmpeg gbm gl gl-wayland gl-x11 glibc-thread-name glob glob-posix gpl html-build iconv jpeg lcms2 libarchive libass libavdevice libdl libm libmpv-shared libplacebo libplacebo-next librt linux-fstatfs luajit manpage-build memfd_create noexecstack pipewire plain-gl posix pulse shaderc stdatomic threads uchardet vaapi vaapi-drm vaapi-egl vaapi-libplacebo vaapi-wayland vaapi-x-egl vaapi-x11 vector vt.h vulkan wayland wayland_protocols_1_24 x11 xv zlib
[cplayer] Built with NDEBUG.
[cplayer] Reading config file /etc/mpv/encoding-profiles.conf
[cplayer] Applying profile 'default'...
[cplayer] Reading config file /home/vladi/.config/mpv/config
[cplayer] Applying profile 'default'...
[cplayer] Setting option 'hwdec' = 'vaapi' (flags = 4)
[cplayer] Setting option 'vo' = 'gpu' (flags = 4)
[cplayer] Setting option 'gpu-context' = 'wayland' (flags = 4)
[cplayer] Setting option 'ao' = 'pulse' (flags = 4)
[cplayer] Setting option 'v' = '' (flags = 8)
[cplayer] Waiting for scripts...
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.12 (SIMPLE) HarfBuzz-ng 7.3.0 (COMPLEX)
[osd/libass] Setting up fonts...
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Done loading scripts.
[cplayer] Running hook: ytdl_hook/on_load
[ytdl_hook] ytdl:// hook 
[ytdl_hook] not a ytdl:// url 
[ffmpeg] Opening rtsp://admin:xxx@spycam1.denof.sin/live
[cplayer] Set property: shared-script-properties -> 1
[demux] Trying demuxers for level=request.
[cplayer] Set property: shared-script-properties -> 1
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.12 (SIMPLE) HarfBuzz-ng 7.3.0 (COMPLEX)
[osd/libass] Setting up fonts...
[lavf] Found 'rtsp' at score=100 size=0 (forced).
[lavf] Broken FFmpeg RTSP API => not setting timeout.
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[lavf] Could not set AVOption tls_verify='0'
[lavf] Could not set AVOption icy='1'
[lavf] avformat_find_stream_info() finished after 0 bytes.
[demux] Detected file format: rtsp (libavformat)
[demux] Stream is not seekable.
[cplayer] Opening done: rtsp://admin:xxxx@spycam1.denof.sin/live
[cplayer] Running hook: ytdl_hook/on_preloaded
[lavf] select track 0
[lavf] select track 1
[cplayer]  (+) Video --vid=1 (h264 1920x1080 20.000fps)
[cplayer]  (+) Audio --aid=1 (pcm_alaw 1ch 16000Hz)
[display-tags] File tags:
[display-tags]  Comment: live
[display-tags]  Title: Session streamed by the WYZE Media Server
[vo/gpu/opengl] Initializing GPU context 'wayland'
[vo/gpu/wayland] Registered for protocol wl_shm
[vo/gpu/wayland] Registered for protocol zwp_linux_dmabuf_v1
[vo/gpu/wayland] Registered for protocol wl_compositor
[vo/gpu/wayland] Registered for protocol wl_subcompositor
[vo/gpu/wayland] Registered for protocol wl_data_device_manager
[vo/gpu/wayland] Registered for protocol wp_viewporter
[vo/gpu/wayland] Registered for protocol xdg_wm_base
[vo/gpu/wayland] Registered for protocol wl_seat
[vo/gpu/wayland] Registered for protocol wp_presentation
[vo/gpu/wayland] Registered for protocol zxdg_decoration_manager_v1
[vo/gpu/wayland] Registered for protocol zwp_idle_inhibit_manager_v1
[vo/gpu/wayland] Registered for protocol wl_output
[vo/gpu/wayland] Registered for protocol wl_output
[vo/gpu/wayland] Registered output Acer Technologies VG271U (0x31):
[vo/gpu/wayland]    x: 0px, y: 0px
[vo/gpu/wayland]    w: 2560px (600mm), h: 1440px (340mm)
[vo/gpu/wayland]    scale: 1
[vo/gpu/wayland]    Hz: 119.998000
[vo/gpu/wayland] Registered output Ancor Communications Inc ASUS MG279 (0x32):
[vo/gpu/wayland]    x: 0px, y: 0px
[vo/gpu/wayland]    w: 2560px (600mm), h: 1440px (340mm)
[vo/gpu/wayland]    scale: 1
[vo/gpu/wayland]    Hz: 119.878000
[vo/gpu/wayland] Enabling server decorations
[vo/gpu/opengl] EGL_VERSION=1.5
[vo/gpu/opengl] EGL_VENDOR=Mesa Project
[vo/gpu/opengl] EGL_CLIENT_APIS=OpenGL OpenGL_ES 
[vo/gpu/opengl] Trying to create Desktop OpenGL context.
[vo/gpu/wayland] GL_VERSION='4.6 (Core Profile) Mesa 23.0.3'
[vo/gpu/wayland] Detected desktop OpenGL 4.6.
[vo/gpu/wayland] GL_VENDOR='AMD'
[vo/gpu/wayland] GL_RENDERER='AMD Radeon RX 5700 XT (navi10, LLVM 15.0.7, DRM 3.52, 6.3.7)'
[vo/gpu/wayland] GL_SHADING_LANGUAGE_VERSION='4.60'
[vo/gpu/wayland] Loaded extension GL_KHR_debug.
[vo/gpu] Testing FBO format rgba16f
[vo/gpu] Using FBO format rgba16f.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vd] Container reported FPS: 20.000000
[vd] Codec list:
[vd]     h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd]     h264_v4l2m2m (h264) - V4L2 mem2mem H.264 decoder wrapper
[vd] Opening decoder h264
[vd] Looking at hwdec h264-vaapi...
[vo/gpu] Loading hwdec drivers for format: 'vaapi'
[vo/gpu] Loading hwdec driver 'vaapi'
[vo/gpu/vaapi] using EGL dmabuf interop
[vo/gpu/vaapi] Trying to open a x11 VA display...
[vo/gpu/vaapi] Trying to open a wayland VA display...
[vo/gpu/vaapi/vaapi] Initialized VAAPI: version 1.18
[vo/gpu/vaapi] Going to probe surface formats (may log bogus errors)...
[vo/gpu/vaapi] Done probing surface formats.
[vd] Trying hardware decoding via h264-vaapi.
[vd] Selected codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
[vf] User filter list:
[vf]   (empty)
[ad] Codec list:
[ad]     pcm_alaw - PCM A-law / G.711 A-law
[ad] Opening decoder pcm_alaw
[ad] Requesting 1 threads for decoding.
[ad] Selected codec: pcm_alaw (PCM A-law / G.711 A-law)
[af] User filter list:
[af]   (empty)
[cplayer] Starting playback...
[vd] Pixel formats supported by decoder: vaapi_vld yuv420p
[vd] Codec profile: Main (0x4d)
[vd] Requesting pixfmt 'vaapi_vld' from decoder.
[vd] Using hardware decoding (vaapi).
[vd] Decoder format: 1920x1080 [0:1] vaapi[nv12] bt.709/bt.709/bt.1886/limited/auto CL=mpeg2/4/h264
[vf] [in] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] (disabled)
[vf] [autorotate] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [autorotate] (disabled)
[vf] [convert] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [convert] (disabled)
[vf] [out] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[af] [in] 16000Hz mono 1ch s16
[af] [userspeed] 16000Hz mono 1ch s16
[af] [userspeed] (disabled)
[af] [convert] 16000Hz mono 1ch s16
[ao] Trying audio driver 'pulse'
[ao/pulse] requested format: 16000 Hz, mono channels, s16
[ao/pulse] Library version: 16.1.0
[ao/pulse] Proto: 35
[ao/pulse] Server proto: 4294967295
[ao/pulse] Channel layouts:
[ao/pulse]  - #fl
[ao/pulse]  - #fr
[ao/pulse]  - #fc
[ao/pulse]  - #lfe
[ao/pulse]  - #bl
[ao/pulse]  - #br
[ao/pulse]  - #flc
[ao/pulse]  - #frc
[ao/pulse]  - #bc
[ao/pulse]  - #sl
[ao/pulse]  - #sr
[ao/pulse]  - #tc
[ao/pulse]  - #tfl
[ao/pulse]  - #tfc
[ao/pulse]  - #tfr
[ao/pulse]  - #tbl
[ao/pulse]  - #tbc
[ao/pulse]  - #tbr
[ao/pulse] result: mono
[ao/pulse] device buffer: 1600 samples.
[ao/pulse] using soft-buffer of 3200 samples.
[cplayer] AO: [pulse] 16000Hz mono 1ch s16
[cplayer] AO: Description: PulseAudio audio output
[af] [convert] (disabled)
[af] [out] 16000Hz mono 1ch s16
[cplayer] VO: [gpu] 1920x1080 vaapi[nv12]
[cplayer] VO: Description: Shader-based GPU Renderer
[vo/gpu] reconfig to 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vo/gpu/wayland] Reconfiguring!
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Testing FBO format rgba16f
[vo/gpu] Using FBO format rgba16f.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vo/gpu/wayland] Handling resize on the egl side
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Reported display depth: 10
[vo/gpu/wayland] Given DND offer with mime type text/plain;charset=utf-8
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[vo/gpu/wayland] Surface entered output Acer Technologies VG271U (0x31), scale = 1, refresh rate = 119.998000 Hz
[vo/gpu/wayland] Resizing due to xdg from 1920x1080 to 826x678
[cplayer] first video frame after restart shown
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Reported display depth: 10
[vo/gpu/wayland] Handling resize on the egl side
[vo/gpu] Resize: 826x678
[vo/gpu] Window size: 826x678 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 107) 826x464
[vo/gpu] Video scale: 0.430208/0.429630
[vo/gpu] OSD borders: l=0 t=107 r=0 b=107
[vo/gpu] Video borders: l=0 t=107 r=0 b=107
[vo/gpu] Reported display depth: 10
[vo/gpu] Assuming 119.998000 FPS for display sync.
[cplayer] audio ready
[cplayer] starting audio playback
[cplayer] playback restart complete @ 1.851000, audio=playing, video=playing
[vo/gpu/wayland] Enabling idle inhibitor
[ao/pulse] starting AO
[statusline] AV: 00:00:01 / 00:00:03 (58%) A-V:  0.003
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: shared-script-properties -> 1
[statusline] AV: 00:00:07 / 00:00:08 (84%) A-V: -0.005
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V:  0.000
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V:  0.000
[cplayer] Set property: shared-script-properties -> 1
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V:  0.000
[osd/libass] fontselect: (sans-serif, 400, 0) -> /usr/share/fonts/noto/NotoSans-Regular.ttf, 0, NotoSans-Regular
[osd/libass] fontselect: (mpv-osd-symbols, 400, 0) -> mpv-osd-symbols-Regular, 0, mpv-osd-symbols-Regular
[vo/gpu] Reallocating OSD texture to 1024x256.
[statusline] AV: 00:00:40 / 00:00:42 (97%) A-V: -0.023
github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.