search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.88k stars 1.72k forks source link

[Support]: watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [frigate.output:1243199] #6360

Closed erbain closed 1 year ago

erbain commented 1 year ago

Describe the problem you are having

I've been running Frigate for over a year now with no major issues (moved from ZoneMinder, you guys have done an amazing job and it keeps getting better), upgraded last week to v12 which seemed to go OK with everything still working. Woke up this morning to find the server where Frigate was running completely locked up. After a reboot the syslog shows the log output below being repeated about every 20s.

No idea where to start debugging this, any guidance you can provide would be greatly appreciated.

Version

0.12.0-DA3E197

Frigate config file

mqtt:
  host: 10.10.10.4
  port: 1883

detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: True
  retain:
    days: 7
    mode: all
  events:
    post_capture: 10
    pre_capture: 10
    retain:
      default: 21
      mode: active_objects

snapshots:
  enabled: True
  bounding_box: True
  retain:
    default: 20

birdseye:
  enabled: True
  width: 1920
  height: 1080
  quality: 3
  mode: continuous

ffmpeg:
  hwaccel_args: -c:v h264_cuvid
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac

go2rtc:
  streams:
    drive1:
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.124:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
    gate1:
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.243:554/ch1/main/av_stream
    greenhouse:
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.125:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
    garage-side:
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.122:554/ROH/channel/11
    back-garden1: 
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.165:554/H264/ch1/main/av_stream
    garage-rear: 
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.126:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
    garage-inside: 
      - rtsp://xxxxxxxx:xxxxxxxx@10.10.10.32:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
    playpark: 
      - http://10.10.10.190/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=xxxxxxxx
      - "ffmpeg:rtsp_cam#audio=opus"

objects:
  track:
    - person
    - bicycle
    - motorcycle
    - car
    - dog
    - cat
  filters:
    person:
      min_area: 5000
      max_area: 200000
      min_ratio: 0.3
      max_ratio: 2.5
      #threshold: 0.78

cameras:
  gate1:
    ui:
      order: 1
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.243:554/ch1/main/av_stream
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080
    snapshots:
      required_zones:
        - gate
    record:
      events:
        required_zones:
          - gate
    objects:
      filters:
        person:
          min_area: 13000
    motion:
      mask:
        - 751,519,950,540,1920,281,1920,0,0,0,0,622
    zones:
      gate:
        coordinates: 284,1080,585,938,984,747,1108,500,1920,423,1920,1080

  drive1: 
    ui:
      order: 10 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.124:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080

  greenhouse: 
    ui:
      order: 20 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.125:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
          roles:
            - detect
            - rtmp
    detect:
      width: 1920
      height: 1080
    objects:
      filters:
        person:
          min_area: 3000
          max_area: 80000
          threshold: 0.75
    motion:
      #threshold: 20 
      #contour_area: 30
      mask:
        - 804,846,995,843,1014,617,798,585
        - 1920,584,1272,343,1232,545,1920,763
        - 216,292,92,301,0,326,0,0,535,0,616,237
        - 1289,0,1025,0,1180,84,1373,110,1920,163,1920,0

  garage-side:
    ui:
      order: 60 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.122:554/ROH/channel/11
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080
    motion:
      mask:
        - 501,93,502,48,31,48,33,95
        - 447,0,456,132,1380,591,1387,550,1446,0
    zones:
      garden:
        coordinates: 0,1080,0,107,302,148,336,398,784,1080,784,1080
    snapshots:
      required_zones:
        - garden
    record:
      events:
        required_zones:
          - garden

  back-garden1: 
    ui:
      order: 30 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.165:554/H264/ch1/main/av_stream
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080
    objects:
      filters:
        person:
          min_area: 10000
          max_area: 100000
          threshold: 0.78

  garage-rear: 
    ui:
      order: 50 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.126:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080
    zones:
      garden:
        coordinates: 0,1080,0,97,859,97,1316,1080
    snapshots:
      required_zones:
        - garden
    record:
      events:
        required_zones:
          - garden

  garage-inside: 
    ui:
      order: 40 
    ffmpeg:
      inputs:
        - path: rtsp://xxxxxxxx:xxxxxxxx@10.10.10.32:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - record
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - person
      filters:
        person:
          max_area: 450000
          mask:
            - 422,848,677,766,653,630,386,702

  playpark: 
    ui:
      order: 15 
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - +genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      inputs:
        - path: http://10.10.10.190/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=xxxxxxxx
          roles:
            - record
            - rtmp
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      filters:
        person:
          min_area: 10000
          max_area: 100000

Relevant log output

May  3 06:24:16 wendy kernel: [1588862.292994] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [frigate.output:1243199]
May  3 06:24:16 wendy kernel: [1588862.293383] Modules linked in: rpcsec_gss_krb5 xt_nat veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter aufs overlay bridge stp llc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) snd_hda_codec_hdmi zcommon(PO) znvpair(PO) spl(O) intel_rapl_msr ppdev mei_hdcp cdc_acm binfmt_misc intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm rapl intel_cstate mxm_wmi snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer mei_me snd mei soundcore nvidia_uvm(OE) parport_pc parport mac_hid intel_smartconnect sch_fq_codel msr ramoops reed_solomon nfsd efi_pstore auth_rpcgss nfs_acl lockd grace sunrpc
May  3 06:24:16 wendy kernel: [1588862.293402]  ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear nvidia_drm(POE) nvidia_modeset(POE) uas usb_storage nvidia(POE) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel drm_kms_helper crypto_simd syscopyarea cryptd glue_helper sysfillrect sysimgblt ahci fb_sys_fops libahci i2c_i801 r8169 drm lpc_ich realtek wmi video
May  3 06:24:16 wendy kernel: [1588862.299984] CPU: 0 PID: 1243199 Comm: frigate.output Tainted: P      D W  OEL    5.4.0-144-generic #161-Ubuntu
May  3 06:24:16 wendy kernel: [1588862.300942] Hardware name: MSI MS-7816/Z87-G43 (MS-7816), BIOS V1.11 05/09/2015
May  3 06:24:16 wendy kernel: [1588862.301472] RIP: 0010:xas_find+0x17a/0x1c0
May  3 06:24:16 wendy kernel: [1588862.301972] Code: 0f 84 0b ff ff ff 8d 70 ff 83 e6 3f 83 c6 01 e9 fd fe ff ff 48 89 f8 48 c1 f8 02 85 c0 0f 84 b7 fe ff ff 31 c0 eb 96 48 89 df <e8> 01 fb ff ff 48 85 c0 0f 85 4d ff ff ff 48 8b 7b 18 40 f6 c7 03
May  3 06:24:16 wendy kernel: [1588862.303547] RSP: 0018:ffffbfc8ca12bb90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
May  3 06:24:16 wendy kernel: [1588862.304119] RAX: 0000000000000201 RBX: ffffbfc8ca12bbb8 RCX: 0000000000000000
May  3 06:24:16 wendy kernel: [1588862.304681] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: ffffbfc8ca12bbb8
May  3 06:24:16 wendy kernel: [1588862.305256] RBP: ffffbfc8ca12bba0 R08: ffffbfc8ca12bc88 R09: 0000000002e63ef5
May  3 06:24:16 wendy kernel: [1588862.305815] R10: 000000000000b98c R11: 0000000000000000 R12: ffffffffffffffff
May  3 06:24:16 wendy kernel: [1588862.306381] R13: ffffbfc8ca12bc88 R14: 000000000000000f R15: 0000000000000000
May  3 06:24:16 wendy kernel: [1588862.306944] FS:  00007f0315311740(0000) GS:ffffa0b0bea00000(0000) knlGS:0000000000000000
May  3 06:24:16 wendy kernel: [1588862.307508] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  3 06:24:16 wendy kernel: [1588862.308090] CR2: 00007f031540d92d CR3: 000000053bfdc002 CR4: 00000000001626f0
May  3 06:24:16 wendy kernel: [1588862.308660] Call Trace:
May  3 06:24:16 wendy kernel: [1588862.309261]  find_get_entries+0x1e6/0x270
May  3 06:24:16 wendy kernel: [1588862.309832]  shmem_undo_range+0x2b2/0x930
May  3 06:24:16 wendy kernel: [1588862.310393]  shmem_truncate_range+0x14/0x40
May  3 06:24:16 wendy kernel: [1588862.310944]  shmem_evict_inode+0xf3/0x260
May  3 06:24:16 wendy kernel: [1588862.311484]  ? var_wake_function+0x30/0x30
May  3 06:24:16 wendy kernel: [1588862.312016]  evict+0xd2/0x1b0
May  3 06:24:16 wendy kernel: [1588862.312532]  iput+0x148/0x210
May  3 06:24:16 wendy kernel: [1588862.313063]  do_unlinkat+0x1c5/0x2d0
May  3 06:24:16 wendy kernel: [1588862.313555]  __x64_sys_unlink+0x23/0x30
May  3 06:24:16 wendy kernel: [1588862.314028]  do_syscall_64+0x57/0x190
May  3 06:24:16 wendy kernel: [1588862.314494]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
May  3 06:24:16 wendy kernel: [1588862.314952] RIP: 0033:0x7f031540d957
May  3 06:24:16 wendy kernel: [1588862.315393] Code: Bad RIP value.
May  3 06:24:16 wendy kernel: [1588862.315822] RSP: 002b:00007ffce46c01e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057
May  3 06:24:16 wendy kernel: [1588862.316258] RAX: ffffffffffffffda RBX: 00007ffce46c01f0 RCX: 00007f031540d957
May  3 06:24:16 wendy kernel: [1588862.316684] RDX: 0000000000000019 RSI: 000000000000002f RDI: 00007ffce46c01f0
May  3 06:24:16 wendy kernel: [1588862.317132] RBP: 00007f02f8f76e70 R08: 00007f02f8f78880 R09: 0000000000000000
May  3 06:24:16 wendy kernel: [1588862.317536] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000fad690
May  3 06:24:16 wendy kernel: [1588862.317937] R13: 00007f02f8f76e40 R14: 00007f0307242950 R15: 0000000000fad690

FFprobe output from your camera

N/A

Frigate stats

{"back-garden1":{"camera_fps":5.0,"capture_pid":322,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":342,"pid":301,"process_fps":5.0,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.2"},"1":{"cpu":"0.0","mem":"0.0"},"108":{"cpu":"0.0","mem":"0.0"},"109":{"cpu":"2.7","mem":"1.1"},"117":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.1"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.1"},"137":{"cpu":"0.0","mem":"0.1"},"15":{"cpu":"0.0","mem":"0.0"},"17695":{"cpu":"0.0","mem":"0.0"},"17738":{"cpu":"0.0","mem":"0.0"},"17745":{"cpu":"0.3","mem":"0.0"},"18":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"281":{"cpu":"0.0","mem":"0.2"},"287":{"cpu":"0.7","mem":"0.0"},"288":{"cpu":"5.3","mem":"0.4"},"289":{"cpu":"2.3","mem":"0.4"},"29":{"cpu":"0.0","mem":"0.0"},"292":{"cpu":"0.3","mem":"0.3"},"293":{"cpu":"0.3","mem":"0.3"},"296":{"cpu":"0.7","mem":"0.4"},"297":{"cpu":"2.0","mem":"0.4"},"298":{"cpu":"0.0","mem":"0.0"},"299":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"301":{"cpu":"0.7","mem":"0.4"},"302":{"cpu":"0.0","mem":"0.0"},"304":{"cpu":"5.7","mem":"0.4"},"305":{"cpu":"0.7","mem":"0.3"},"306":{"cpu":"0.3","mem":"0.4"},"307":{"cpu":"0.0","mem":"0.0"},"308":{"cpu":"2.0","mem":"0.3"},"309":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"310":{"cpu":"2.0","mem":"0.3"},"311":{"cpu":"2.0","mem":"0.3"},"313":{"cpu":"0.0","mem":"0.0"},"316":{"cpu":"2.0","mem":"0.3"},"317":{"cpu":"16.0","mem":"0.4"},"318":{"cpu":"0.0","mem":"0.1"},"32":{"cpu":"0.0","mem":"0.0"},"322":{"cpu":"1.7","mem":"0.3"},"323":{"cpu":"9.3","mem":"0.4"},"326":{"cpu":"19.0","mem":"0.4"},"328":{"cpu":"2.0","mem":"0.3"},"331":{"cpu":"0.0","mem":"0.1"},"333":{"cpu":"5.7","mem":"0.3"},"334":{"cpu":"0.0","mem":"0.2"},"336":{"cpu":"2.0","mem":"0.3"},"337":{"cpu":"2.0","mem":"0.3"},"342":{"cpu":"15.3","mem":"0.4"},"346":{"cpu":"14.7","mem":"0.4"},"349":{"cpu":"15.0","mem":"0.4"},"362":{"cpu":"9.3","mem":"0.4"},"40":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"77":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"97":{"cpu":"0.0","mem":"0.1"},"MiB":{"cpu":"24789.4","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"average:","mem":"2.57,"}},"detection_fps":25.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":8.65,"pid":288}},"drive1":{"camera_fps":5.0,"capture_pid":310,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":317,"pid":293,"process_fps":5.0,"skipped_fps":0.0},"garage-inside":{"camera_fps":5.1,"capture_pid":336,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":349,"pid":305,"process_fps":5.1,"skipped_fps":0.0},"garage-rear":{"camera_fps":5.0,"capture_pid":328,"detection_enabled":1,"detection_fps":20.0,"ffmpeg_pid":362,"pid":304,"process_fps":5.0,"skipped_fps":0.0},"garage-side":{"camera_fps":5.1,"capture_pid":316,"detection_enabled":1,"detection_fps":5.0,"ffmpeg_pid":333,"pid":297,"process_fps":5.1,"skipped_fps":0.0},"gate1":{"camera_fps":5.1,"capture_pid":308,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":326,"pid":292,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"NVIDIA GeForce GTX 750 Ti":{"gpu":"12 %","mem":"45.4 %"}},"greenhouse":{"camera_fps":5.1,"capture_pid":311,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":323,"pid":296,"process_fps":5.1,"skipped_fps":0.0},"playpark":{"camera_fps":5.0,"capture_pid":337,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":346,"pid":306,"process_fps":5.0,"skipped_fps":0.0},"service":{"last_updated":1683102390,"latest_version":"0.12.0","storage":{"/dev/shm":{"free":285.6,"mount_type":"tmpfs","total":314.6,"used":28.9},"/media/frigate/clips":{"free":1180310.3,"mount_type":"ext4","total":3936819.7,"used":2556453.3},"/media/frigate/recordings":{"free":1180310.3,"mount_type":"ext4","total":3936819.7,"used":2556453.3},"/tmp/cache":{"free":957.9,"mount_type":"tmpfs","total":1000.0,"used":42.1}},"temperatures":{},"uptime":7622,"version":"0.12.0-da3e197"}}

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Mixed

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

Do you have anything watching jsmpeg streams from a camera or birdseye continuously?

erbain commented 1 year ago

Usually I have birdseye open 24/7 on my laptop, last night I had taken my laptop out to a meeting and had not restarted the stream when I got home. So usually yes, but when the error occurred nothing was watching.

NickM-27 commented 1 year ago

Interesting, maybe try enabling debug logs for output

logger:
  logs:
    frigate.output: debug
erbain commented 1 year ago

I've just done a bit more digging in the logs, looks like the "BUG: soft lockup - CPU#0 stuck for 23s! [frigate.output:1243199]" started last night around 23:00, these continue to around 4:23 this morning when the machine ran out of memory and the OOM killer started slaughtering processes.

I'll enable debugging for frigate as you suggested and report back, thanks for taking the time to look at this.

Syslog around the time Frigate stopped recording but the errors continued:

May  3 04:35:53 wendy kernel: [1582359.031035] frigate.detecto invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=2, oom_score_adj=0
May  3 04:35:53 wendy kernel: [1582359.031870] CPU: 1 PID: 1243397 Comm: frigate.detecto Tainted: P      D W  OEL    5.4.0-144-generic #161-Ubuntu
May  3 04:35:53 wendy kernel: [1582359.032720] Hardware name: MSI MS-7816/Z87-G43 (MS-7816), BIOS V1.11 05/09/2015
May  3 04:35:53 wendy kernel: [1582359.033141] Call Trace:
May  3 04:35:53 wendy kernel: [1582359.033531]  dump_stack+0x6d/0x8b
May  3 04:35:53 wendy kernel: [1582359.033909]  dump_header+0x4f/0x1eb
May  3 04:35:53 wendy kernel: [1582359.034268]  oom_kill_process.cold+0xb/0x10
May  3 04:35:53 wendy kernel: [1582359.034620]  out_of_memory+0x1cf/0x500
May  3 04:35:53 wendy kernel: [1582359.034959]  __alloc_pages_slowpath+0xdde/0xeb0
May  3 04:35:53 wendy kernel: [1582359.035315]  __alloc_pages_nodemask+0x2d0/0x320
May  3 04:35:53 wendy kernel: [1582359.035639]  alloc_pages_current+0x87/0xe0
May  3 04:35:53 wendy kernel: [1582359.035966]  kmalloc_order+0x1f/0x80
May  3 04:35:53 wendy kernel: [1582359.036283]  kmalloc_order_trace+0x24/0xa0
May  3 04:35:53 wendy kernel: [1582359.036598]  __kmalloc+0x22b/0x290
May  3 04:35:53 wendy kernel: [1582359.036904]  proc_do_submiturb+0x367/0xce0
May  3 04:35:53 wendy kernel: [1582359.037217]  usbdev_do_ioctl+0x360/0x1010
May  3 04:35:53 wendy kernel: [1582359.037529]  ? futex_wake+0x8b/0x180
May  3 04:35:53 wendy kernel: [1582359.037837]  usbdev_ioctl+0xe/0x20
May  3 04:35:53 wendy kernel: [1582359.038149]  do_vfs_ioctl+0x407/0x670
May  3 04:35:53 wendy kernel: [1582359.038457]  ? __audit_syscall_entry+0xdb/0x120
May  3 04:35:53 wendy kernel: [1582359.038789]  ksys_ioctl+0x67/0x90
May  3 04:35:53 wendy kernel: [1582359.039120]  __x64_sys_ioctl+0x1a/0x20
May  3 04:35:53 wendy kernel: [1582359.039432]  do_syscall_64+0x57/0x190
May  3 04:35:53 wendy kernel: [1582359.039745]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
May  3 04:35:53 wendy kernel: [1582359.040065] RIP: 0033:0x7f0315411bab
May  3 04:35:53 wendy kernel: [1582359.040385] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
May  3 04:35:53 wendy kernel: [1582359.041422] RSP: 002b:00007f030fff8c30 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May  3 04:35:53 wendy kernel: [1582359.041778] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0315411bab
May  3 04:35:53 wendy kernel: [1582359.042132] RDX: 00007f02e401fe50 RSI: 000000008038550a RDI: 0000000000000055
May  3 04:35:53 wendy kernel: [1582359.042488] RBP: 00007f02e401fe50 R08: 00007f02e4000040 R09: 00000007f02e401f
May  3 04:35:53 wendy kernel: [1582359.042845] R10: 00007ffce46e0090 R11: 0000000000000246 R12: 0000000000000000
May  3 04:35:53 wendy kernel: [1582359.043219] R13: 00000000023033f0 R14: 0000000000000001 R15: 00007f02e4021fc0
May  3 04:35:53 wendy kernel: [1582359.043585] Mem-Info:
May  3 04:35:53 wendy kernel: [1582359.043946] active_anon:394912 inactive_anon:174785 isolated_anon:8
May  3 04:35:53 wendy kernel: [1582359.043946]  active_file:156750 inactive_file:2170413 isolated_file:0
May  3 04:35:53 wendy kernel: [1582359.043946]  unevictable:4776 dirty:25057 writeback:0 unstable:0
May  3 04:35:53 wendy kernel: [1582359.043946]  slab_reclaimable:3985049 slab_unreclaimable:785563
May  3 04:35:53 wendy kernel: [1582359.043946]  mapped:182185 shmem:47494 pagetables:7132 bounce:0
May  3 04:35:53 wendy kernel: [1582359.043946]  free:56843 free_pcp:1832 free_cma:0
May  3 04:35:53 wendy kernel: [1582359.046031] Node 0 active_anon:1579648kB inactive_anon:699140kB active_file:627000kB inactive_file:8681652kB unevictable:19104kB isolated(anon):32kB isolated(file):0kB mapped:728740kB dirty:100448kB writeback:0kB shmem:189976kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 249856kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
May  3 04:35:53 wendy kernel: [1582359.047603] Node 0 DMA free:15884kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May  3 04:35:53 wendy kernel: [1582359.048826] lowmem_reserve[]: 0 3292 31862 31862 31862
May  3 04:35:53 wendy kernel: [1582359.049284] Node 0 DMA32 free:124916kB min:6976kB low:10344kB high:13712kB active_anon:110196kB inactive_anon:92168kB active_file:11612kB inactive_file:193492kB unevictable:0kB writepending:17804kB present:3613072kB managed:3547332kB mlocked:0kB kernel_stack:0kB pagetables:36kB bounce:0kB free_pcp:4340kB local_pcp:0kB free_cma:0kB
May  3 04:35:53 wendy kernel: [1582359.051145] lowmem_reserve[]: 0 0 28570 28570 28570
May  3 04:35:53 wendy kernel: [1582359.051639] Node 0 Normal free:86572kB min:60568kB low:89824kB high:119080kB active_anon:1470760kB inactive_anon:606704kB active_file:615628kB inactive_file:8488344kB unevictable:19104kB writepending:82424kB present:29868032kB managed:29263712kB mlocked:19104kB kernel_stack:16848kB pagetables:28492kB bounce:0kB free_pcp:3144kB local_pcp:16kB free_cma:0kB
May  3 04:35:53 wendy kernel: [1582359.053628] lowmem_reserve[]: 0 0 0 0 0
May  3 04:35:53 wendy kernel: [1582359.054266] Node 0 DMA: 1*4kB (U) 1*8kB (U) 2*16kB (U) 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15884kB
May  3 04:35:53 wendy kernel: [1582359.055462] Node 0 DMA32: 4334*4kB (UE) 13326*8kB (UE) 1*16kB (H) 0*32kB 0*64kB 2*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 125496kB
May  3 04:35:53 wendy kernel: [1582359.056594] Node 0 Normal: 1384*4kB (UMEH) 10128*8kB (UMEH) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 86560kB
May  3 04:35:53 wendy kernel: [1582359.057663] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
May  3 04:35:53 wendy kernel: [1582359.058253] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May  3 04:35:53 wendy kernel: [1582359.058876] 2381410 total pagecache pages
May  3 04:35:53 wendy kernel: [1582359.059504] 6454 pages in swap cache
May  3 04:35:53 wendy kernel: [1582359.060099] Swap cache stats: add 2220954, delete 2214747, find 4479023/4802311
May  3 04:35:53 wendy kernel: [1582359.060709] Free swap  = 3423192kB
May  3 04:35:53 wendy kernel: [1582359.061306] Total swap = 4194300kB
May  3 04:35:53 wendy kernel: [1582359.061874] 8374273 pages RAM
May  3 04:35:53 wendy kernel: [1582359.062470] 0 pages HighMem/MovableOnly
May  3 04:35:53 wendy kernel: [1582359.063069] 167541 pages reserved
May  3 04:35:53 wendy kernel: [1582359.063641] 0 pages cma reserved
May  3 04:35:53 wendy kernel: [1582359.064162] 0 pages hwpoisoned
May  3 04:35:53 wendy kernel: [1582359.064695] Tasks state (memory values in pages):
May  3 04:35:53 wendy kernel: [1582359.065220] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
May  3 04:35:53 wendy kernel: [1582359.065744] [    459]     0   459    49921    18019   413696        0          -250 systemd-journal
May  3 04:35:53 wendy kernel: [1582359.066776] [    488]     0   488     1346       42    49152        0             0 blkmapd
May  3 04:35:53 wendy kernel: [1582359.067303] [    496]     0   496     6008     1484    73728        0         -1000 systemd-udevd
May  3 04:35:53 wendy kernel: [1582359.068308] [    948]     0   948    70116     4634    98304        0         -1000 multipathd
May  3 04:35:53 wendy kernel: [1582359.068786] [   1439]     0  1439      849       50    45056        0             0 rpc.idmapd
May  3 04:35:53 wendy kernel: [1582359.069268] [   1467]     0  1467     2894      420    49152        0         -1000 auditd
May  3 04:35:53 wendy kernel: [1582359.069722] [   1471]   124  1471     1775      868    53248        0             0 rpcbind
May  3 04:35:53 wendy kernel: [1582359.070168] [   1529]   100  1529     4799     1505    77824        0             0 systemd-network
May  3 04:35:53 wendy kernel: [1582359.071103] [   1582]   101  1582     6138     2338    98304        0             0 systemd-resolve
May  3 04:35:53 wendy kernel: [1582359.072007] [   1606]     0  1606    59089     1592    90112        0             0 accounts-daemon
May  3 04:35:53 wendy kernel: [1582359.072827] [   1607]   131  1607     2275      970    65536        0             0 avahi-daemon
May  3 04:35:53 wendy kernel: [1582359.073632] [   1608]   103  1608     2199     1279    61440        0          -900 dbus-daemon
May  3 04:35:53 wendy kernel: [1582359.074030] [   1609]     0  1609    64916     3593   143360        0             0 NetworkManager
May  3 04:35:53 wendy kernel: [1582359.074782] [   1617]     0  1617    20471      807    61440        0             0 irqbalance
May  3 04:35:53 wendy kernel: [1582359.075209] [   1619]     0  1619   111665      594    90112        0             0 lxcfs
May  3 04:35:53 wendy kernel: [1582359.075615] [   1620]     0  1620     9339     3930   114688        0             0 networkd-dispat
May  3 04:35:53 wendy kernel: [1582359.076312] [   1628]     0  1628     1290      476    53248        0             0 nvidia-persiste
May  3 04:35:53 wendy kernel: [1582359.077051] [   1633]     0  1633    58847     1983    90112        0             0 polkitd
May  3 04:35:53 wendy kernel: [1582359.077411] [   1637]   102  1637    56135     1181    73728        0             0 rsyslogd
May  3 04:35:53 wendy kernel: [1582359.077765] [   1641]     0  1641     2855     1444    61440        0             0 smartd
May  3 04:35:53 wendy kernel: [1582359.078119] [   1643]     0  1643   311275     4722   311296     1401          -900 snapd
May  3 04:35:53 wendy kernel: [1582359.078475] [   1646]     0  1646    58249     1056    77824        0             0 switcheroo-cont
May  3 04:35:53 wendy kernel: [1582359.079227] [   1649]     0  1649     4431     1605    73728        0             0 systemd-logind
May  3 04:35:53 wendy kernel: [1582359.079990] [   1651]     0  1651     4859     1850    73728        0             0 systemd-machine
May  3 04:35:53 wendy kernel: [1582359.080749] [   1652]     0  1652    30915     1683    94208        0             0 thermald
May  3 04:35:53 wendy kernel: [1582359.081122] [   1653]     0  1653    98477     2959   122880        0             0 udisksd
May  3 04:35:53 wendy kernel: [1582359.081491] [   1657]     0  1657     3422     1024    65536        0             0 wpa_supplicant
May  3 04:35:53 wendy kernel: [1582359.082248] [   1658]     0  1658    41022     1141    77824        0             0 zed
May  3 04:35:53 wendy kernel: [1582359.082614] [   1662]   131  1662     2140       89    57344        0             0 avahi-daemon
May  3 04:35:53 wendy kernel: [1582359.083369] [   1698]   114  1698   245435    11473   409600      922             0 mongod
May  3 04:35:53 wendy kernel: [1582359.083773] [   1712]     0  1712   376164     4027   311296        1          -999 containerd
May  3 04:35:53 wendy kernel: [1582359.084145] [   1726]     0  1726    78778     2430   102400        0             0 ModemManager
May  3 04:35:53 wendy kernel: [1582359.084876] [   1744]   136  1744    18720      902    57344        0             0 ntpd
May  3 04:35:53 wendy kernel: [1582359.085249] [   1752]     0  1752     8407     2700   106496     4555             0 rpc.mountd
May  3 04:35:53 wendy kernel: [1582359.085618] [   1800]     0  1800    17283     3298   126976        0             0 nmbd
May  3 04:35:53 wendy kernel: [1582359.085984] [   1814]     0  1814     1759      679    53248       47             0 cron
May  3 04:35:53 wendy kernel: [1582359.086345] [   1817]     0  1817     3376     1797    65536        0             0 openvpn
May  3 04:35:53 wendy kernel: [1582359.086703] [   1824]     0  1824     1798      532    57344       66             0 rc.local
May  3 04:35:53 wendy kernel: [1582359.087077] [   1833]     0  1833     2162      567    53248      171             0 cron
May  3 04:35:53 wendy kernel: [1582359.087467] [   1837]     0  1837   158203    10211  1413120     2944             0 node
May  3 04:35:53 wendy kernel: [1582359.087815] [   1838]     0  1838     1004      572    49152        0             0 atd
May  3 04:35:53 wendy kernel: [1582359.088210] [   1845]     0  1845     3071     1444    61440        0         -1000 sshd
May  3 04:35:53 wendy kernel: [1582359.088572] [   1871]     0  1871      652      132    40960       21             0 sh
May  3 04:35:53 wendy kernel: [1582359.088925] [   1873]     0  1873     8528     3974   106496     3046             0 IDrive:CRON
May  3 04:35:53 wendy kernel: [1582359.089282] [   1926]     0  1926   384537     6428   397312      856          -500 dockerd
May  3 04:35:53 wendy kernel: [1582359.089639] [   1942]     0  1942   389309     7363   352256        0             0 libvirtd
May  3 04:35:53 wendy kernel: [1582359.089992] [   1944]     0  1944    28941     4215   122880        1             0 unattended-upgr
May  3 04:35:53 wendy kernel: [1582359.090711] [   1959]     0  1959     6865     4771    94208        1             0 ddclient - slee
May  3 04:35:53 wendy kernel: [1582359.091576] [   1992]     0  1992    21252     5474   163840        0             0 smbd
May  3 04:35:53 wendy kernel: [1582359.091969] [   2044]     0  2044    20655     2223   147456        0             0 smbd-notifyd
May  3 04:35:53 wendy kernel: [1582359.092705] [   2045]     0  2045    20657     1880   143360        0             0 cleanupd
May  3 04:35:53 wendy kernel: [1582359.093098] [   2054]     0  2054    21057     4276   155648        0             0 samba-bgqd
May  3 04:35:53 wendy kernel: [1582359.093475] [   2091]     0  2091      622      111    45056       20             0 none
May  3 04:35:53 wendy kernel: [1582359.093864] [   2255]     0  2255     9543      786    69632        0             0 master
May  3 04:35:53 wendy kernel: [1582359.094207] [   2262]   116  2262     9661      768    73728        0             0 qmgr
May  3 04:35:53 wendy kernel: [1582359.094547] [   2279]   117  2279     2364      422    61440        0             0 dnsmasq
May  3 04:35:53 wendy kernel: [1582359.094884] [   2280]     0  2280     2357       90    61440        0             0 dnsmasq
May  3 04:35:53 wendy kernel: [1582359.095243] [   2687]     0  2687    17809     2466   131072        0          -900 virtlogd
May  3 04:35:53 wendy kernel: [1582359.095575] [   2711]     0  2711    10637     2421   122880     5324             0 miniserv.pl
May  3 04:35:53 wendy kernel: [1582359.095951] [   2864]   111  2864    38287      748    61440        0             0 rtkit-daemon
May  3 04:35:53 wendy kernel: [1582359.096586] [   3386]   115  3386   350453    88697  1220608    10745             0 mongod
May  3 04:35:53 wendy kernel: [1582359.096919] [  11440]   125 11440    19938     1884    73728        0             0 mosquitto
May  3 04:35:53 wendy kernel: [1582359.097251] [ 640237]     0 640237    62403     1745   102400        0             0 upowerd
May  3 04:35:53 wendy kernel: [1582359.097606] [1242742]     0 1242742   307586      600   151552        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.098261] [1242749]     0 1242749   289217      607   147456        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.098895] [1242763]     0 1242763   307586      568   159744        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.099591] [1242769]     0 1242769   307650      606   147456        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.100344] [1242785]     0 1242785   289217      621   151552        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.101042] [1242792]     0 1242792   307650      559   155648        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.101823] [1242807]     0 1242807   307650      551   143360        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.102641] [1242814]     0 1242814   289153      572   139264        0          -500 docker-proxy
May  3 04:35:53 wendy kernel: [1582359.103480] [1242831]     0 1242831   180123      879   106496        0          -998 containerd-shim
May  3 04:35:53 wendy kernel: [1582359.104317] [1242857]     0 1242857       51        2    28672        4             0 s6-svscan
May  3 04:35:53 wendy kernel: [1582359.104740] [1242923]     0 1242923       51        2    28672        3             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.105550] [1242925]     0 1242925       48        0    24576        2             0 s6-linux-init-s
May  3 04:35:53 wendy kernel: [1582359.106366] [1242932]     0 1242932       51        2    28672        3             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.107354] [1242933]     0 1242933       51        2    28672        2             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.108239] [1242934]     0 1242934       51        2    28672        4             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.109111] [1242935]     0 1242935       51        2    28672        2             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.110001] [1242936]     0 1242936       51        2    28672        2             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.110953] [1242937]     0 1242937       51        2    28672        3             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.111889] [1242938]     0 1242938       51        2    28672        4             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.112831] [1242939]     0 1242939       51        2    28672        2             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.113697] [1242940]     0 1242940       51        2    28672        4             0 s6-supervise
May  3 04:35:53 wendy kernel: [1582359.114643] [1242949]     0 1242949      129        2    28672       14             0 s6-fdholderd
May  3 04:35:53 wendy kernel: [1582359.115547] [1242950]     0 1242950       45        0    24576        1             0 s6-ipcserverd
May  3 04:35:53 wendy kernel: [1582359.116399] [1242987] 65534 1242987       68       24    36864        0             0 s6-log
May  3 04:35:53 wendy kernel: [1582359.116838] [1242988] 65534 1242988       67        0    40960        5             0 s6-log
May  3 04:35:53 wendy kernel: [1582359.117248] [1242990] 65534 1242990       68        0    40960        1             0 s6-log
May  3 04:35:53 wendy kernel: [1582359.117652] [1243006]     0 1243006   181663     1930   180224     2596             0 go2rtc
May  3 04:35:53 wendy kernel: [1582359.118052] [1243018]     0 1243018      971      374    45056       33             0 bash
May  3 04:35:53 wendy kernel: [1582359.118446] [1243019]     0 1243019   709680   105096  1642496     7048             0 python3
May  3 04:35:53 wendy kernel: [1582359.118867] [1243027]     0 1243027   145418      772   110592      433             0 nginx
May  3 04:35:53 wendy kernel: [1582359.119285] [1243039]     0 1243039   216931     6065   282624      519             0 nginx
May  3 04:35:53 wendy kernel: [1582359.119687] [1243040]     0 1243040   217670     6546   286720      463             0 nginx
May  3 04:35:53 wendy kernel: [1582359.120084] [1243041]     0 1243041   211184      541   233472      530             0 nginx
May  3 04:35:53 wendy kernel: [1582359.120471] [1243042]     0 1243042   218660     8262   294912      462             0 nginx
May  3 04:35:53 wendy kernel: [1582359.120854] [1243191]     0 1243191   148293     7396   376832    11654             0 frigate.logger
May  3 04:35:53 wendy kernel: [1582359.121660] [1243197]     0 1243197     3562      227    69632     1226             0 python3
May  3 04:35:53 wendy kernel: [1582359.122076] [1243198]     0 1243198   242304    11766   495616    15636             0 frigate.detecto
May  3 04:35:53 wendy kernel: [1582359.122844] [1243199]     0 1243199   431505    14461   688128    15776             0 frigate.output
May  3 04:35:53 wendy kernel: [1582359.123655] [1243203]     0 1243203   272042    19597   630784    15110             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.124415] [1243204]     0 1243204   271892    19799   630784    15166             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.125231] [1243205]     0 1243205    64360     9182   290816      567             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.125629] [1243207]     0 1243207   272440    19458   638976    15108             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.126397] [1243209]     0 1243209   271995    18939   630784    15114             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.127251] [1243211]     0 1243211    31703       86   163840      742             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.127669] [1243212]     0 1243212   271522    19115   630784    15182             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.128452] [1243213]     0 1243213    31703       39   176128      741             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.128854] [1243214]     0 1243214   273238    20790   647168    15093             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.129638] [1243215]     0 1243215   271259    18655   630784    15206             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.130486] [1243216]     0 1243216   273623    20855   651264    15052             0 frigate.process
May  3 04:35:53 wendy kernel: [1582359.131287] [1243217]     0 1243217    31703       50   163840      742             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.131705] [1243218]     0 1243218   270810    10518   512000    15663             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.132536] [1243219]     0 1243219   270810     9816   503808    15671             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.133396] [1243222]     0 1243222   270810     9889   503808    15595             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.134288] [1243223]     0 1243223    31703       72   163840      742             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.134767] [1243224]     0 1243224   270810    16661   561152    15678             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.135678] [1243228]     0 1243228   270810    10007   507904    15710             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.136590] [1243229]     0 1243229    31703       81   163840      741             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.137090] [1243231]     0 1243231   270810    10654   512000    15591             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.137995] [1243233]     0 1243233 11742079    27229   544768     3147             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.138443] [1243234]     0 1243234 11733821    26293   512000     2659             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.138881] [1243235]     0 1243235   270810     9794   503808    15660             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.139784] [1243240]     0 1243240 11730781    24662   503808     2736             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.140245] [1243241]     0 1243241    31703       35   167936      741             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.140739] [1243242]     0 1243242   270810    10960   520192    15331             0 frigate.capture
May  3 04:35:53 wendy kernel: [1582359.141701] [1243247]     0 1243247 11733196    27598   503808      930             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.142214] [1243248]     0 1243248 11733789    26400   512000     2704             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.142669] [1243252]     0 1243252 11733671    27933   507904     1126             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.143164] [1243254]     0 1243254 11733605    26154   499712     2707             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.143633] [1243257]     0 1243257    31703      102   167936      742             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.144057] [1243261]     0 1243261    61928    14190   344064      689             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.144454] [1497270]     0 1497270 11746832    35336   548864        0             0 ffmpeg
May  3 04:35:53 wendy kernel: [1582359.144847] [1745107]   112 1745107    90726    13935   794624    38708             0 Plex Media Serv
May  3 04:35:53 wendy kernel: [1582359.145620] [1745308]   112 1745308    16067     1350   151552     6963             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.146461] [1745354]   112 1745354    10727       18   106496      459             0 Plex Tuner Serv
May  3 04:35:53 wendy kernel: [1582359.147320] [1745386]   112 1745386    13864      142   122880     5945             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.148145] [1745734]   112 1745734    13867      166   126976     5951             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.148980] [1745737]   112 1745737    13960       82   135168     6033             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.149852] [1745738]   112 1745738    13928       76   131072     6057             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.150630] [1745788]   112 1745788    16606      142   151552     8716             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.151468] [1745843]   112 1745843    21892      177   196608    12438             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.152288] [1745868]   112 1745868    16713       69   155648     8865             0 Plex Script Hos
May  3 04:35:53 wendy kernel: [1582359.153116] [2532589]     0 2532589    43589     2703   110592      132             0 (ogrotate)
May  3 04:35:53 wendy kernel: [1582359.154034] [2532673]     0 2532673     4241     1428    77824        0             0 systemd-user-ru
May  3 04:35:53 wendy kernel: [1582359.154871] [2560587]     0 2560587    97258     7507   192512        0             0 fwupd
May  3 04:35:53 wendy kernel: [1582359.155356] [2572205]   116 2572205     9614     1558    69632        0             0 pickup
May  3 04:35:53 wendy kernel: [1582359.155801] [2572898]     0 2572898      620      223    45056        0             0 sleep
May  3 04:35:53 wendy kernel: [1582359.156214] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=c60d212c31b57561162f7b53db0ffcb84c9449b1aab175ad5dd88e41eb0b8c19,mems_allowed=0,global_oom,task_memcg=/docker/c60d212c31b57561162f7b53db0ffcb84c9449b1aab175ad5dd88e41eb0b8c19,task=python3,pid=1243019,uid=0
May  3 04:35:53 wendy kernel: [1582359.157616] Out of memory: Killed process 1243019 (python3) total-vm:2838720kB, anon-rss:413716kB, file-rss:6384kB, shmem-rss:284kB, UID:0 pgtables:1604kB oom_score_adj:0
May  3 04:36:16 wendy kernel: [1582382.318863] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [frigate.output:1243199]
May  3 04:36:16 wendy kernel: [1582382.319372] Modules linked in: rpcsec_gss_krb5 xt_nat veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter aufs overlay bridge stp llc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) snd_hda_codec_hdmi zcommon(PO) znvpair(PO) spl(O) intel_rapl_msr ppdev mei_hdcp cdc_acm binfmt_misc intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm rapl intel_cstate mxm_wmi snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer mei_me snd mei soundcore nvidia_uvm(OE) parport_pc parport mac_hid intel_smartconnect sch_fq_codel msr ramoops reed_solomon nfsd efi_pstore auth_rpcgss nfs_acl lockd grace sunrpc
May  3 04:36:16 wendy kernel: [1582382.319393]  ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear nvidia_drm(POE) nvidia_modeset(POE) uas usb_storage nvidia(POE) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel drm_kms_helper crypto_simd syscopyarea cryptd glue_helper sysfillrect sysimgblt ahci fb_sys_fops libahci i2c_i801 r8169 drm lpc_ich realtek wmi video
May  3 04:36:16 wendy kernel: [1582382.327942] CPU: 0 PID: 1243199 Comm: frigate.output Tainted: P      D W  OEL    5.4.0-144-generic #161-Ubuntu
May  3 04:36:16 wendy kernel: [1582382.329073] Hardware name: MSI MS-7816/Z87-G43 (MS-7816), BIOS V1.11 05/09/2015
May  3 04:36:16 wendy kernel: [1582382.329656] RIP: 0010:xas_load+0x45/0x80
May  3 04:36:16 wendy kernel: [1582382.330229] Code: 00 10 00 00 77 03 5b 5d c3 0f b6 48 fe 48 8d 70 fe 38 4b 10 77 f0 48 8b 53 08 48 d3 ea 83 e2 3f 89 d0 48 8d 44 c6 28 48 8b 00 <48> 89 73 18 48 89 c1 83 e1 03 48 83 f9 02 75 18 48 3d fd 00 00 00
May  3 04:36:16 wendy kernel: [1582382.332035] RSP: 0018:ffffbfc8ca12bb78 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
May  3 04:36:16 wendy kernel: [1582382.332637] RAX: fffff326880c9800 RBX: ffffbfc8ca12bbb8 RCX: 0000000000000000
May  3 04:36:16 wendy kernel: [1582382.333234] RDX: 0000000000000001 RSI: ffffa0a95d09e238 RDI: ffffbfc8ca12bbb8
May  3 04:36:16 wendy kernel: [1582382.333829] RBP: ffffbfc8ca12bb80 R08: ffffbfc8ca12bc88 R09: 0000000002e63ef5
May  3 04:36:16 wendy kernel: [1582382.334426] R10: 000000000000b98c R11: 0000000000000000 R12: ffffffffffffffff
May  3 04:36:16 wendy kernel: [1582382.335061] R13: ffffbfc8ca12bc88 R14: 000000000000000f R15: 0000000000000000
May  3 04:36:16 wendy kernel: [1582382.335664] FS:  00007f0315311740(0000) GS:ffffa0b0bea00000(0000) knlGS:0000000000000000
May  3 04:36:16 wendy kernel: [1582382.336264] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  3 04:36:16 wendy kernel: [1582382.336852] CR2: 00007f031540d92d CR3: 000000053bfdc002 CR4: 00000000001626f0
May  3 04:36:16 wendy kernel: [1582382.337436] Call Trace:
May  3 04:36:16 wendy kernel: [1582382.338012]  xas_find+0x17f/0x1c0
May  3 04:36:16 wendy kernel: [1582382.338574]  find_get_entries+0x1e6/0x270
May  3 04:36:16 wendy kernel: [1582382.339160]  shmem_undo_range+0x2b2/0x930
May  3 04:36:16 wendy kernel: [1582382.339706]  shmem_truncate_range+0x14/0x40
May  3 04:36:16 wendy kernel: [1582382.340240]  shmem_evict_inode+0xf3/0x260
May  3 04:36:16 wendy kernel: [1582382.340823]  ? var_wake_function+0x30/0x30
May  3 04:36:16 wendy kernel: [1582382.341425]  evict+0xd2/0x1b0
May  3 04:36:16 wendy kernel: [1582382.342043]  iput+0x148/0x210
May  3 04:36:16 wendy kernel: [1582382.342586]  do_unlinkat+0x1c5/0x2d0
May  3 04:36:16 wendy kernel: [1582382.343107]  __x64_sys_unlink+0x23/0x30
May  3 04:36:16 wendy kernel: [1582382.343595]  do_syscall_64+0x57/0x190
May  3 04:36:16 wendy kernel: [1582382.344077]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
May  3 04:36:16 wendy kernel: [1582382.344555] RIP: 0033:0x7f031540d957
May  3 04:36:16 wendy kernel: [1582382.345014] Code: Bad RIP value.

Frigate Log: 

s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
s6-rc: info: service go2rtc-healthcheck successfully stopped
2023-05-03 04:35:53.441668292  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
2023-05-03 04:36:23.488101215  [INFO] Service NGINX exited with code 256 (by signal 9)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
s6-rc: info: service frigate successfully stopped
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
s6-rc: info: service nginx-log successfully stopped
2023-05-03 04:36:23.494515400  exit OK
s6-rc: info: service frigate-log successfully stopped
2023-05-03 04:36:23.497646653  [INFO] The go2rtc service exited with code 0 (by signal 0)
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service frigate successfully started
2023-05-03 07:19:25.724173808  [INFO] Preparing go2rtc config...
xbmcnut commented 1 year ago

@erbain Any chance it's related to this issue? And any advice you can provide on capturing persistent logs would be appreciated. https://github.com/blakeblackshear/frigate/issues/6469#issue-1706911584

Inrego commented 1 year ago

I would say it's unrelated. I'm the author of that other issue, and I'm seeing nowhere near half of my ram being used. Also, my syslog doesn't show anything of the like.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

robinostlund commented 1 year ago

@erbain did you find any workaround for this issue? i have also started to get the same problems with soft lockup šŸ˜„

Inrego commented 1 year ago

@Inrego did you find any workaround for this issue? i have also started to get the same problems with soft lockup šŸ˜„

Are you sure you meant to tag me? I specifically said that this issue is unrelated to the issues I was having

robinostlund commented 1 year ago

@Inrego sorry i was supposed to tag @erbain , i have edited my post šŸ˜„

erbain commented 1 year ago

@robinostlund nope sorry, it only ever happened to me that one time, I enabled debugging but as it's not recurred I've not got any new info.