Open dfunkt opened 2 years ago
Any update on this? We've been having this exact same crash consistently while hardware encoding a live RSTP input
Sometimes when encoding a livestream using ffmpeg
the hardware encoder fails, leaving the ISP unusable, one can actually copy the stream after the crash but not re-use the hardware encoder unless rebooting the system
Hardware encode a RSTP stream using h264_v4l2m2m
encoder
Command
ffmpeg -rtsp_transport tcp -nostdin -fflags +igndts -use_wallclock_as_timestamps 1 -r 12.5 -flags +global_header -y -i rtsp://192.0.2.3/stream/jpeg -vf scale=-1:720,showinfo -c:v h264_v4l2m2m -copytb 0 -copyts -num_output_buffers 32 -num_capture_buffers 16 -pix_fmt yuv420p -f mpegts -movflags faststart -b:v 3M -fps_mode passthrough -segment_time 600 -reset_timestamps 1 -f segment -strftime 1 -segment_format mpegts -segment_atclocktime 1 /home/pi/output/videos/%Y%m%d_%H%M%S_600.ts
Raspberry Pi CM4 4GB RAM
cat /etc/rpi-issue:
Raspberry Pi reference 2022-04-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 27a8050c3c06e567c794620394a8c2d74262a516, stage4
vcgencmd version:
Mar 24 2022 13:19:26
Copyright (c) 2012 Broadcom
version e5a963efa66a1974127860b42e913d2374139ff5 (clean) (release) (start)
uname -a:
Linux C9 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux
Aug 9 10:01:52 C9 kernel: [18705.248875] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
Aug 9 10:01:52 C9 kernel: [18705.248901] Mem abort info:
Aug 9 10:01:52 C9 kernel: [18705.248907] ESR = 0x96000045
Aug 9 10:01:52 C9 kernel: [18705.248914] EC = 0x25: DABT (current EL), IL = 32 bits
Aug 9 10:01:52 C9 kernel: [18705.248957] SET = 0, FnV = 0
Aug 9 10:01:52 C9 kernel: [18705.248964] EA = 0, S1PTW = 0
Aug 9 10:01:52 C9 kernel: [18705.248970] FSC = 0x05: level 1 translation fault
Aug 9 10:01:52 C9 kernel: [18705.248976] Data abort info:
Aug 9 10:01:52 C9 kernel: [18705.248981] ISV = 0, ISS = 0x00000045
Aug 9 10:01:52 C9 kernel: [18705.248987] CM = 0, WnR = 1
Aug 9 10:01:52 C9 kernel: [18705.248993] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000092a48000
Aug 9 10:01:52 C9 kernel: [18705.249001] [0000000000000008] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
Aug 9 10:01:52 C9 kernel: [18705.249022] Internal error: Oops: 96000045 [#1] PREEMPT SMP
Aug 9 10:01:52 C9 kernel: [18705.249029] Modules linked in: cdc_ether qmi_wwan cdc_wdm option usb_wwan 8021q garp stp llc emc2301(O) binfmt_misc brcmfmac brcmutil ftdi_sio usbserial cfg80211 rfkill snd_bcm2835(C) i2c_mux_pinctrl bcm2835_codec(C) i2c_mux v4l2_mem2mem bcm2835_v4l2(C) bcm2835_isp(C) videobuf2_dma_contig bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_pcm videobuf2_common raspberrypi_hwmon snd_timer videodev vc_sm_cma(C) snd mc i2c_bcm2835 w1_therm rpivid_mem w1_gpio wire cn uio_pdrv_genirq uio nvmem_rmem i2c_dev fuse drm drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
Aug 9 10:01:52 C9 kernel: [18705.249142] CPU: 1 PID: 16509 Comm: ffmpeg Tainted: G C O 5.15.32-v8+ #1538
Aug 9 10:01:52 C9 kernel: [18705.249150] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
Aug 9 10:01:52 C9 kernel: [18705.249155] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 9 10:01:52 C9 kernel: [18705.249161] pc : vchiq_mmal_port_enable+0xd8/0x158 [bcm2835_mmal_vchiq]
Aug 9 10:01:52 C9 kernel: [18705.249178] lr : vchiq_mmal_port_enable+0x114/0x158 [bcm2835_mmal_vchiq]
Aug 9 10:01:52 C9 kernel: [18705.249187] sp : ffffffc00e0cba60
Aug 9 10:01:52 C9 kernel: [18705.249190] x29: ffffffc00e0cba60 x28: 0000000000000002 x27: dead000000000100
Aug 9 10:01:52 C9 kernel: [18705.249201] x26: dead000000000122 x25: ffffff804cdc0220 x24: ffffff804cdc0008
Aug 9 10:01:52 C9 kernel: [18705.249210] x23: ffffff804cdc0000 x22: ffffff804cdc0178 x21: 0000000000000000
Aug 9 10:01:52 C9 kernel: [18705.249219] x20: 0000000000000000 x19: ffffff8092090918 x18: 0000000000000011
Aug 9 10:01:52 C9 kernel: [18705.249228] x17: 0000000000000000 x16: 0000000000000000 x15: ffffffffffffffff
Aug 9 10:01:52 C9 kernel: [18705.249236] x14: ffffffc008e8a9c8 x13: ffffffc08e0cb7d7 x12: ffffffc00e0cb7dd
Aug 9 10:01:52 C9 kernel: [18705.249245] x11: bcbcbcbcbcbcbcbc x10: 000000000000003a x9 : ffffffc00892654c
Aug 9 10:01:52 C9 kernel: [18705.249254] x8 : 00000000000000aa x7 : 0000000000000002 x6 : ffffffc00e0cb6e0
Aug 9 10:01:52 C9 kernel: [18705.249263] x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffffff8045ca701c
Aug 9 10:01:52 C9 kernel: [18705.249271] x2 : 99d17e1a4f890200 x1 : 0000000000000000 x0 : 0000000000000000
Aug 9 10:01:52 C9 kernel: [18705.249280] Call trace:
Aug 9 10:01:52 C9 kernel: [18705.249283] vchiq_mmal_port_enable+0xd8/0x158 [bcm2835_mmal_vchiq]
Aug 9 10:01:52 C9 kernel: [18705.249293] bcm2835_codec_start_streaming+0x214/0x408 [bcm2835_codec]
Aug 9 10:01:52 C9 kernel: [18705.249303] vb2_start_streaming+0x74/0x160 [videobuf2_common]
Aug 9 10:01:52 C9 kernel: [18705.249319] vb2_core_streamon+0x98/0x198 [videobuf2_common]
Aug 9 10:01:52 C9 kernel: [18705.249331] vb2_streamon+0x24/0x78 [videobuf2_v4l2]
Aug 9 10:01:52 C9 kernel: [18705.249342] v4l2_m2m_streamon+0x34/0x88 [v4l2_mem2mem]
Aug 9 10:01:52 C9 kernel: [18705.249355] v4l2_m2m_ioctl_streamon+0x20/0x30 [v4l2_mem2mem]
Aug 9 10:01:52 C9 kernel: [18705.249366] v4l_streamon+0x2c/0x38 [videodev]
Aug 9 10:01:52 C9 kernel: [18705.249403] __video_do_ioctl+0x188/0x410 [videodev]
Aug 9 10:01:52 C9 kernel: [18705.249434] video_usercopy+0x310/0x7d0 [videodev]
Aug 9 10:01:52 C9 kernel: [18705.249464] video_ioctl2+0x20/0x38 [videodev]
Aug 9 10:01:52 C9 kernel: [18705.249495] v4l2_ioctl+0x48/0x68 [videodev]
Aug 9 10:01:52 C9 kernel: [18705.249525] __arm64_sys_ioctl+0xb0/0xf0
Aug 9 10:01:52 C9 kernel: [18705.249534] invoke_syscall+0x4c/0x110
Aug 9 10:01:52 C9 kernel: [18705.249543] el0_svc_common.constprop.3+0xfc/0x120
Aug 9 10:01:52 C9 kernel: [18705.249549] do_el0_svc+0x2c/0x90
Aug 9 10:01:52 C9 kernel: [18705.249555] el0_svc+0x24/0x60
Aug 9 10:01:52 C9 kernel: [18705.249563] el0t_64_sync_handler+0x90/0xb8
Aug 9 10:01:52 C9 kernel: [18705.249569] el0t_64_sync+0x1a0/0x1a4
Aug 9 10:01:52 C9 kernel: [18705.249577] Code: f2fbd5bb f2fbd5ba 1400000c a9400261 (f9000420)
Aug 9 10:01:52 C9 kernel: [18705.249583] ---[ end trace 467d91e1301520a6 ]---
Experiencing the same issue when the encoder runs for multiple days (unrelated to Jellyfin).
ffmpeg doesn't give any errors, just freezes and I see this in dmesg:
[321584.070481] bcm2835-codec bcm2835-codec: Mutex fail
Same issue here, seems to be especially an issue when seeking fast for me (Raspberry Pi 4 Model B Rev 1.4 (DT))
Same issue here
Still seeing the same on RPi 3 with kernel version:
Linux raspberrypi 6.1.21-v7+ #1642 SMP Mon Apr 3 17:20:52 BST 2023 armv7l GNU/Linux
Using the encoder from time to time for a few minutes and at some point it just stops working with the following crash dump:
[133329.621107] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[133329.625135] [00000004] *pgd=00000000
[133329.627097] Internal error: Oops: 805 [#1] SMP ARM
[133329.628896] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter br_netfilter bridge cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc overlay 8021q garp stp llc brcmfmac it913x af9033 regmap_i2c brcmutil sha256_generic cfg80211 dvb_usb_af9035 dvb_usb_v2 dvb_core bcm2835_codec(C) bcm2835_v4l2(C) videobuf2_vmalloc v4l2_mem2mem rfkill bcm2835_isp(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig snd_bcm2835(C) videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon snd_pcm videodev snd_timer snd vc_sm_cma(C) mc uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[133329.648879] CPU: 0 PID: 23115 Comm: ffmpeg Tainted: G C 6.1.21-v7+ #1642
[133329.653402] Hardware name: BCM2835
[133329.655683] PC is at vchiq_mmal_port_enable+0xcc/0x108 [bcm2835_mmal_vchiq]
[133329.658135] LR is at vchiq_release_service+0x30/0x40
[133329.660533] pc : [<7f1ba64c>] lr : [<80857f50>] psr: 60000013
[133329.662955] sp : bc24dd60 ip : 00000000 fp : 00000002
[133329.665320] r10: 00000000 r9 : 85c00194 r8 : 85c00004
[133329.667617] r7 : 85c00000 r6 : 85c0010c r5 : 00000000 r4 : 8415b620
[133329.669935] r3 : 04195133 r2 : 00000000 r1 : 00000000 r0 : 00000100
[133329.672243] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[133329.674569] Control: 10c5383d Table: 29e9006a DAC: 00000055
[133329.676843] Register r0 information: non-paged memory
[133329.679072] Register r1 information: NULL pointer
[133329.681202] Register r2 information: NULL pointer
[133329.683217] Register r3 information: non-paged memory
[133329.685147] Register r4 information: slab kmalloc-4k start 8415b000 pointer offset 1568 size 4096
[133329.688882] Register r5 information: NULL pointer
[133329.690721] Register r6 information: non-slab/vmalloc memory
[133329.692504] Register r7 information: non-slab/vmalloc memory
[133329.694186] Register r8 information: non-slab/vmalloc memory
[133329.695776] Register r9 information: non-slab/vmalloc memory
[133329.697317] Register r10 information: NULL pointer
[133329.698801] Register r11 information: non-paged memory
[133329.700242] Register r12 information: NULL pointer
[133329.701625] Process ffmpeg (pid: 23115, stack limit = 0x74a0cb71)
[133329.703031] Stack: (0xbc24dd60 to 0xbc24e000)
[133329.704441] dd60: 00000000 85c0010c 00000002 8415b000 00000000 7f25c300 a9c90178 852a4840
[133329.707387] dd80: 8415b0e8 7f256f74 81005008 7e91616c bc24de74 00000011 40045612 a9c90130
[133329.710446] dda0: a9c90278 a9c90178 7f1d8f2c 00000001 852a4b50 7f25990c 85037180 7f2a30e4
[133329.713696] ddc0: a9c90178 a9c90000 81005008 7f1d8f2c 00000001 7f2a3744 0000000a 7f2e2ae8
[133329.717352] dde0: 852a48a0 40045612 81005008 7f1ddd54 00000000 00000000 8415b000 00000000
[133329.721203] de00: 00000000 bc24de74 00000001 7f1f3ce8 81005008 20000013 b6b304d4 20000013
[133329.725217] de20: 811d9274 04195133 b7117610 40045612 40045612 81005008 00000009 bc24de74
[133329.729659] de40: 00000000 7e916108 00000000 7f1de814 bc24de73 b7117610 85037180 7f1ddb2c
[133329.734500] de60: 00000000 00000000 00000000 00000000 00e2d1d8 0000000a 00000000 00000000
[133329.739667] de80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000254
[133329.745035] dea0: 75c49000 81005008 00000000 00000040 bc24dfb0 bc24deec 000001d8 80309f64
[133329.750481] dec0: bc24df70 81005008 a6c95840 80353e88 81005008 000001e7 a6ca1800 80b7af8c
[133329.755923] dee0: 82414600 80109c4c bc24df40 a9e0d870 00100cca 04195133 75c49000 40045612
[133329.761375] df00: 81005008 00000000 00000009 85037181 85037180 7e916108 830bac50 8036b880
[133329.766830] df20: 00000036 40000028 76f0235a 00000000 00000009 00000000 40045612 00000000
[133329.772285] df40: 7e916108 00000000 7e916108 00000000 40045612 00000000 734ef900 00000000
[133329.777742] df60: 81005008 04195133 00000007 87b3b600 bc24dfb0 734ef900 00000036 04195133
[133329.783203] df80: 87b3b600 40045612 734ef900 734ef900 00000036 80100264 87b3b600 00000036
[133329.788663] dfa0: 00000000 80100254 40045612 734ef900 00000009 40045612 7e916108 7e916108
[133329.794120] dfc0: 40045612 734ef900 734ef900 00000036 00000009 734b2880 734ef800 00000000
[133329.799575] dfe0: 76400cc8 7e915fd8 75f73957 76f0235a 60000030 00000009 00000000 00000000
[133329.805040] vchiq_mmal_port_enable [bcm2835_mmal_vchiq] from bcm2835_codec_start_streaming+0x1f4/0x3bc [bcm2835_codec]
[133329.810637] bcm2835_codec_start_streaming [bcm2835_codec] from vb2_start_streaming+0x64/0x170 [videobuf2_common]
[133329.816249] vb2_start_streaming [videobuf2_common] from vb2_core_streamon+0x88/0x178 [videobuf2_common]
[133329.821863] vb2_core_streamon [videobuf2_common] from v4l2_m2m_streamon+0x28/0x58 [v4l2_mem2mem]
[133329.827456] v4l2_m2m_streamon [v4l2_mem2mem] from __video_do_ioctl+0x228/0x454 [videodev]
[133329.833215] __video_do_ioctl [videodev] from video_usercopy+0x138/0x5dc [videodev]
[133329.839122] video_usercopy [videodev] from sys_ioctl+0x120/0x984
[133329.842123] sys_ioctl from __sys_trace_return+0x0/0x10
[133329.844836] Exception stack(0xbc24dfa8 to 0xbc24dff0)
[133329.847483] dfa0: 40045612 734ef900 00000009 40045612 7e916108 7e916108
[133329.852647] dfc0: 40045612 734ef900 734ef900 00000036 00000009 734b2880 734ef800 00000000
[133329.857783] dfe0: 76400cc8 7e915fd8 75f73957 76f0235a
[133329.860349] Code: e250a000 1affffdc e8940006 e3a00c01 (e5812004)
Any following attempts to use the encoder result with the following log messages:
[137232.618666] bcm2835-codec bcm2835-codec: Mutex fail
[143268.420861] bcm2835-codec bcm2835-codec: Mutex fail
[161930.260120] bcm2835-codec bcm2835-codec: Mutex fail
[161960.690046] bcm2835-codec bcm2835-codec: Mutex fail
[162039.325677] bcm2835-codec bcm2835-codec: Mutex fail
I'm guessing some null pointer is being dereferenced after the lock is acquired and then it's never released blocking future attempts.
Describe the bug
Sometimes when I try to play something that requires transcoding it just hangs infinitely and the info attached in the logs below pops up in dmesg.
Steps to reproduce the behaviour
Try to transcode content in Jellyfin (nightly version) with the V4L2 encoder.
Device (s)
Raspberry Pi 4 Mod. B
System
Logs