Streamer locks up, restarting service puts it into zombie state

[charredchar]

I've had this happen a couple of times now, if I leave it running for a day or two it eventually stops responding. To attempt and resolve it I restart the service but it never successfully does and ends up stuck as a zombie process.

65218 4036 26.4 0.0 0 0 ? ZNsl Jan18 221:49 [camera-streamer] <defunct>

The only way I can find to fix it is to reboot the Pi, not great if I am running a print.

[ayufan]

@charredchar

Do you have anything in dmesg where it happens? This appears to be zombie process, that holds v4l2 devices. This could happen when it runs out of memory on v4l2 m2m processing, where the app deadlocks and cannot do anything with it.

Also, what kernel version do you use?

[charredchar]

Sadly, I don't know the exact time it happens because I usually come back to my PC and see it isn't loading the stream. But this is the first error I see in dmesg;

[50463.263871] bcm2835-codec bcm2835-codec: bcm2835_codec_stop_streaming: Failed disabling i/p port, ret -62
[50465.343870] bcm2835-codec bcm2835-codec: bcm2835_codec_flush_buffers: Timeout waiting for buffers to be returned - 2 outstanding
[50468.383910] bcm2835_mmal_vchiq: timed out waiting for sync completion
[50468.383935] bcm2835-codec bcm2835-codec: bcm2835_codec_stop_streaming: Failed enabling component, ret -62
[50468.383961] ------------[ cut here ]------------
[50468.383975] WARNING: CPU: 1 PID: 4047 at drivers/media/common/videobuf2/videobuf2-core.c:1995 __vb2_queue_cancel+0x1e0/0x258 [videobuf2_common]
[50468.384061] Modules linked in: cdc_acm nft_counter xt_DSCP xt_tcpudp nft_compat nf_tables nfnetlink cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc imx219 8021q garp stp llc brcmfmac brcmutil vc4 cfg80211 v3d gpu_sched snd_soc_hdmi_codec cec raspberrypi_hwmon rfkill i2c_mux_pinctrl i2c_mux bcm2835_unicam drm_kms_helper v4l2_dv_timings v4l2_fwnode v4l2_async bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) i2c_bcm2835 bcm2835_mmal_vchiq(C) snd_soc_core vc_sm_cma(C) videobuf2_vmalloc rpivid_hevc(C) v4l2_mem2mem videobuf2_dma_contig snd_bcm2835(C) videobuf2_memops videobuf2_v4l2 videobuf2_common snd_compress snd_pcm_dmaengine videodev snd_pcm mc snd_timer snd syscopyarea sysfillrect sysimgblt uio_pdrv_genirq fb_sys_fops nvmem_rmem uio drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[50468.385075] CPU: 1 PID: 4047 Comm: camera-streamer Tainted: G         C        5.15.84-v7l+ #1613

Is there anything I can do to release it so I don't need to reboot every time?

[ayufan]

What device it is?

[charredchar]

An imx219 module installed on a PiCam V2 board.

[ayufan]

Camera I don't think is a problem. I ask about RPI device and if you could also tell me gpu and cma memory settings. I remember that this was pretty fragile if run on the limit for available memory. The v4l2 does not like at all memory fragmentation and in some cases it might take a long time to output frame that locks kernel driver.

[charredchar]

My apologies. It is a Pi 4 1GB with the GPU memory set to 256 in the boot/config.txt. I do not know where to find the CMA memory setting.

[ayufan]

dmesg | grep -i cma should do the trick.

You could also try to change: set GPU mem to 160, but CMA to 256MB.

[charredchar]

It looks like CMA might already be set to 256MB, but to be sure where can I set this? I'll change GPU to 160 and reboot the Pi.

[    0.000000] Reserved memory: created CMA memory pool at 0x000000001ec00000, size 256 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Memory: 627136K/917504K available (10240K kernel code, 1386K rwdata, 3308K rodata, 2048K init, 591K bss, 28224K reserved, 262144K cma-reserved, 131072K highmem)
[    5.285381] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[    5.303933] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[    5.334509] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[    5.340434] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[    5.344284] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[50468.384061] Modules linked in: cdc_acm nft_counter xt_DSCP xt_tcpudp nft_compat nf_tables nfnetlink cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc imx219 8021q garp stp llc brcmfmac brcmutil vc4 cfg80211 v3d gpu_sched snd_soc_hdmi_codec cec raspberrypi_hwmon rfkill i2c_mux_pinctrl i2c_mux bcm2835_unicam drm_kms_helper v4l2_dv_timings v4l2_fwnode v4l2_async bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) i2c_bcm2835 bcm2835_mmal_vchiq(C) snd_soc_core vc_sm_cma(C) videobuf2_vmalloc rpivid_hevc(C) v4l2_mem2mem videobuf2_dma_contig snd_bcm2835(C) videobuf2_memops videobuf2_v4l2 videobuf2_common snd_compress snd_pcm_dmaengine videodev snd_pcm mc snd_timer snd syscopyarea sysfillrect sysimgblt uio_pdrv_genirq fb_sys_fops nvmem_rmem uio drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6

[ayufan]

Then - I would not change it.

What other settings you have for camera-streamer? How you run it, with what res?

[charredchar]

I used the default camera-streamer/service/camera-streamer-pi219-8MP.service file. I cloned the repo yesterday so it should be the newest version.

[ayufan]

OK.

@charredchar People say that this is due some throttling happening on device: https://github.com/raspberrypi/firmware/issues/1627. So, it appears that in some cases those timeouts configured might be too low. Can you validate if you observe anything out of this issue linked?

[charredchar]

I don't think I am having any of those issues as Fluidd would have alerted me of any overheating, throttling or under-volting (I've seen the under-volt warning before changing wiring) but I can try over_voltage=2 and see what happens.

[pocek]

Hi @charredchar. Were you able to solve the issue?

[charredchar]

Hi @charredchar. Were you able to solve the issue?

Sorry, I did not solve this directly. I eventually enabled Shell Commands in Klipper and had my printer stop the Camera-Streamer service any time a print finished and start it before a print. I did this to save on resources when idle (which dropped Pi temps so I could turn off fans) but it also meant Camera-Streamer never ran for long.

I can leave this open if anyone else has issues you want to verify, if not feel free to close it.