[BUG] Potential memory usage issue on Pi Zero W

[darksidelemm]

Describe the bug I'm using the picamera2 library to capture JPEG images for transmission to ground from a high-altitude balloon payload. I've been using the legacy camera interface for many years with great success, but would now like to use the new Pi Camera v3, which requires using the new picamera2 library.

The process I am using is:

• Setup Camera - Happens here: https://github.com/projecthorus/wenet/blob/picamera2/tx/WenetPiCamera2.py#L92
• Capture images to JPG - happens here: https://github.com/projecthorus/wenet/blob/picamera2/tx/WenetPiCamera2.py#L152

Every 'round' of transmission, I save 5x images to disk in sequence, pick the largest file (which when using JPEG is handily the one with the most 'detail' in it), and then do some processing (e.g. resizing) and transmit the image.

This runs fine for a while, but eventually the thread capturing the images crashes with a stack trace (it appears to occur when i call self.cam.capture_file)

I'm wondering if this is a lack-of-memory issue? The Pi Zero W doesn't have that much memory to begin with after all. (In fact enabling the legacy camera interface on bullseye on a Pi Zero W results in so much memory being used that the Pi will no longer boot...) Given I am only trying to get a single still image, is there any way to reduce the amount of buffers used even further (i believe buffer_count already defaults to 1, but I think there are still multiple buffers used).

To Reproduce Maybe a little bit difficult. The script linked above may work on other installs, but is expecting other hardware that will not exist.

Expected behaviour It keeps running without crashing :-)

Console Output, Screenshots

I get the following message in dmesg when the issue occurs:

[  263.439987] vc_sm_cma_vchi_rx_ack: received response 3246, throw away...
[  263.440071] vc_sm_cma_import_dmabuf: imported vc_sm_cma_get_buffer failed -512
[  689.150530] 8<--- cut here ---
[  689.150619] Unable to handle kernel paging request at virtual address bfa97c30
[  689.150657] pgd = a0b70ebb
[  689.150687] [bfa97c30] *pgd=00000000
[  689.150731] Internal error: Oops: 80000005 [#1] ARM
[  689.150762] Modules linked in: imx708 dw9807_vcm spidev vc4 brcmfmac snd_soc_hdmi_codec cec 8021q garp stp llc brcmutil sha256_generic libsha256 drm_kms_helper cfg80211 snd_soc_core cdc_acm bcm2835_unicam snd_compress snd_pcm_dmaengine i2c_mux_pinctrl syscopyarea i2c_mux v4l2_dv_timings sysfillrect sysimgblt fb_sys_fops v4l2_fwnode v4l2_async bcm2835_codec(C) raspberrypi_hwmon bcm2835_v4l2(C) v4l2_mem2mem bcm2835_isp(C) bcm2835_mmal_vchiq(C) rfkill videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common snd_bcm2835(C) snd_pcm videodev snd_timer spi_bcm2835 i2c_bcm2835 snd mc vc_sm_cma(C) fixed uio_pdrv_genirq uio drm i2c_dev fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[  689.151190] CPU: 0 PID: 919 Comm: python3 Tainted: G         C        5.15.84+ #1613
[  689.151231] Hardware name: BCM2835
[  689.151250] PC is at 0xbfa97c30
[  689.151281] LR is at video_usercopy+0x47c/0x5a8 [videodev]
[  689.151919] pc : [<bfa97c30>]    lr : [<bf25bc24>]    psr: 60000013
[  689.151947] sp : c260de10  ip : c260de10  fp : c260deec
[  689.151970] r10: c0c2f028  r9 : 00000000  r8 : 00000000
[  689.151992] r7 : c260de3c  r6 : 00000000  r5 : c0445611  r4 : c0505611
[  689.152016] r3 : 00000000  r2 : bf27da40  r1 : 48f04fd9  r0 : c51484e8
[  689.152041] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  689.152071] Control: 00c5387d  Table: 07070008  DAC: 00000055
[  689.152093] Register r0 information: slab kmalloc-4k start c5148000 pointer offset 1256 size 4096
[  689.152162] Register r1 information: non-paged memory
[  689.152194] Register r2 information: 58-page vmalloc region starting at 0xbf253000 allocated at load_module+0xac8/0x2788
[  689.152265] Register r3 information: NULL pointer
[  689.152294] Register r4 information: non-slab/vmalloc memory
[  689.152325] Register r5 information: non-slab/vmalloc memory
[  689.152353] Register r6 information: NULL pointer
[  689.152379] Register r7 information: non-slab/vmalloc memory
[  689.152406] Register r8 information: NULL pointer
[  689.152432] Register r9 information: NULL pointer
[  689.152458] Register r10 information: non-slab/vmalloc memory
[  689.152486] Register r11 information: non-slab/vmalloc memory
[  689.152513] Register r12 information: non-slab/vmalloc memory
[  689.152544] Process python3 (pid: 919, stack limit = 0x6eae114f)
[  689.152583] Stack: (0xc260de10 to 0xc260e000)
[  689.152615] de00:                                     c260de3b 00000122 c302d100 00000000
[  689.152652] de20: 00000000 00000000 00000000 00000000 c457bd20 b1efd5dc 001694f4 00000000
[  689.152690] de40: 00000001 00e3d000 00002000 00000000 00000000 000002b1 00000000 0001efd3
[  689.152727] de60: 00000000 00000000 00000000 00000000 00000000 00001646 00000004 00000027
[  689.152767] de80: 00e3d000 00000000 00000000 c04e0794 c260dedc c260dea0 c008072c c04e0780
[  689.152802] dea0: 00000003 00000000 165481c8 00000000 220114c5 48f04fd9 c260df68 48f04fd9
[  689.152838] dec0: c260def8 bf25bd50 c457bd21 00000000 c457bd20 c0c2f028 b1efd5dc 0000001d
[  689.152877] dee0: c260defc c260def0 bf25bd6c bf25b7b4 c260df14 c260df00 bf253148 bf25bd5c
[  689.152913] df00: c0445611 c457bd21 c260dfa4 c260df18 c02244d4 bf253108 00000000 c0c2f028
[  689.152950] df20: c0008ff0 00c5387d c260df6c c260df38 c001472c c0013788 00000000 00000000
[  689.152985] df40: 00000000 00000000 c0cac080 60000010 c0008ff0 00c5387d 00000000 c0009104
[  689.153022] df60: c260df94 c260df70 c00fb210 c00fc4ec b531d134 48f04fd9 ffffffff b1efd5dc
[  689.153058] df80: b532a124 b1efda44 00000036 c00083e4 c260c000 00000000 00000000 c260dfa8
[  689.153096] dfa0: c0008260 c02243cc b1efd5dc b532a124 0000001d c0445611 b1efd5dc 00000004
[  689.153133] dfc0: b1efd5dc b532a124 b1efda44 00000036 b2712f60 fffffff8 b1efd7f4 b2701638
[  689.153170] dfe0: b50fad6c b1efd56c b5079d90 b6dc3d4c a0000010 0000001d 00000000 00000000
[  689.153200] Backtrace: 
[  689.153225] [<bf25b7a8>] (video_usercopy [videodev]) from [<bf25bd6c>] (video_ioctl2+0x1c/0x24 [videodev])
[  689.154051]  r10:0000001d r9:b1efd5dc r8:c0c2f028 r7:c457bd20 r6:00000000 r5:c457bd21
[  689.154086]  r4:bf25bd50
[  689.154105] [<bf25bd50>] (video_ioctl2 [videodev]) from [<bf253148>] (v4l2_ioctl+0x4c/0x64 [videodev])
[  689.154879] [<bf2530fc>] (v4l2_ioctl [videodev]) from [<c02244d4>] (sys_ioctl+0x114/0x9b0)
[  689.155311]  r5:c457bd21 r4:c0445611
[  689.155333] [<c02243c0>] (sys_ioctl) from [<c0008260>] (ret_fast_syscall+0x0/0x1c)
[  689.155382] Exception stack(0xc260dfa8 to 0xc260dff0)
[  689.155414] dfa0:                   b1efd5dc b532a124 0000001d c0445611 b1efd5dc 00000004
[  689.155454] dfc0: b1efd5dc b532a124 b1efda44 00000036 b2712f60 fffffff8 b1efd7f4 b2701638
[  689.155487] dfe0: b50fad6c b1efd56c b5079d90 b6dc3d4c
[  689.155518]  r10:00000000 r9:c260c000 r8:c00083e4 r7:00000036 r6:b1efda44 r5:b532a124
[  689.155553]  r4:b1efd5dc
[  689.155582] Code: bad PC value
[  689.155610] ---[ end trace 761b121f01d6b23c ]---

A look at /proc/meminfo when things were happening normally showed:

$ cat /proc/meminfo
MemTotal:         439752 kB
MemFree:          214120 kB
MemAvailable:     211724 kB
Buffers:            3396 kB
Cached:            32780 kB
SwapCached:         7012 kB
Active:            35992 kB
Inactive:          17428 kB
Active(anon):      11828 kB
Inactive(anon):     5432 kB
Active(file):      24164 kB
Inactive(file):    11996 kB
Unevictable:          16 kB
Mlocked:              16 kB
SwapTotal:        102396 kB
SwapFree:          78332 kB
Dirty:               324 kB
Writeback:             0 kB
AnonPages:         16664 kB
Mapped:            16332 kB
Shmem:                16 kB
KReclaimable:       8196 kB
Slab:              18136 kB
SReclaimable:       8196 kB
SUnreclaim:         9940 kB
KernelStack:         904 kB
PageTables:         1232 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      322272 kB
Committed_AS:     291956 kB
VmallocTotal:     573440 kB
VmallocUsed:        5040 kB
VmallocChunk:          0 kB
Percpu:               64 kB
CmaTotal:         262144 kB
CmaFree:           97052 kB

... also i'm seeing the same amount of CmaFree once the capture software has crashed, so I'm guessing whatever buffers were allocated haven't been unallocated.

Hardware : Platform: Pi Zero W Camera: Pi Camera v3 Distro: Raspbian Bullseye

[davidplowman]

Hi, I think it's more likely to be a power supply issue as we have seen these on other Pi Zeros running libcamera. @naushir might be able to suggest some configuration tweaks that may help.

[darksidelemm]

Could temperature also be a factor? I was noticing that with the camera running continuously, I was quickly hitting >90 degrees C core temperature, and things would start to get a but unstable.

I've moved to only starting the camera on for the few seconds i'm doing image capture and then stopping it afterwards, which has kept the temperatures reasonable (<80 degrees C).

I'll definitely look into power issues, good point on that one.

[naushir]

Can you try adding over_voltage=4 at the bottom of /boot/config.txt, reboot and see if things improve?

[darksidelemm]

Will try that out when I get a chance next. What do different settings for over_voltage do? (e.g. I'm guessing it bumps up some core voltages?)

[naushir]

That's correct, it will help if the board is not drawing enough power at the nominal settings. More details can be found here

[Arnold1]

hi @darksidelemm,

did you fix your issue - if yes how?

how well does picamera2 work with Raspberry Pi Camera Module V3 and Raspberry Pi Zero W? is the cpu of the Raspberry Pi Zero W good enough?

i think of a similar project - make a photo or live video - and send it over wireless or LTE to a server. would that be possible?

Thanks

[darksidelemm]

So this is probably not a memory issue, so I'm going to close the issue to avoid that confusion.

I ended up being able to use picamera2. + PiCam v3 in my application by shutting the camera down between image captures (one capture approx every 40 seconds). This was mainly to lower CPU usage and heat generated, as if I kept it on continuously I would end up overheating the RPi. This was at full resolution however, so at lower resolutions you might be OK.

[Arnold1]

@darksidelemm does it mean you use 2 cameras at the same time? do you have some code to share? i would like to try it to.

wouldnt the Raspberry Pi Zero 2 W better for what you do? - it has a faster cpu...

[darksidelemm]

My code is pretty application specific, though:

• Camera is setup here: https://github.com/projecthorus/wenet/blob/picamera2/tx/WenetPiCamera2.py#L93
• Captures occur here: https://github.com/projecthorus/wenet/blob/picamera2/tx/WenetPiCamera2.py#L144