Closed sam598 closed 2 years ago
davidplowman
commented
2 years ago Hi, thanks for reporting this. Could you please post:
cat /proc/meminfo.Thanks.
sam598
commented
2 years ago MemTotal: 372896 kB
MemFree: 86572 kB
MemAvailable: 157176 kB
Buffers: 8820 kB
Cached: 110548 kB
SwapCached: 11200 kB
Active: 66876 kB
Inactive: 116708 kB
Active(anon): 10168 kB
Inactive(anon): 63744 kB
Active(file): 56708 kB
Inactive(file): 52964 kB
Unevictable: 16 kB
Mlocked: 16 kB
SwapTotal: 2097148 kB
SwapFree: 2041340 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 54124 kB
Mapped: 63548 kB
Shmem: 9696 kB
KReclaimable: 11972 kB
Slab: 29108 kB
SReclaimable: 11972 kB
SUnreclaim: 17136 kB
KernelStack: 1864 kB
PageTables: 4208 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2283596 kB
Committed_AS: 1038700 kB
VmallocTotal: 1687552 kB
VmallocUsed: 6188 kB
VmallocChunk: 0 kB
Percpu: 528 kB
CmaTotal: 262144 kB
CmaFree: 39252 kB# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details
# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1
# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16
# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720
# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1
# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1
# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2
# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4
# uncomment for composite PAL
#sdtv_mode=2
#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800
# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on
# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18
# Additional overlays and parameters are documented /boot/overlays/README
# Enable audio (loads snd_bcm2835)
dtparam=audio=on
# Automatically load overlays for detected cameras
camera_auto_detect=1
# Automatically load overlays for detected DSI displays
display_auto_detect=1
# Enable DRM VC4 V3D driver
dtoverlay=vc4-kms-v3d
max_framebuffers=2
# Disable compensation for displays with overscan
disable_overscan=1
[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1
[all]
[pi4]
# Run as fast as firmware / board allows
arm_boost=1
[all]
gpu_mem=128Occurs with an HDMI display and separate mouse and keyboard over a USB hub, as well as over SSH without the HDMI and USB connected. It is being powered from a 5V 3.4A Max power source over USB power.
Full Raspberry Pi OS 32Bit installed using Raspberry Pi Imager on March 18th 2022.
I don't believe so. pidof X returns nothing.
This was a completely fresh install of the OS for the sole purpose of running Picamera2. Following the install instructions, enabling glamor, increasing the swap space to 2048 and gpu memory to 128. The same errors still occur if I revert back to 64.
sam598
commented
2 years ago The same errors still occur if I revert back to 64.
This was with preview_configuration at full res (which I believe has 3 or 4 buffers). If I run the original script over SSH with display and keyboard unplugged and with GPU set to 64 the script runs normally.
davidplowman
commented
2 years ago The most obvious thing that I see in your /proc/meminfo is that you have very little free CMA. Here are some things to try:
vcgencmd get_mem gpu will tell you how much you have.Then have a look at your /proc/meminfo and let's see if the CMA is in better shape.
sam598
commented
2 years ago I tried a new clean install of Raspberry Pi OS Lite 32Bit. It needs the following things to also be installed:
sudo apt install libatlas-base-dev
python -m pip install numpy
python -m pip install PillowHowever I am getting the same Unable to request 2 buffers: Cannot allocate memory error:
[0:09:37.336817177] [2486] INFO IPAManager ipa_manager.cpp:138 libcamera is not installed. Adding '/home/pi/dev/libcamera/build/src/ipa' to the IPA search path
[0:09:37.353484470] [2486] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3447-c6bb050a
[0:09:37.358872318] [2487] WARN CameraSensorProperties camera_sensor_properties.cpp:141 No static properties available for 'imx477'
[0:09:37.358968464] [2487] WARN CameraSensorProperties camera_sensor_properties.cpp:143 Please consider updating the camera sensor properties database
[0:09:37.359033828] [2487] ERROR CameraSensor camera_sensor.cpp:551 'imx477 10-001a': Camera sensor does not support test pattern modes.
[0:09:37.387341346] [2487] INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/pi/dev/libcamera/src/ipa/raspberrypi/data'
[0:09:37.418727292] [2487] INFO RPI raspberrypi.cpp:1337 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
2022-03-20T21:55:19.648Z | INFO | Initialization successful.
2022-03-20T21:55:19.649Z | INFO | Camera now open.
[0:09:37.424198421] [2486] INFO Camera camera.cpp:1028 configuring streams: (0) 4048x3040-XBGR8888
[0:09:37.424992482] [2487] INFO RPI raspberrypi.cpp:757 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
2022-03-20T21:55:19.655Z | INFO | Configuration successful!
[0:09:37.757330846] [2487] ERROR V4L2 v4l2_videodevice.cpp:1192 /dev/video14[15:cap]: Unable to request 2 buffers: Cannot allocate memory
Traceback (most recent call last):
File "/home/pi/dev/volcapcam/picamtest.py", line 8, in <module>
picam2.configure(config)
File "/home/pi/dev/picamera2/picamera2/picamera2.py", line 458, in configure
self.configure_(camera_config)
File "/home/pi/dev/picamera2/picamera2/picamera2.py", line 447, in configure_
self.logger.critical("Failed to allocate buffers.")
AttributeError: 'Picamera2' object has no attribute 'logger'Here is the output of cat /proc/meminfo on the new card:
MemTotal: 437860 kB
MemFree: 234264 kB
MemAvailable: 335572 kB
Buffers: 21160 kB
Cached: 117136 kB
SwapCached: 6420 kB
Active: 71168 kB
Inactive: 81332 kB
Active(anon): 2052 kB
Inactive(anon): 12616 kB
Active(file): 69116 kB
Inactive(file): 68716 kB
Unevictable: 16 kB
Mlocked: 16 kB
SwapTotal: 2097148 kB
SwapFree: 2088444 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 7908 kB
Mapped: 16712 kB
Shmem: 448 kB
KReclaimable: 17060 kB
Slab: 32440 kB
SReclaimable: 17060 kB
SUnreclaim: 15380 kB
KernelStack: 1016 kB
PageTables: 1004 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2316076 kB
Committed_AS: 131944 kB
VmallocTotal: 1622016 kB
VmallocUsed: 6020 kB
VmallocChunk: 0 kB
Percpu: 528 kB
CmaTotal: 262144 kB
CmaFree: 183592 kBAfter rebooting and running the script from the Lite OS command line using a mouse and keyboard, the script works.
After rebooting and logging in through SSH and running the script, I get this error:
[0:00:50.333655316] [806] INFO IPAManager ipa_manager.cpp:138 libcamera is not installed. Adding '/home/pi/dev/libcamera/build/src/ipa' to the IPA search path
[0:00:50.360200240] [806] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3447-c6bb050a
[0:00:50.365729220] [809] WARN CameraSensorProperties camera_sensor_properties.cpp:141 No static properties available for 'imx477'
[0:00:50.365827338] [809] WARN CameraSensorProperties camera_sensor_properties.cpp:143 Please consider updating the camera sensor properties database
[0:00:50.365892021] [809] ERROR CameraSensor camera_sensor.cpp:551 'imx477 10-001a': Camera sensor does not support test pattern modes.
[0:00:50.394530805] [809] INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/pi/dev/libcamera/src/ipa/raspberrypi/data'
[0:00:50.426442429] [809] INFO RPI raspberrypi.cpp:1337 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media2 and ISP device /dev/media0
2022-03-20T23:54:24.665Z | INFO | Initialization successful.
2022-03-20T23:54:24.666Z | INFO | Camera now open.
[0:00:50.431914797] [806] INFO Camera camera.cpp:1028 configuring streams: (0) 4048x3040-XBGR8888
[0:00:50.432725000] [809] INFO RPI raspberrypi.cpp:757 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
2022-03-20T23:54:24.672Z | INFO | Configuration successful!
[0:00:50.659514349] [809] ERROR V4L2 v4l2_videodevice.cpp:1192 /dev/video14[15:cap]: Unable to request 1 buffers: Cannot allocate memory
[0:00:50.662486063] [809] ERROR RPI raspberrypi.cpp:999 Failed to allocate buffers
2022-03-20T23:54:24.901Z | ERROR | Camera did not start properly.
Traceback (most recent call last):
File "/home/pi/dev/volcapcam/picamtest.py", line 10, in <module>
picam2.start()
File "/home/pi/dev/picamera2/picamera2/picamera2.py", line 491, in start
self.start_(controls)
File "/home/pi/dev/picamera2/picamera2/picamera2.py", line 485, in start_
raise RuntimeError("Camera did not start properly.")
RuntimeError: Camera did not start properly.Logging out of SSH (without rebooting) and running the script using mouse and keyboard input works. Then logging in to SSH (still without rebooting) works.
Rebooting, logging in through SSH, and then running the script using mouse and keyboard fails. Then logging out of SSH and trying the script again with mouse and keyboard works.
Does it make sense for SSH to take up enough memory on the pi to make this big of a difference? Or could it be a permissions/access issue separate from the previous ones?
davidplowman
commented
2 years ago Hi, so this is all sounding very strange. I don't see how an ssh process can affect the amount of CMA that you have. Does running grep -i cma /proc/meminfo show any differences if you run it just before the working or non-working cases?
Another couple of things to check:
dmesg show anything useful? Maybe run sudo dmesg -C first to clear it, then have at look at dmesg after a failure. Is there anything in there that looks like an error?sudo cat /sys/kernel/debug/vcsm-cma/state. This should list all the allocated CMA. Is that showing any differences?sam598
commented
2 years ago It gets even stranger. After booting up the device seems to have about 160000 of cma free, then after running the camera script (or really running anything from command line) another 60000-80000 of cma becomes free. I don't think SSH is to blame, since I was able to run successfully after running grep -i cma /proc/meminfo right after a failed run.
These were a few test runs I did. Success failure is pretty random, but then success is consistent once enough cma is free.
- Boot
- Run grep -i cma /proc/meminfo with mouse + keyboard
CmaTotal: 262144 kB
CmaFree: 165488 kB
- Run script with mouse + keyboard successfully without error
- Login with SSH
- Run grep -i cma /proc/meminfo through SSH
CmaTotal: 262144 kB
CmaFree: 223856 kB
- Run script through SSH successfully without error- Boot
- Login with SSH
- Run grep -i cma /proc/meminfo through SSH
CmaTotal: 262144 kB
CmaFree: 167300 kB
- Run script through SSH succesfully without error
- Run grep -i cma /proc/meminfo through SSH
CmaTotal: 262144 kB
CmaFree: 224568 kB- Boot
- Login with SSH
- Run script through SSH FAILS with error
- Run grep -i cma /proc/meminfo through SSH
CmaTotal: 262144 kB
CmaFree: 203508 kB
- Run script through SSH succesfully without errorThis is what I go at the end of dmesg after 2 failed attempts in a row followed by 2 successful attempts:
[ 12.536996] imx477 10-001a: Device found is imx477
[ 12.537306] imx477 10-001a: Consider updating driver imx477 to match on endpoints
[ 12.758198] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[ 14.142192] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 16.244293] ICMPv6: process `dhcpcd' is using deprecated sysctl (syscall) net.ipv6.neigh.wlan0.retrans_time - use net.ipv6.neigh.wlan0.retrans_time_ms instead
[ 18.186695] Bluetooth: Core ver 2.22
[ 18.186814] NET: Registered protocol family 31
[ 18.186822] Bluetooth: HCI device and connection manager initialized
[ 18.190062] Bluetooth: HCI socket layer initialized
[ 18.190082] Bluetooth: L2CAP socket layer initialized
[ 18.190110] Bluetooth: SCO socket layer initialized
[ 18.256979] Bluetooth: HCI UART driver ver 2.3
[ 18.256998] Bluetooth: HCI UART protocol H4 registered
[ 18.257092] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 18.257325] Bluetooth: HCI UART protocol Broadcom registered
[ 18.894409] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 18.894425] Bluetooth: BNEP filters: protocol multicast
[ 18.894446] Bluetooth: BNEP socket layer initialized
[ 18.915929] NET: Registered protocol family 38
[ 18.963567] cryptd: max_cpu_qlen set to 1000
[ 28.121337] cma: cma_alloc: alloc failed, req-size: 12018 pages, ret: -16
[ 28.121362] bcm2835-isp bcm2835-isp: dma_alloc_coherent of size 49225728 failed
[ 33.755947] cam1-reg: disabling
[ 33.755969] cam-dummy-reg: disabling
[ 43.595961] cma: cma_alloc: alloc failed, req-size: 12018 pages, ret: -16
[ 43.595989] bcm2835-isp bcm2835-isp: dma_alloc_coherent of size 49225728 failedsudo cat /sys/kernel/debug/vcsm-cma/state doesn't list anything under resources.
davidplowman
commented
2 years ago I've been trying some of this as well, and I do see slightly strange behaviour in that the free CMA seems somewhat variable, and indeed can occasionally go up. We'll investigate some more.
In the meantime, there's probably some more advice I can give. The default image formats that Picamera2 (currently) chooses are not always the best ones. By default it selects RGBA type images because those are the only ones that work with all the different kinds of preview windows as well as the hardware video encoder, in OpenCV and the usual Python still image encoders. But they are also much the most memory hungry ones.
So you would almost certainly have more luck if you set the format like this:
config = picam2.still_configuration(main={"format": "RGB888"})In fact this format works OK with the DRM/KMS preview window (which is what you would get if you aren't running X Windows), and also in those various image encoders, so there's probably no reason not to use it. But there's no way round the fact that on a Pi Zero 2 you're going to have to be a bit careful - a single 12MP RGBA buffer occupies 48MB (an RGB only buffer is better, but still 36MB each), and the camera system needs several to keep the frames flowing through.
Do you have a particular application in mind? As you can see, there are definite pros and cons to some of the image format and resolution choices, and they can vary depending on the platform.
sam598
commented
2 years ago Thank for the the additional advice. Anecdotally specifying the format seems more reliable, but I do still occasionally get the error after bootup. But after a minute or two the error just seems to go away without any intervention.
Could it be that there is some background process from boot up that is still freeing up memory after logging in? Or perhaps even the camera itself needs more time after the pi has been turned on?
The project I am working on is synchronized still photography of multiple Pi HQ Cameras. I am planning to use the external sync signal feature (https://forums.raspberrypi.com/viewtopic.php?t=281913) but haven't gotten to that point yet.
For my particular use case I'm not planning on doing any realtime previews on the pi itself, and I really only need the first received frame from the camera (exposure, gain and whitebalance would all be manually fixed and set in the start command). I am curious if there is a way to just use 1 buffer for this use case, but it seems like the library and API are built around a sequential camera feed.
davidplowman
commented
2 years ago Yes, we're going to look into the mysterious CMA issues, it feels somewhat like a fragmentation issue though it's not clear to me how it's happening.
You should be able to get by with a single set of buffers (pass buffer_count=1). Normally we allocate at least 2 because some of the preview windows hang on to a buffer until the next one arrives, so that would clearly grind to a halt otherwise. But one buffer should be fine so long as you don't use those preview windows and don't mind dropping frames in between. The way these cameras work is that by the time it gives you a frame the camera is already writing the next one to memory. With a single buffer the system will end up dropping frames until it gets the buffer back again. But it doesn't sound like this is an issue for you.
davidplowman
commented
2 years ago Just to update you with our progress on this. We have found that the Linux kernel is a bit over-zealous in trying to use CMA memory for itself, and this particularly afflicts platforms where the CMA area is at least half the the total system memory. This clearly includes the Pi Zero 2 as the CMA area defaults to 256MB and the total memory is 512MB.
So we'll put in a patch to improve the situation there, and I've verified that the original script starts reliably now. That will appear through sudo rpi-update in due course. Though on the lower memory platforms, tailoring the pixel formats, buffer counts and sizes to the specific use-case, is always likely to remain something to watch out for.
davidplowman
commented
2 years ago The latest images (released yesterday) should have this kernel fix in, in fact when I installed a fresh OS Lite onto a Pi Zero today, updated and rebooted it, it reports 240MB free CMA (out of 256MB) right away.
Armed with a bit more information about formats (for example prefer "RGB888" if you aren't using the X-Windows OpenGL accelerated preview) and the number of buffers you need, I'm going to mark this one as resolved. But please get back to us if you're still experiencing problems or would like more advice in this area. Thanks!
sam598
commented
2 years ago I installed a brand new install of the latest official OS Lite 32. After startup the original example code I posted at the beginning of this issue failed for the first 10-15 seconds.
But as expected this modified version works even immediately after startup. So far I have tested it 10 reboots in a row without issue.
from picamera2.picamera2 import *
import time
picam2 = Picamera2()
picam2.start_preview()
config = picam2.still_configuration(main={"format": "RGB888", "buffer_count": 1})
picam2.configure(config)
picam2.start()
time.sleep(2)
image = picam2.capture_file("test.jpg")
picam2.stop()Thank you again for all of your help!
davidplowman
commented
2 years ago Hmm, maybe I'm wrong about that fix being in the latest image. If you could bear to perform one more experiment....
Would you be able to run sudo rpi-update (don't do this on an image with anything on it that you couldn't bear to lose), re-boot and retry the original script? That really will have that fix, so we could judge whether the behaviour is the same.
Miniontoby
commented
1 year ago I installed a brand new install of the latest official OS Lite 32. After startup the original example code I posted at the beginning of this issue failed for the first 10-15 seconds.
But as expected this modified version works even immediately after startup. So far I have tested it 10 reboots in a row without issue.
from picamera2.picamera2 import * import time picam2 = Picamera2() picam2.start_preview() config = picam2.still_configuration(main={"format": "RGB888", "buffer_count": 1}) picam2.configure(config) picam2.start() time.sleep(2) image = picam2.capture_file("test.jpg") picam2.stop()Thank you again for all of your help!
Your
config = picam2.still_configuration(main={"format": "RGB888", "buffer_count": 1})should be
config = picam2.still_configuration(main={"format": "RGB888"}, buffer_count=1)At least that's what worked by me
With a Raspberry Pi Zero 2 W running Buster 32bit and an HQ Camera this is the code I am using:
and this is the output:
Increasing the buffer size at full resolution to 3 or greater:
config = picam2.still_configuration(buffer_count=3)Causes a different error:
Lowering the resolution by half works:
config = picam2.still_configuration(main={"size": (2028, 1520)})Preview configuration also works at half resolution:
config = picam2.preview_configuration(main={"size": (2028, 1520)})But a preview configuration at full resolution results in the same
Not enough buffers provided by V4L2VideoDeviceerror:config = picam2.preview_configuration(main={"size": (4056, 3040)})