tturpin
commented
1 year ago I found a kind of "workaround" : having a pool of processes copying the mapped memory in parallel make things faster, at the expense of an insane CPU usage. I'm don't have any background in the domain of hardware, but I find this surprising, it makes me think that there must be a better way.
My testing code starts from with the same "mmapped" buffer (with only gray information), and then it runs up to 4 parallel tasks, each of which copies a slice of the buffer into a pre-allocated numpy array. The resulting framerate is as follows:
- 30FPS with 1 thread (so, there is some slight overhead w.r.t the version without threads)
- 43FPS with 2 threads
- 45FPS with 3 threads
- 46FPS with 4 threads It doesn't run 4x faster unfortunately, and processing the extracted frames will add to the load, but it definitely has some effect and using 2 threads may be worth it. Reducing the resolution to 1920x1080, the 56FPS sensor limit can be reached.
Here is my testing code (ugly, I'm a beginner with Python):
from picamera2 import Picamera2
from picamera2.request import _MappedBuffer
import time
import numpy as np
import concurrent.futures
res=(2304, 1296)
framerate=56.03
picam2 = Picamera2()
config = picam2.create_video_configuration({"size": res},lores={"size": res},controls={"FrameRate": framerate},buffer_count=2)
picam2.align_configuration(config)
picam2.configure(config)
picam2.start()
s=picam2.stream_configuration("lores")["stride"]
(w,h)=picam2.stream_configuration("lores")["size"]
slices = 3
slice_len=int(w*h/slices)
def copy_slice(result, m,i):
start=i*slice_len
stop=(i+1)*slice_len
np.copyto(result[start:stop], np.array(m, copy=False,dtype=np.uint8)[start:stop])
executor = concurrent.futures.ThreadPoolExecutor(max_workers=slices)
class _GrayBuffer(_MappedBuffer):
def __enter__(self):
import mmap
fd = self._MappedBuffer__fb.planes[0].fd
planes_metadata = self._MappedBuffer__fb.metadata.planes
buflen = planes_metadata[0].bytes_used
self._MappedBuffer__mm = mmap.mmap(fd,buflen,mmap.MAP_SHARED,mmap.PROT_READ | mmap.PROT_WRITE)
return self._MappedBuffer__mm
def make_gray(self):
import mmap
fd = self._MappedBuffer__fb.planes[0].fd
planes_metadata = self._MappedBuffer__fb.metadata.planes
buflen = planes_metadata[0].bytes_used
m = mmap.mmap(fd,buflen,mmap.MAP_SHARED,mmap.PROT_READ | mmap.PROT_WRITE)
result = np.empty(buflen,dtype=np.uint8)
future = [executor.submit(copy_slice, result, m, i) for i in range(slices)]
for f in future:
f.result()
return result
def make_gray_buffer(request):
return _GrayBuffer(request,"lores").make_gray()
last_time=time.time()
last_count=0
count=0
while True:
t0=time.time()
count=count+1
if t0 >= last_time + 1 :
print(count-last_count)
last_count=count
last_time=t0
request = picam2.capture_request()
frame=make_gray_buffer(request)
request.release()
gray=frame.reshape((h,s))
davidplowman
catubc
Hi,
I would like to do some processing of high resolution video at the highest possible framerate, but I don't need color information. The sensor modes say that a 2304x1296 resolution is supported with a maximum framerate of 56.03, but it seems that retrieving the frames from picamera2's buffers takes too much time for that, and I don't understand why it is. According to Linux perf profiling, most of the time is spent in __memcpy_generic (one cpu core is always 100% used) and the amount of data to copy does not match the processor's memory capabilities if I understand correctly.
If I naively get RGB frames from the "main" buffer with capture_array, I have 10 FPS. Such frames should be roughly 12MB large with a 4byte pixel size. That's a transfer speed of 120MB per second.
Then I tried to use the YUV format instead, which is more compact and well suited for black and white extraction. Here is a small test:
This yields a 22FPS speed, which is consistent with the data size which is half the rgb size if I'm correct. Finally, I managed to improve speed a little further by modifying request.make_array to only copy the luminance part of YUV frames:
This yields roughtly 31FPS, which is better, but not twice faster, I have no idea why.
But according to "tinymembench", my Raspberry Pi 3B can move 1000MB per second in standard memcpy, which is an order of magnitude higher than what I see. And if I understand how frames handling works, the frames are supposed to be placed in memory by the libcamera framework, and request.make_buffer only copies them to free the buffers...
I'm sure that it's the call to make_buffer which is the bottleneck, because if I only do "capture_request" and "release" without doing anything, I get the expected 56FPS framerate.
So what am I missing ? Is there a fundamental reason why I cannot go faster, or could it be a performance issue in the implementation ?
Thanks !