Apply zoom only to specific splitter port

[philipp-mohr]

Hi, is there a way to apply the zoom feature only to one splitter port for efficient processing of ROI, but still record on another splitter port the full image to a file/stream?

I applied the zoom+resizing in python+opencv but this costs too much of processing time.

thanks in advance and best regards

[6by9]

Potentially. This is a function of the firmware, not Picamera. The current zoom/crop is done within the hardware processing for the camera.

video_splitter is just sending that image to multiple places (potentially with format conversion). It does not support cropping at that point (although it could in theory, except it throws up a load of issues).

resize does support cropping, although it appears not to be exposed via MMAL (only IL). I've got some other changes to make in the MMAL/IL interface code, so I'll add that in too.

I need to expose it on the ISP component too at some point soon. I can't recall whether PiCamera is using resize or ISP by default at the moment - first glance seems to say resize (encoders.py).

[philipp-mohr]

Would be awesome if you can implement that feature. As you have probably seen I am using the resize option inside the camera.capture_continuous method. Will I be able to adjust the zoom/crop every iteration of the for loop with instant effect on the next frame, when you have exposed cropping in the resize method?

More details on my application: I want to process images with open cv to track an object in front of the camera. To do this more efficiently I resized the image to a lower resolution. When the object gets far away from the camera it is lost in the pixels of the resized image. To resolve this I want to adjust the zoom of the camera after every processed frame.

from picamera.array import PiRGBArray
import picamera
import time

camera = picamera.PiCamera()
camera.resolution = (1640, 922)
camera.framerate = 25
time.sleep(0.5)
size_resize = (213, 120)
rawCapture = PiRGBArray(camera, size=size_resize)

for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True,
                                               resize=size_resize,
                                               splitter_port=3)
    frame = frame.array
    new_zoom_parameters = get_zoom_parameters_to_center_object(frame)
    camera.zoom = new_zoom_parameters
    rawCapture.truncate(0)

[waveform80]

I need to expose it on the ISP component too at some point soon. I can't recall whether PiCamera is using resize or ISP by default at the moment - first glance seems to say resize (encoders.py).

@6by9 It's using resize at the moment. I did try using ISP but there was some issue I ran into (sorry, can't remember off the top of my head - something about interactions with the splitter I think; anyway gpio-zero and another project are currently taking up all my time but I should get some time to devote to picamera in a few weeks).

Will I be able to adjust the zoom/crop every iteration of the for loop with instant effect on the next frame, when you have exposed cropping in the resize method?

@piveloper Unlikely: in your other ticket (#421) you noted there's a delay between assignment to the zoom property and application of that property. I'd assume there'll be a similar delay in any implementation here. That said, from your description in #421 I would've thought recording at multiple resolutions would've been preferable but maybe I'm mis-understanding something. As mentioned above I'll try and have a more thorough look at the open tickets in a few weeks.

[philipp-mohr]

@waveform80 The problem with multiple resolution (a lower one for processing) is that if the object gets far away, the resolution is too small for any recognition. Therefore my approach has been to zoom into the raw high resolution image and process a resized version of the zoomed image. Due to movements of the object the centerpoint of the zoom has to be adjusted constantly.

I implemented the zoom/cropping with open cv on the high resolution image but the pi is too slow to handle such large IO in Python at acceptable framerates. So my idea was to outsource this work in a more efficient way outside the python+opencv loop. But doing this with camera.zoom leads to the mentioned delay of some frames, which would make this approach useless.

According to @6by9 the camera.zoom is done in the picamera driver. Would have hoped that doing the zoom more at the end of the processing pipeline(maybe ISP accelerated) would cause a reduced and more predictable delay.

[6by9]

@waveform80 Thanks the confirmation of using resize. I can fully understand the lack of time with other projects. Shout if/when you're next looking at these things as ISP component is still being developed.

I have changes pending at the office that should expose the crop parameters supported through resize. I'll try to get them sorted tomorrow along with a bundle of other things. Adding it to ISP may take a few more days.

@piveloper I think resize doing the crop may give you what you want, but I'm not going to even attempt to look at adding functionality to the Python side - too many bad memories of using Python on a previous project. I'll leave that bit up to waveform80 :-)

[vrosca]

a quick lazy question - with the latest userland code, can cropping be done by the resizer component downstream of a splitter?

Tried setting the resizer output format as below, seems to be ignored (both vc.ril.resize and vc.ril.isp behave the same way) - outputPort.format.es.video.crop.x = x; outputPort.format.es.video.crop.y = y; outputPort.format.es.video.crop.width = width; outputPort.format.es.video.crop.height = height;

Do I need to set a different parameter on the resizer component? One of the new ones as in (https://github.com/raspberrypi/userland/commit/29835184aa70c13d77a11d721e934a7298bdf46a#diff-cd294817b3d6afc602205fbf6cce5450)? From a cursory look, that seemed to apply to the camera component

[6by9]

format.es.video.crop.x and format.es.video.crop.y are ignored. format.es.video.crop.width and format.es.video.crop.height define the size of the active area of the buffer (format.es.video.width and format.es.video.height have to be aligned, whereas the crop parameters don't).

MMAL_PARAMETER_CROP is the same as the IL OMX_IndexConfigCommonInputCrop when used on input ports (OMX_IndexConfigCommonOutputCrop on output ports, but that isn't used anywhere).

It appears to be supported on isp, resize, and video_render. isp and resize crop the input before resizing to the format requested. video_render uses it to set up the source rectangle to be displayed iff it hasn't been set through OMX_IndexConfigDisplayRegion.

[vrosca]

talk about fast feedback, thank you! If I got this straight, I should try to set MMAL_PARAMETER_CROP on the resizer input port and see what comes out of it.

[6by9]

If I got this straight, I should try to set MMAL_PARAMETER_CROP on the resizer input port and see what comes out of it.

That looks right. Give it a go!

[vrosca]

I ended up with an MMAL_ENOSYS (Function not implemented).

Do I need to upgrade the firmware?

vcgencmd version Sep 21 2016 13:17:10 Copyright (c) 2012 Broadcom version 2eaf52cc53435b5ce67253af1487f9a4f9f96e2d (clean) (release)

And any way to upgrade to the latest firmware while still staying on kernel 4.4 ?

Thanks

PS: I built and installed the latest userland from source, that worked fine

[6by9]

You need to be on firmware since July 11th 2017

SKIP_KERNEL=1 sudo rpi-update a4e1434 would get that specific release or just SKIP_KERNEL=1 sudo rpi-update will get the latest firmware. (Usual warnings apply over the use of rpi-update).

If you want to do it manually, then dowload the .dat and .elf files from https://github.com/Hexxeh/rpi-firmware

[vrosca]

Thanks a lot for your help - can confirm it works! Now feeding two h264 video encoders in parallel. I'll have to check why it seems to jam after a few minutes - will write back here if it is indeed linked to the resize.

I assume setting the new param MMAL_PARAMETER_RESIZE_T / MMAL_RESIZEMODE_T with MMAL_RESIZE_CROP will result in no stretching?

[6by9]

TBH I can't remember with the resize component.

Notes for the MMAL_PARAMETER_RESIZE_T are at https://github.com/raspberrypi/userland/blob/master/interface/mmal/mmal_parameters_camera.h#L919, but are pretty much lifted from the IL docs. Only if you set preserve_aspect_ratio will it avoid stretching. That is all around automatically setting the output buffer parameters based on the input port format. There's nothing stopping you setting it manually. AIUI You are correct that MMAL_RESIZE_CROP will crop instead of stretching, but it'd take a bit of digging through the code to really confirm that.

NB I believe this is all orthogonal to MMAL_PARAMETER_CROP.

[vrosca]

Thanks again - just want to add that the jams have nothing to do with the resize component, they happen due to buffering all frames between two keyframes at application level - with the old firmware, keyframes occurred quite frequently, even when filming static scenes, while the new firmware seems to happily allow tens of thousands of regular frames, until memory is exhausted.

[6by9]

Intra I frame rate is configurable. Default should be 60 frames. You're posting under the PiCamera repo - are you actually using the Python library as you seem to be hacking around at the MMAL level? In PiCamera you're looking at intra_period property mentioned at http://picamera.readthedocs.io/en/latest/api_encoders.html#picamera.PiVideoEncoder Under MMAL you're looking at MMAL_PARAMETER_INTRAPERIOD.

[vrosca]

Thanks I'm familiar with that flag. Guilty as charged, I'm actually hacking around the MMAL level with a JNI/JNA java binding. Was actually trying not to further hijack this thread - however your helpfulness makes that difficult :)