[HOW-TO] capture cropped Preview as still or video

[RobertFlatt]

Please only ask one question per issue! [HOW-TO] capture cropped Preview as still or video?

Describe what it is that you want to accomplish The docs say "ScalarCrop" can be used to implement a zoom, this works. How to capture a zoom'd preview as a photo or video?

Describe alternatives you've considered The zoom is lost when the configuration changes - so the provided capture routines destroy the zoom info and always capture full size.

Additional context This issue is related to https://github.com/raspberrypi/picamera2/issues/492 in the sense that (even with that issue's proposed workaround) the feature is available for a preview only, not for capture.

[yingliuzhizhuzed]

I feel that from the perspective of picamera2, the flat preview operation cannot affect the capture and recording. They may act on different ports. I don't know much about C/C++, so code like libcamera makes me headache (- _ -)

[RobertFlatt]

In my experiments the 'main' and 'lores' streams zoom together. I didn't look at '/dev/video11'. which would be important for video encoding.

The issue is methods like switch_mode_and_capture_image loose the zoom info due to the 'switch_mode', so the captured image is not zoom'd and on return from that method the preview is no longer zoom'd,

[davidplowman]

Hi, thanks for the report. Indeed, controls are generally meant to be persistent across mode switches so it doesn't seem quite right. I'll have to go and look down into what libcamera itself is doing there. In the meantime, does this kind of thing help as a workaround?

import time
from picamera2 import Picamera2

picam2 = Picamera2()
preview_config = picam2.create_preview_configuration(controls={"ScalerCrop": (1000, 750, 2000, 1500)})
picam2.configure(preview_config)
picam2.start(show_preview=True)
time.sleep(1)
capture_config = picam2.create_still_configuration(controls={"ScalerCrop": (1000, 750, 2000, 1500)})
picam2.switch_mode_and_capture_file(capture_config, "test.jpg")

(Tested on an HQ cam)

[RobertFlatt]

The issue is, I think, thread synchronization. It appears to behave as if this is executed too late https://github.com/raspberrypi/picamera2/blob/main/picamera2/picamera2.py#L970

Workaround is based on this https://github.com/raspberrypi/picamera2/blob/main/apps/app_full.py#L78-#L85

This works for video, this preview is cropped and stays cropped during configuration change, and the recorded video is cropped. In place of picam2.switch_mode() use this switch_config()

The double call to self.update_scalar_crop() is required. I think because of some sync with libcamera.

    def switch_config(self, new_config):
        scalar_crop = self.get_scalar_crop()
        self.picam2.stop()
        self.picam2.configure(new_config)
        self.update_scaler_crop(scalar_crop)
        self.picam2.start()
        self.update_scaler_crop(scalar_crop)

    def get_scalar_crop(self):
        meta = self.picam2.capture_metadata()
        scalar_crop = None
        if 'ScalerCrop' in meta:
            scalar_crop = meta['ScalerCrop']
        return scalar_crop

    def update_scaler_crop(self, scaler_crop):
        if scaler_crop:
            with self.picam2.controls as controls:
                controls.ScalerCrop = scaler_crop

However if I try this for photo (I can live with the extra latency),

                self.switch_config(self.photo_config)
                self.picam2.capture_file(path)
                self.switch_config(self.preview_config)

The preview is good, but NEW ISSUE:

The file capture has the correct cropped contents, but save assumes an aspect ratio of 4:3. This fails e.g when crop portrait from landscape, in this case the image is horizontally stretched, because the width and height info is ignored. I think because request.helpers.make_array() depends on config["size"] which is the un-cropped size, which may or may not have the same aspect ratio as the crop.

This is inconsistent with video, I cropped a 4:3 stream to 16:9 and recorded a 16:9 video. Different test case, but this is the same behavior that is missing in the photo case.

I perhaps confirmed this make_array behavior for a photo_config, using this test which assumes I know the crop is portrait 3:4 (not generally true). The test was successful in that it captured a portrait 3:4 image (content was offset, test has side effects - it is not a workaround)

     self.picam2.set_controls({''size} : (size[1]*3//4 , size[1]))

The placeholder arguments for width and height are there in CompletedRequest, they are just not used or exposed.

[davidplowman]

Hi, thanks for the update. If I could perhaps ask to clarify a few points:

• Does the workaround I posted above work for you or not, it wasn't quite clear to me whether you had tried it.
• Could you perhaps post the simplest test case for the original problem, perhaps just as a simple script (no Qt application, rather like what I posted), then I can be sure I'm running the same thing as you. Please also say what camera you are using.
• This new issue that you have raised, this is unrelated? Perhaps we could file that as a new bug as I'm a bit confused as to what issues we actually have here now.
• Could you also post the simplest test case you can for this new issue? Posting really simple but complete test cases really helps because I can be confident that I'm actually investigating the right problem.

Thanks!

[RobertFlatt]

Does the workaround I posted above work for you or not, it wasn't quite clear to me whether you had tried it.

I didn't try it as it does not address the issue. Because this test assumes an invariant crop, any success would be a false positive.

Could you perhaps post the simplest test case for the original problem

To your test case append:

crop = picam2.capture_metadata()['ScalerCrop']
with picam2.controls as controls:
     controls.ScalerCrop = (crop[0] //2 , crop[1] //2 )  
time.sleep(2)                      # makes it easier to see the issue in a display
capture_config_1 = picam2.create_still_configuration()
picam2.switch_mode_and_capture_file(capture_config_1, "test_1.jpg")

test_1.jpg should be different from test.jpg , but it is not. You may also see the issue in an associated display stream, where the display crops then incorrectly un-crops.

This new issue that you have raised, this is unrelated? Perhaps we could file that as a new bug as I'm a bit confused as to what issues we actually have here now.

Two different issues in pc2 implementation. This one is about passing crop information between configurations and ultimately I think in part a thread synchronization issue. The other one about jpg size and hence aspect ratio when saving certain cropped images. Both are illustrated using 'ScalerCrop'

Could you also post the simplest test case you can for this new issue? Posting really simple but complete test cases really helps because I can be confident that I'm actually investigating the right problem.

https://github.com/raspberrypi/picamera2/issues/505

[davidplowman]

Hi, I tried pasting together my example with the alteration you posted above, but I'm still not sure that I'm recreating the problem you want me to look at.

By pasting these two bits together, I get the preview running with the defined crop, and the still configuration is trying to apply even more crop. Is this right, or should the still crop merely be the same as the preview crop? Also, setting the ScalerCrop to a tuple of just 2 values fails, there should be 4.

Currently I'm working on the assumption that you want a different crop in the capture than in the preview, so my test case looks like this:

import time
from picamera2 import Picamera2

picam2 = Picamera2()
max_size = picam2.camera_properties['PixelArraySize']
crop = (max_size[0] // 4, max_size[1] // 4, max_size[0] // 2, max_size[1] // 2)
print("Crop:", crop)
preview_config = picam2.create_preview_configuration()
picam2.configure(preview_config)
picam2.start(show_preview=True)
time.sleep(2)

picam2.set_controls({"ScalerCrop": crop})
capture_config = picam2.create_still_configuration()
picam2.switch_mode_and_capture_file(capture_config, "test.jpg")

And the problem is related to whether the jpg is cropped or not? Anyway, if you could confirm and we can agree on exactly what the test case is, that would be a big help.

[davidplowman]

To add a few more thoughts on this test case.

There is an issue with whether the new crop gets seen and handled in time. If it gets sent to libcamera before the camera stops then it will either happen correctly (if there is still some time before the camera is stopped), or it could get lost (if the camera is stopped before libcamera gets round to processing it). In the former case, there's the additional bug that I mentioned where the crop is not preserved across a mode switch, which it should be.

Then there's the case where, if the system is stopped before it's even sent to libcamera, then the crop will get applied as part of the configure process. I think this is probably a bug, and any pending controls should be cleared out when a new configure happens.

The workarounds for the problem would be either to wait for the crop to take effect or to supply the crop to the new configuration (so you'd have capture_config = picam2.create_still_configuration(controls={"ScalerCrop": crop}).

[RobertFlatt]

A short response, I will try to be more detailed in a couple of days. Among other things a picamera2 update arrived and my custom preview suddenly had bit rot - fixed but too much going on here.

There is an issue with whether the new crop gets seen and handled in time.

This is I think the root of the issue, and I suspect why the rather strange double assignment in app_full.py is a work around.

The workarounds for the problem would be either to wait

Yes, but consider the case of capture between zoom events, where the zoom events are close in time due to being triggered by a scroll wheel. The app can't wait because these are UI events, it could possibly recreate the configuration with each wheel click but this doesn't get to the cause of the issue. The workaround from full_app.py (in earlier post) works well.

[ Getting to the cause matters because there is a related behavior with the zoom.py example. I copied that code into my app and triggered it with a scroll wheel - and lost events with a fast scroll (I expect you can see the same thing by decreasing the delay - I didn't try). The value of ScalerCrop is read {perhaps several times} before it is updated. The workaround is again found in app_full.py. Where the zoom state is never read from ScalerCrop - only written. The app keeps a local copy of the state which is guaranteed to be current when read. ]

[davidplowman]

Thanks for the feedback.

Yes, there are indeed various things going on.

There's a general behaviour in libcamera that controls have to be passed with a request, but you're not guaranteed when those controls will be applied. The official answer is, I believe, that you have to wait to see the result in the metadata to be sure they've happened before you stop (or mode-switch) the camera. (There is work going on to make it easier to know when controls have happened, but that's still quite some way out.)

zoom.py uses capture_metadata to synchronise scaler crop updates with camera frames - if you remove it then the zoom is "instant". You can do exactly the same thing in a Qt app, if you want. Even this doesn't totally guarantee you'll get one zoom step per frame, but you can expect it to be reasonably close. (Note that you can't wait for each step to be applied before sending the next - the pipelined nature of controls means this would be really slow.)

I've talked about the bug where the scaler crop doesn't get preserved across mode switches, and I will get round to fixing that at some point, but I still think the best workaround (and which applies even when that is fixed) is to store the scaler crop that you want in the application, and to re-apply it when you create the capture mode.

[RobertFlatt]

to store the scaler crop that you want in the application

Got it, thats OK

Now the question becomes what is the earliest I can get ScalarCrop. I think the answer is immediately after picam2.start() . If this is correct I have the same read ScalarCrop latency issue in a different place.

To estimate ScalarCrop at start time I can use picam2.sensor_resolution for size, but it looks like the ScalarCrop align offset is in units of stream resolution not sensor resolution. See below.

I can see why it might be, in the general case the relationship between sensor and stream resolution is possibly not simple. See https://github.com/raspberrypi/picamera2/blob/main/picamera2/picamera2.py#L461 But mixed units in ScalerCrop seems unlikely as it could be a complex hardware implementation.

Could you clarify the units of ScalerCrop indexes 0 and 1? Thanks.

config[stream_name]['size'] == (1640, 1232)    #before align_configuration
config[stream_name]['size'] == (1632, 1232)    #after align_configuration
picam2.sensor_resolution == (3280, 2464)
# In the next line Index 0 and Index 2 appear to be in different units 8+3264==3272 not 3280
picam2.capture_metadata()['ScalerCrop'] == (8, 0, 3264, 2464)  

[davidplowman]

Hi, getting hold of the scaler crop before starting the camera is indeed not quite straightforward.

You can ask for picam2.camera_controls['ScalerCrop'] after doing configure (but before start). Doing it before configure would give you just placeholder values. Anyway, the maximum value there (the 2nd item in the tuple) will be what the first frame will have, after being adjusted for the aspect ratio of the output image.

I note that the third value in the tuple (nominally the default) is reporting None, which seems a bit unhelpful. I'd have thought libcamera ought to be able to put in there the value that you want, so I'll put that on my list of things to look at.

Another alternative would be to check picam2.sensor_modes, which lists all the available camera modes along with this crop value for each mode. Of course you don't generally know which of the modes will actually get chosen, though you can force the choice of a mode by specifying the raw parameter in create_xxx_configuration.

The scaler crop is always specified in units of the pixel array size (picam2.camera_properties['PixelArraySize']). This is nearly always the same as picam2.sensor_resolution. The latter only differs for some unusual cameras that advertise no full resolution modes, where it gives you the resolution of the largest mode instead. The only camera like this, to the best of my knowledge, is the new V3 camera, and only when operating in HDR mode.

[davidplowman]

Just to answer the 3264+8 != 3280 question, it's because the 3264 pixels are centred within the full 3280. So 8 + 3264 + 8 == 3280!

[RobertFlatt]

Just to answer the 3264+8 != 3280 question

I should have seen that, thanks for your patience.

picam2.sensor_modes[whatever]['crop_limits']

Worked for me, but it is (reasonably) the unaligned crop, so the first two locations are ( 0, 0,

If I then set ScalerCrop with those zero offsets (it sticks, I can read back) the image is good. But if I don't align_configuration the image is scrambled. So the align operation is critical, but the result can be ignored!!! Presumably there is some other magic going on.

Question: Is it reasonable for me to calculate the crop offset from the stream offset, scaled by stream size to 'crop_limits' ratio (rounded) ?

picam2.camera_controls['ScalerCrop']

The second element of the tuple is not something I can index or iterate so this was a dead end.

This thread turned into a long story. Subject to my one question above, I think I'm complete and I'd suggest close.

[davidplowman]

... If I then set ScalerCrop with those zero offsets (it sticks, I can read back) the image is good. But if I don't align_configuration the image is scrambled. So the align operation is critical, but the result can be ignored!!! Presumably there is some other magic going on.

Are you able to provide a simple standalone script that I can run to demonstrate the problem? If you could do that then I could give a precise answer, and even fix the problem if there is one.

To answer more vaguely, in general things are supposed to work whether "aligned" or not. The problem with "unaligned" images is that they have "padding bytes" on the end of each row. Some software understands this, some software doesn't and thinks that the length of each row must correspond exactly to the number of pixels in it. Even when downstream processing doesn't understand padding, it can of course always be removed, though at some expense. So it can all depend on what is happening downstream with the image.

Question: Is it reasonable for me to calculate the crop offset from the stream offset, scaled by stream size to 'crop_limits' ratio (rounded) ?

I'm not completely sure what you mean by "the crop offset from the stream offset". Would it be possible to provide a little code snippet showing exactly what these values are and what the calculation is? I'd then be able to give you a precise answer.

Thanks!

[RobertFlatt]

Are you able to provide a simple standalone script that I can run to demonstrate the problem?

Not really a problem, just a comment on random strangeness. I saw this with my preview (currently using yuv420 and opengl - that may change). So it may be my issue, and simple standalone would include Kivy,.....

I'm not completely sure what you mean by "the crop offset from the stream offset"

This is easier

#
# In my case ScalerCrop state is kept in the app, due to a latency issue.
# Particularly I want to know ScalerCrop before picam2.start()
#
# This example is obviously trivial, as it tries to replicate the
# default behavior. And hardcodes an assumption of v2.1
# But consider some arbitary initial crop.
#

from picamera2 import Picamera2

picam2 = Picamera2(0)
preview_config = picam2.create_preview_configuration(
    main = {'size' :(1640,1232)})              # (1640,1232) assumes v2.1
picam2.configure(preview_config)

size_before = preview_config['main']['size']   # (1640, 1232)
picam2.align_configuration(preview_config)            
size_after = preview_config['main']['size']    # (1632, 1232) 

# Index 5 assumes v2.1
crop_limits = picam2.sensor_modes[5]['crop_limits']  # (0, 0, 3280, 2464)

scale = crop_limits[2] / size_before[0]
crop_offset_x = ((size_before[0] - size_after[0]) * scale) // 2
crop_offset_y = ((size_before[1] - size_after[1]) * scale) // 2
scaler_crop = (int(crop_offset_x), int(crop_offset_y),
               crop_limits[2], crop_limits[3])       # (8, 0, 3280, 2464)

# The ScalerCrop estimate is good enough, use it.
# But is it good enough in the general case ???????
picam2.controls.ScalerCrop = scaler_crop 

# picam2.start()   # and so on....

[davidplowman]

Hi, I think that code looks OK. The only thing I can see that's missing would be to add the mode's offsets. In this particular case they're zero and so don't matter, but more generally I would use

scaler_crop = (int(crop_offset_x) + crop_limits[0], int(crop_offset_y) + crop_limits[1],
              crop_limits[2], crop_limits[3])       # (8, 0, 3280, 2464)

That should work for non-full FoV modes as well.

There are of course other ways to accomplish this. After the "align" operation, you could adjust the output height so that you're back to your original aspect ratio, but I don't really see any advantage either way.

I guess the final thing is the question of why it's necessary to align the output size. As I said before, things should really "just work", though it can depend on what's happening to the image downstream.

[RobertFlatt]

Thanks.

I have 2/3 of a prototype but I'm kinda burnt out on this, I'm going too do something more productive and come back another day.