search

raspberrypi / firmware

This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
5.06k stars 1.68k forks source link

V4L2 ISP produces green frames when resizing 9248x6944 (64MPix) images to some output resolutions #1844

Closed 6by9 closed 3 months ago

6by9 commented 6 months ago

From https://github.com/raspberrypi/linux/pull/5708#issuecomment-1810575998

Arducam's new 64MP camera causes the ISP to produce green images.

I've put together a basic V4L2 M2M app to load the raw file it produces and feed it into /dev/video12. Resizing to the 1280x720 logs a firmware side assert

2900265.970: assert( Too many tiles for allocated space ) failed; ../../../../../middleware/ISP/core/isp_tilecalc.c::isp_calc_tile_parameters line 523
----------------
2900266.020: assert( Finalise failed ) failed; ../../../../../middleware/openmaxil/components/isp.c::isp_set_buffers line 3693
----------------

Throughput is impressive as it claims to complete the frame in around 280usec for 9248x6944 to 1280x720! Conversion to 4624x3472 works, but takes a more leisurely 165ms.

ISP_MAX_TILES_PER_FRAME is defined as 650. We may be able to just increase it.

6by9 commented 6 months ago

Resize to 1280x720 and 1280x960 both need 45x15 = 675 tiles, exceeding the allocation of 650. 3840x2160, 3840x2880, 1920x1440 and 1920x1080 all need 44x15 = 660 tiles.

@naushir @davidplowman Increasing the number of tiles appears to work, and it looks to only be a memory allocation thing, so how high do we go? Memory says that cropping (ie digital zoom) increases the number of tiles, but that could be my memory failing, and if true I have no idea by how much it would increase.

naushir commented 6 months ago

675 tiles is crazy!! I see no reason not to bump the limit up, maybe to 700 or 750?

6by9 commented 6 months ago

Crazy high, or crazy low?!

I did hear mutterings about a ~120MPix sensor, and I haven't looked at how many tiles that would require. Reality is that you'll hit more issues with the size of the CMA heap on vc4 before that is realistic, and the rolling shutter effect when readout is likely to be almost 1sec is likely to make it not possible on Pi4, so probably ignore it for now.

I'll see if I can get some malloc figures to see how much the memory usage goes up by for 750 tiles.

naushir commented 6 months ago

Crazy high, or crazy low?!

Crazy high. It ought to work, but that's a lot of overhead.

6by9 commented 6 months ago

Updated firmware at https://drive.google.com/file/d/1n_e-qFXIjDcdz8cuXgVSPtO404FHd-5R/view?usp=sharing that should allow for 800 tiles.

6by9 commented 3 months ago

Fix merged into the firmware tree 28th Nov 2023. Closing.