[HOW-TO] Reduce Rolling Shutter with HQ-Camera

[mmoollllee]

We want to use the HQ-Camera attached to a slightly moving object. Currently we're facing blurry parts in the image, due to rolling shutter of the camera.

How can we reduce this effect as good as possible, operating with a RPi 3A+? Is the ExposureTime in metadata the only value we have to consider to reduce?

Hope to find some tipps on how to make the most out of it :)

Our Code:

# Limit ExposureTime to reduce blur due to GlobalShutter by daytime
if (timeofday == "day"):
   AeExposureMode = controls.AeExposureModeEnum.Short
   FrameDurationLimits = (50, 5000)
else:
   AeExposureMode = controls.AeExposureModeEnum.Long
   FrameDurationLimits = (100, 10000000)

with Picamera2() as picam2:
   picam2.start(show_preview=False)

   picam2.set_controls({
      "AeFlickerMode": controls.AeFlickereModeEnum.FlickerOff,
      "AeMeteringMode": controls.AeMeteringModeEnum.Custom if metering else controls.AeMeteringModeEnum.Spot,
      "AeExposureMode": AeExposureMode,
      "FrameDurationLimits": FrameDurationLimits
   })

   # auto exposure might take a few moments settle
   time.sleep(0.5)

   ctrl={}
   capture_config = picam2.create_still_configuration(ctrl)
   time.sleep(2)

   metadata = picam2.capture_metadata()

   picam2.switch_mode_and_capture_file(capture_config, "file.jpg")

   picam2.stop()

[davidplowman]

Yes, I think the exposure time is the only thing that will affect the blurriness of the final image. You can keep this short by setting it directly (and the analogue gain will adapt according to the exposure time chosen), or by selecting an appropriate exposure mode.

I notice that you've chosen the "short" exposure mode which is an appropriate thing to do. If you wanted, you are free to edit the parameters of the "short" exposure mode in the camera tuning file (search for imx296.json on the Pi's filesystem).

FrameDuration can limit the exposure time, so it's a reasonable way to constrain what the AE algorithm will do, but won't directly affect blurriness (except by forcing a different exposure time to be used).

[mmoollllee]

For a deeper understanding of what is happening: Does it make a difference if we use a faster RaspberryPi? Or is the speed limited by the camera sensor?

[davidplowman]

No. It's all down to the camera sensor. I think the only factor is the exposure time.

You mentioned that you're using the HQ camera. I just wanted to add that the HQ cam is not a "global shutter camera". It employs a "rolling shutter", like most of our cameras (except the "Global Shutter Camera" that we also sell). The rolling shutter can have an effect on an image, in that the top of the image is not exposed at the same moment as the bottom, so you can get some strange motion artefacts. However, the blurriness of any part of the image is not affected by when the exposure happened, only how long it was.

[mmoollllee]

I'm so sorry, I've been talking about rolling shutter and wrote global shutter in the beginning... I changed the title here for anyone searching for this... Thanks for the informations! I'll give it a try and report back :)

[mmoollllee]

So with the following resulting metadata we're still having too much rolling shutter. Our camera is attached to a big crane, slowly bending with the wind or something. I can't imagein this to be too much movement? Anything else to speed this up? Something like a stream of images with capture_array() instead of switch_mode_and_capture_file()?

{'SensorTimestamp': 11633767665000, 'ScalerCrop': (2, 0, 4052, 3040), 'ColourCorrectionMatrix': (1.9359593391418457, -0.8042237758636475, -0.13173162937164307, -0.2940618395805359, 1.8133236169815063, -0.5192596912384033, -0.07473071664571762, -0.5588080883026123, 1.633538842201233), 'AnalogueGain': 1.0, 'ExposureTime': 561, 'ColourTemperature': 5190, 'ColourGains': (3.059422254562378, 1.4930689334869385), 'SensorBlackLevels': (4096, 4096, 4096, 4096), 'AeLocked': True, 'FocusFoM': 6634, 'Lux': 30111.322265625, 'FrameDuration': 24994, 'DigitalGain': 1.042435646057129}

[davidplowman]

The exposure time in your metadata is extremely short, so I don't think you can improve that. The capture should have the same exposure time, but of course you may be subject to increased rolling shutter effects. I guess I'm a bit confused about what the problem really is - motion blur, rolling shutter effects, or lens problems? Could you maybe clarify that?

You can certainly capture a stream of images, there's no problem with that. Though if single images are showing a problem, then you may end up with a stream of images with the same problem. Capturing a stream of images is usually limited by the time taken to write them out. The HQ cam can run up to 10fps in the full resolution mode, though to achieve that you need fast SSDs - a micro SD card is unlikely to keep up.

[mmoollllee]

Here's an example If you click the last day in the timeline at the bottom and hover over those images you can see, that little movements of the crane cause parts of the image to be blurry and distorted. I'd except capture time to be less than 100ms, but then this seems to be too much of distortion...

[davidplowman]

Thanks for the link. I must confess I'm a bit puzzled by what's going on. Sometimes the details are much clearer, and sometimes they go much blurrier. Also, in the same image, some parts of the image get better, whilst other bits get worse.

Can you just confirm that the posted JPEGs are the same ones that came out of the camera? I'd just like to rule out any effect from post-processing or transcoding.

You're obviously using quite a wide angle lens. Can you tell us anything more about this lens? Is there any adjustability or "play" (looseness) in the lens mechanism? If I had to make a wild guess, I'd be a bit suspicious about vibrations, moving either the whole module or maybe the lens. It's high up so wind vibration may be plausible, and maybe the crane might vibrate too?

If there is some kind of vibration that can affect even a short exposure, then you'd expect to see the effect come and go as you scan down the image, because of the rolling shutter. You'd expect the Global Shutter Cam to have a similar degree of blur, but for the effect to be the same everywhere. As regards what you can do to mitigate these problems, I guess shielding the module to protect it from wind, perhaps cushioning the module housing, might all help at least to some extent.

There is some precedent for this kind of thing because we have seen what I would describe as "similar" effects when people use the v3 camera module in windy conditions. Because the lens is on the end of a spring (it has an autofocus mechanism), it is quite prone to wind vibrations and you can see distortion and blurriness come and go as you scan down the image. But having said that, it's difficult to be certain and I couldn't rule out more or different problems as well.

[mmoollllee]

Okay you nailed it to the lens and you're right. It seems to be just a little little bit loose! And there seems to be some erruption, combined with the rolling shutter to make this effect. I'm sorry I was on the wrong track. Thank you so much for your time and the final hint!