[HOW-TO] Optimizing PiCam v3 Autofocus for a High-Altitude Balloon launch

[darksidelemm]

Describe what it is that you want to accomplish Figure out how to make the autofocus behave better on a High-Altitude Balloon flight. We've been launching PiCameras of various types on high altitude balloon payloads for many years. The fixed/adjustable focus PiCam2 and PiCamHQ cameras work pretty well, though the PiCam2 image quality leaves a little to be desired, and the PiCamHQ can be quite heavy if you put a good lens on it. The PiCam v3 looks to be a nice option... however there are issues.

Normally we want to set the focus to near-infinity. I usually adjust the focus using a distant house down my street. On the PiCam v3 we can set the lens position to 0.0, which on-ground looks fine. However, on our first flight with a PiCam v3 in this configuration, we found that the actual lens position appears to drift with temperature, resulting in fuzzy images as the camera got cold, then better again as it got warmer at the top of the flight (>30km altitude). See here and here for examples of the fuzzy images.

So, onto trying out Autofocus.

The typical image payload configuration has the camera pointing roughly at the horizon. I say roughly, as the payloads swing and spin under the balloon, so the field of view is always moving around. A PiCamv3 had been successfully used on a balloon launch in the UK with good results, but in that case there was a 'target' (Babbage the teddybear!) in the centre of the frame, so the autofocus probably had an easier job.

We bit the bullet and Andrew KE5GDB tried out a PiCam v3 in continuous autofocus mode without a focus target on a launch a few weeks ago. Our observations:

• Autofocus seemed to work fairly well just after launch.
• On ascent, the images started to get blurrier, with the best results seeming to be when the camera was angled slightly downwards, which makes me think the default autofocus zone (roughly in the centre of the frame I think?) is not ideal.
• At the 'top' of the fight, good images were produced. There may still be temperature effects involved here.
• Some example images from this flight are available here.

So, some questions:

• I could set an AfWindow so that focusing is performed using the lower-centre part of the frame. Is the 'reference' for the AfWindow rectangles the top-left of the frame? e.g. if I wanted roughly the lower middle of a full-res PiCamV3 frame, would (1152, 1296, 2304, 1036) be appropriate?
• Are there any other autofocus optimisations that might help with the payload movement issue? Range constraints? Speed?
• I'd like to query and send down the lens position via our telemetry link (~1Hz updates) to investigate the behaviour of the autofocus during a flight - to see if it's hunting, or if the average focus position is drifting with temperature - is querying this via get_camera_metadata() while images are not being saved an appropriate way of doing this?
• If there are still temperature issues, is there any way to 'pre-adjust' the lens to try and correct for these effects? (probably with some reduction in how close the lens will be able to focus)

I get that this use-case is pretty out-of-spec for the camera, but it would be great if we could figure out a way to make use of them in a flight.

Describe alternatives you've considered

• Fixed lens position at infinity: Doesn't work, as the focus mechanism appears to drift with temperature. We tried this on a previous flight, resulting in lots of very blurry images as the payload ascended through the tropopause (the coldest part of the ascent).
• Trying to heat the camera to avoid temperature drift: Probably not impossible, but difficult and increases the electrical complexity of the payload.

Additional context While the entire code is a bit long, I'm essentially doing this:

# Set focus, white balance, exposure settings
self.cam.set_controls(
            {'AwbMode': self.whitebalance,
            'AeMeteringMode': controls.AeMeteringModeEnum.Matrix,
            'NoiseReductionMode': controls.draft.NoiseReductionModeEnum.Off}
            )
self.cam.set_controls({"AfMode": controls.AfModeEnum.Continuous})
# Set AfWindows
self.cam.set_controls({"AfWindows": [self.af_window_rectangle]})

# Start the camera (self.cam is the picam2 object)
self.cam.start()

# Go into a long running loop, containing:

    # Capture 5 images in a row
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    self.cam.capture_file(...)
    # Some analysis to pick the sharpest image here (biggest JPEG file size), then some resizing, takes ~30-60 seconds
    # Image is then put into a queue to be transmitted via our radio downlink.

   # Additional 10 second delay here, though the other processing delays mean it can be up to a minute

[davidplowman]

Hi, l'll try and answer a few of your questions, and then perhaps pass on some of the others!

1. Yes, using self.cam.capture_metadata() would be the way to capture information about the camera system. There should be quite a lot in there, including the LensPosition, and also the AfState which will tell you whether the lens is currently scanning, or whether it's in a "focused" or "failed to focus" state.

2. The AfWindow you gave - (1152, 1296, 2304, 1036) - looks good to me, taking the middle half horizontally, and just under half of the image vertically, starting half way down.

3. These lens mechanisms are known to behave differently according to both temperature and orientation (and probably other things too). It is possible to mitigate some of these effects in the camera tuning file - perhaps I can ask @njhollinghurst to comment on what changes might be helpful? Thanks!

[njhollinghurst]

I can't think of anything very helpful. The tuning file can be edited for a static offset - which might help in cases where a negative lens position would otherwise be required (since negative values are not allowed). It might also make AF more stable by reducing the loop gain. But this kind of imagery is challenging for PDAF as it doesn't contain much in the way of horizontal deltas (vertical gratings).

[darksidelemm]

On the question of offsets - is the 0.0 'infinity focus' position at the end of the lens travel range? Or can it go past this?

What I'm still pretty unsure about is if the thermal issue we encountered in the fixed-lens-position flight was the infinity focus position ending up being at some 'negative travel' location. Doing a flight with autofocus enabled, and logging/transmitting the lens position throughout the flight might shed some light on how this focus point 'moves' with temperature.

[njhollinghurst]

Clipping at 0.0 is a software limit only. In fact, the standard tuning has a tiny bias so 0.0 is slightly beyond infinity in most cameras, to allow for variation.

If necessary, this margin can be increased by lowering the mapping from dioptres to hardware lens driver settings (values between 0 and 1023, where 512 means "zero drive current" and 1023 pushes the lens out for nearest focus; the usable range when horizontal and at room temperature is about 300-960 before the lens hits its end-stops). This can be done either in Python, or by copying and editing the camera tuning file.

I also don't know the mechanism of the thermal effect: whether it causes an absolute shift in lens position or merely scales the current (the difference from 512) or something else...

Is this the standard or the wide-angle lens?

[darksidelemm]

Standard lens.

[njhollinghurst]

Well I don't know if it will help, but the following code snippet would offset everything by 3 dioptres, so that to focus on infinity you would now need to set lens position to +3.0

from picamera2 import Picamera2, Preview
from libcamera import controls

OFFSET_DIOPTRES = -3.0

tuning = Picamera2.load_tuning_file("imx708.json")
map = Picamera2.find_tuning_algo(tuning, "rpi.af")["map"]
print(map)
offset_hw = OFFSET_DIOPTRES * (map[3]-map[1])/(map[2]-map[0])
for i in range(1, len(map), 2):
    map[i] += offset_hw
print(Picamera2.find_tuning_algo(tuning, "rpi.af")["map"])

picam = Picamera2(0, tuning=tuning)

In short it's lowering the dioptres->driver mapping by 96 by reducing map[1] and map[3] by that amount. The hack enables more lens travel in the "beyond infinity" direction. (Assuming that's the correct direction of offset, which isn't clear.)

If you can discover the actual range of useful settings, set map[1] to start at the smallest such value. You could then set find_tuning_algo(tuning, "rpi.af")["ranges"]["normal"]["max"] to the spread of useful settings (in Dioptres), which will restrict the search range that Autofocus will sweep through; it doesn't restrict manual lens positions.

[darksidelemm]

I can certainly give this a go! Related question - If I use horizontal and vertical flipping, is the AfWindow applied before or after the flip is performed?

[njhollinghurst]

That is a good question. It ought to apply to the flipped image. I will have to do some more tests to check it happens consistently.

[darksidelemm]

Thanks! I did some tests myself, but without much conclusive results as I don't think my 'test scene' was working all that well.

[njhollinghurst]

I have just done some tests and I think it's correct. Both Phase and Contrast statistics align with the image as received (after any flips, which are implemented inside the sensor).

[It won't necessarily work the same on every sensor, e.g. those without hardware "flip" capability, but we don't currently support PDAF on any others.]

[darksidelemm]

OK, this is really good to know!

I'm hoping to do a launch sometime in November using a PiCam v3 and autofocus. My thinking right now is I will use:

• AfWindows set as above (approx lower middle of frame)
• Offset the lens by maybe 1 dioptre (still need to test this).

We will also now be sending the lens position in our downlinked telemetry, so we'll have this data at 1 Hz time resolution - should be interesting to see if it drifts during flight!

I did get some 'fixed focus' IMX708 lenses from ArduCam to try.. unfortunately these really are fixed focus, the lens is glued in place, with the focus set too close to be useful...