search

raspberrypi / picamera2

New libcamera based python library
BSD 2-Clause "Simplified" License
837 stars 179 forks source link

[BUG] LensPosition control ignored in still configuration with AfMode set to manual. #800

Open Ataman opened 11 months ago

Ataman commented 11 months ago

Describe the bug Setting AfMode = AfModeEnum.Manual and LensPosition = 10.0 (Or anything else between 0 and 10) with Picamera2.set_controls() does not set the lens position on supported cameras.

To Reproduce Script:

from picamera2 import Picamera2
from libcamera import controls

picam2 = Picamera2()
picam2.configure(picam2.create_still_configuration())
picam2.start()

picam2.set_controls({"AfMode" : controls.AfModeEnum.Manual})
picam2.set_controls({"LensPosition" : 10.0})
picam2.capture_file("test.jpg")

picam2.stop()
picam2.close()
picam2 = None

Expected behaviour An image file focused at approx. 10 centimeters when LensPosition is set to 10.0.

Console Output, Screenshots Output looks fine:

[71:09:34.910826613] [7361]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[71:09:35.219248128] [7362]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/media0
[71:09:35.221504125] [7362]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[71:09:35.314907978] [7361]  INFO Camera camera.cpp:1033 configuring streams: (0) 4608x2592-BGR888 (1) 4608x2592-SBGGR10_CSI2P
[71:09:35.318746972] [7362]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected unicam format: 4608x2592-pBAA

Hardware : Raspberry Pi Zero W + Raspberry Camera Module V3

Additional context picamera2 version: 0.3.12-2 libcamera version: 0.0.5+83-bde9b04f Everything works fine using libcamera-apps. The following command produces the expected image focused at 10 centimeters: libcamera-jpeg -n --autofocus-mode manual --lens-position 10.0 -o test.jpg

davidplowman commented 11 months ago

Hi, it always takes a few frames for a control to take effect - what happens if you put a time.sleep(0.5) before the capture?

Ataman commented 11 months ago

time.sleep(0.5) didn't change much. time.sleep(2) however does. This sounds like a rather hacky workaround though. Shouldn't capture_file() block until the previous operation has been finished?

davidplowman commented 11 months ago

The libcamera design is that controls happen asynchronously from capturing frames. Normally I think you would be expected to check the image metadata to discover when the change has happened, though I couldn't totally guarantee how well that works with the lens mechanism as that is totally asyncrhonous.

But I don't really understand why you need to wait for 2 seconds. The delay getting controls through the pipeline should be a few frames at most, plus a few milliseconds for the lens actually to move. You might also have more luck if you set the controls after configure but before start. Though that doesn't explain what you're seeing.

Ataman commented 11 months ago

Considering libcamera-jpeg and libcamera-still work, there has to be some kind of solution.

What I noticed is that libcamera-apps seem to open the camera always twice. My snippet above however only opens it once.

Output of libcamera-jpeg:

[1:48:19.479036355] [1297]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[1:48:19.905627584] [1299]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/media0
[1:48:19.906123581] [1299]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[1:48:19.911383547] [1297]  INFO Camera camera.cpp:1033 configuring streams: (0) 2304x1296-YUV420
[1:48:19.913515533] [1299]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA
[1:48:25.225158049] [1297]  INFO Camera camera.cpp:1033 configuring streams: (0) 4608x2592-YUV420 (1) 4608x2592-SBGGR10_CSI2P
[1:48:25.236350977] [1299]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected unicam format: 4608x2592-pBAA
Ataman commented 10 months ago

The folks over at MediaMTX had a similar issue with auto focus not working in video but found a solution: https://github.com/bluenviron/mediamtx/issues/2326

Maybe these issues are connected somehow? I'm not sure.

davidplowman commented 10 months ago

I'm not sure, it seems to be a very different environment and they seem to have fixed something.

To make progress with this, I'd love to see some tests that show that AF can be turned on/off and the AfMode metadata confirms that this is happening. Can we confirm that when it's in continuous mode, it works as expected? Can we confirm that in manual mode, it never moves spontaneously? Can we confirm that we you set the lens position in manual mode then it does move as instructed (just with some delay) and then stays there? Could you also include the precise script you're using for this test. Thanks.

Ataman commented 10 months ago

They fixed it by doing an initial scan via libcamera. Something picamera2 might be missing? (The libcamera-still application also seems to open the camera twice, maybe to achieve the same thing?)

I wrote a test per your request (requires the exifread module):

from picamera2 import Picamera2
from libcamera import controls
import exifread
import time

test_file = "test.jpg"
log_file = "log.txt"
tests = [
#   af_mode                          lens_pos    sleep_time
    [controls.AfModeEnum.Auto,       None,       0.5],
    [controls.AfModeEnum.Auto,       None,       1.0],
    [controls.AfModeEnum.Auto,       None,       2.0],
    [controls.AfModeEnum.Continuous, None,       0.5],
    [controls.AfModeEnum.Continuous, None,       1.0],
    [controls.AfModeEnum.Continuous, None,       2.0],
    [controls.AfModeEnum.Manual,     10.0,       0.5],
    [controls.AfModeEnum.Manual,     10.0,       1.0],
    [controls.AfModeEnum.Manual,     10.0,       2.0]
]

def take_snapshot(af_mode, lens_pos, sleep_time):
    """ Instantiates a new Picamera2 instance, configures 
    it according to test params and captures to test_file."""

    picam2 = Picamera2()
    picam2.configure(picam2.create_still_configuration())
    picam2.start()

    picam2.set_controls({"AfMode" : af_mode})
    if lens_pos:
        picam2.set_controls({"LensPosition" : lens_pos})
    time.sleep(sleep_time)
    picam2.capture_file(test_file)

    picam2.stop()
    picam2.close()

def log_exif(af_mode, lens_pos, sleep_time):
    """ Reads EXIF data from current test_file and 
    appends it to log_file."""

    with open(test_file, 'rb') as p:
        tags = exifread.process_file(p)
        with open(log_file, 'a') as l:
            l.write(f"### EXIF Tags for snapshot with af_mode={af_mode}, lens_pos={lens_pos}, sleep_time={sleep_time}\n")
            for k,v in tags.items():
                l.write(f"{k}: {v}\n")
            l.write("\n")

for test in tests:
    take_snapshot(test[0], test[1], test[2])
    log_exif(test[0], test[1], test[2])

Logs on my end:

### EXIF Tags for snapshot with af_mode=AfModeEnum.Auto, lens_pos=None, sleep_time=0.5
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:12
Image ExifOffset: 151
EXIF ExposureTime: 13329/500000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:12

### EXIF Tags for snapshot with af_mode=AfModeEnum.Auto, lens_pos=None, sleep_time=1.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:18
Image ExifOffset: 151
EXIF ExposureTime: 24739/1000000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:18

### EXIF Tags for snapshot with af_mode=AfModeEnum.Auto, lens_pos=None, sleep_time=2.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:26
Image ExifOffset: 151
EXIF ExposureTime: 1257/62500
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:26

### EXIF Tags for snapshot with af_mode=AfModeEnum.Continuous, lens_pos=None, sleep_time=0.5
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:33
Image ExifOffset: 151
EXIF ExposureTime: 3507/200000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:33

### EXIF Tags for snapshot with af_mode=AfModeEnum.Continuous, lens_pos=None, sleep_time=1.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:40
Image ExifOffset: 151
EXIF ExposureTime: 4719/250000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:40

### EXIF Tags for snapshot with af_mode=AfModeEnum.Continuous, lens_pos=None, sleep_time=2.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:48
Image ExifOffset: 151
EXIF ExposureTime: 663/31250
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:48

### EXIF Tags for snapshot with af_mode=AfModeEnum.Manual, lens_pos=10.0, sleep_time=0.5
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:24:54
Image ExifOffset: 151
EXIF ExposureTime: 21111/1000000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:24:54

### EXIF Tags for snapshot with af_mode=AfModeEnum.Manual, lens_pos=10.0, sleep_time=1.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:25:01
Image ExifOffset: 151
EXIF ExposureTime: 367/20000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:25:01

### EXIF Tags for snapshot with af_mode=AfModeEnum.Manual, lens_pos=10.0, sleep_time=2.0
Image Make: Raspberry Pi
Image Model: /base/soc/i2c0mux/i2c@1/imx708@1a
Image Software: Picamera2
Image DateTime: 2023:11:07 10:25:09
Image ExifOffset: 151
EXIF ExposureTime: 9793/500000
EXIF ISOSpeedRatings: 112
EXIF DateTimeOriginal: 2023:11:07 10:25:09

Currently, it doesn't seem to store any information about focus. However, when reading the EXIF data on a snapshot taken with libcamera-still there is this information:

Subject Distance: 0.7178750897 m

I assume that is the focus distance for the given lens position at that time.

davidplowman commented 10 months ago

You're right that Picamera2 doesn't store the lens position in the exif data. I didn't know libcamera-apps even did that. Is the problem here that there's no exif data for the lens? Or is it that the lens is or is not moving correctly? I'm afraid I'm starting to lose the thread of what we think is wrong here. Maybe you could post another short Python script (as you did initially) that does something wrong and we can look again at that? Thanks.

Ataman commented 10 months ago

Is the problem here that there's no exif data for the lens?

No, I don't need the EXIF data.

Or is it that the lens is or is not moving correctly?

That is indeed the problem. The lens is not moving reliably in some cases.

I updated the script to take seperate pictures for each test and reordered the tests by sleep time:

from picamera2 import Picamera2
from libcamera import controls
import exifread
import time

test_file = "test.jpg"
log_file = "log.txt"
tests = [
#   af_mode                          lens_pos    sleep_time
    [controls.AfModeEnum.Auto,       None,       0.5], #0
    [controls.AfModeEnum.Continuous, None,       0.5], #1
    [controls.AfModeEnum.Manual,     10.0,       0.5], #2
    [controls.AfModeEnum.Auto,       None,       1.0], #3
    [controls.AfModeEnum.Continuous, None,       1.0], #4
    [controls.AfModeEnum.Manual,     10.0,       1.0], #5
    [controls.AfModeEnum.Auto,       None,       2.0], #6
    [controls.AfModeEnum.Continuous, None,       2.0], #7
    [controls.AfModeEnum.Manual,     10.0,       2.0], #8
    [controls.AfModeEnum.Auto,       None,       5.0], #9
    [controls.AfModeEnum.Continuous, None,       5.0], #10
    [controls.AfModeEnum.Manual,     10.0,       5.0]  #11

]

def take_snapshot(af_mode, lens_pos, sleep_time, index):
    """ Instantiates a new Picamera2 instance, configures 
    it according to test params and captures to test_file."""

    picam2 = Picamera2()
    picam2.configure(picam2.create_still_configuration())
    picam2.start()

    picam2.set_controls({"AfMode" : af_mode})
    if lens_pos:
        picam2.set_controls({"LensPosition" : lens_pos})
    time.sleep(sleep_time)
    picam2.capture_file(f"{index}_{test_file}")

    picam2.stop()
    picam2.close()

def log_exif(af_mode, lens_pos, sleep_time, index):
    """ Reads EXIF data from current test_file and 
    appends it to log_file."""

    with open(f"{index}_{test_file}", 'rb') as p:
        tags = exifread.process_file(p)
        with open(log_file, 'a') as l:
            l.write(f"### EXIF Tags for snapshot with af_mode={af_mode}, lens_pos={lens_pos}, sleep_time={sleep_time}\n")
            for k,v in tags.items():
                l.write(f"{k}: {v}\n")
            l.write("\n")

idx = 0
for test in tests:
    take_snapshot(test[0], test[1], test[2], idx)
    log_exif(test[0], test[1], test[2], idx)
    idx = idx + 1

Of all tests, only 7, 8, 10 & 11 achieved the correct focus. All others resulted in blurry images (including test 9 with AfMode:Auto and 5 seconds sleep before capture).

davidplowman commented 10 months ago

Thanks for the code. When using Auto mode, I don't actually see it triggering an autofocus cycle. Maybe have a look at section 5.2.4 of the manual to see if that helps.

Putting those aside for the moment, that means we have this:

import time
from picamera2 import Picamera2
from libcamera import controls

picam2 = Picamera2()
config = picam2.create_still_configuration()
picam2.start(config)
picam2.set_controls({'AfMode': controls.AfModeEnum.Manual, 'LensPosition': 10.0})
time.sleep(1.0)
picam2.capture_file("this_is_blurry.jpg")
time.sleep(1.0)
picam2.capture_file("this_is_sharp.jpg")

Does that sound like an appropriate test case to discuss?

One thing I notice is that you're using a full resolution configuration which will give you a very slow framerate by default. That means that requests (containing AF commands, for example) get queued up and take longer to be handled. I'm still a bit surprised, though, TBH. Perhaps try adding buffer_count=3 to your configuration, that should at least get you up to the camera's max framerate (still not so high in the full res mode).

Another alternative would be to run in a faster preview mode, and then do a mode switch just for the capture. That's how all the libcamera-apps work.

Ataman commented 10 months ago

Yes that's a good test case for this issue and works as expected. I will look into trying to use the preview mode without an actual connected display.

I did indeed miss triggering the autofocus cycle with AfMode:Auto.

WillisAllen commented 2 months ago

This issue is still present. I've been trying to make a script to find the appropriate lens position to focus on targets that are a set distance away. All the pictures are the same. I've tried sleeping for up to 10 seconds after setting the controls but nothing. I've also tried autofocus_cycle() which is the only thing that seems to actually change anything

davidplowman commented 2 months ago

@WillisAllen Am I right in thinking this is now a new issue being raised? Is so, could you maybe file a new report, please? Don't forget to include:

Thanks!

davidplowman commented 3 weeks ago

Hi, and thanks for the report. Could you maybe file it as a new issue, please - no point spamming others in this thread who aren't having this problem. Thanks!