[BUG] Metadata, ExposureTime and DigitalGain - are they correctly reported/set?

[cpixip]

Testing a RP4/HQ sensor configuration with the latest picamer2-lib, I noticed strange behaviours of metadata, specifically the reported DigitalGain and ExposureTime, with respect to a newly requested exposure value.

Here's my test program:

import time

from picamera2 import Picamera2

# different raw modes to test
rawModes   = [{"size":(1332,  990),"format":"SRGGB10"},
              {"size":(2028, 1080),"format":"SRGGB12"},
              {"size":(2028, 1520),"format":"SRGGB12"},
              {"size":(4056, 3040),"format":"SRGGB12"}]

# default resolution and noise mode
mode       = 3
noiseMode  = 0

# creating the camera
picam2 = Picamera2()

# we start with a blank still_configuration
config = picam2.create_still_configuration()

# increase the amount of buffers
config['buffer_count'] = 4

# make sure the queue is used
config['queue']        = True

# select the appropriate raw mode (make sure .sensor always stays empty!
config['raw']          = rawModes[mode]

# this is what we want for preview/LDR capture
config['main']         = {"size":rawModes[mode]['size'],'format':'RGB888'}

# config the noise processing
config['controls']['NoiseReductionMode']  = noiseMode

# enlarge the possible exposure range
config['controls']['FrameDurationLimits'] =  (100, 32_000_000)

picam2.configure(config)
picam2.set_controls({'AnalogueGain':1.0})

for xTime in range(4000, 100_000,+200):

    picam2.start(show_preview=False)
    picam2.set_controls({'ExposureTime':xTime})

    data = []
    for it in range(0,5):
        time.sleep(0.1)
        metadata     = picam2.capture_metadata()
        data.append(metadata['DigitalGain'])

    ExposureTime = metadata['ExposureTime']

    print(f'R:{xTime:>8} S:{ExposureTime:>8} Delta:{xTime-ExposureTime:>6}     {data}')

    picam2.stop()

It sets up the camera and than loops over some exposure values for testing. Mainly it reports DigitalGain for every capture.

Now, the ExposureTime returned in the metadata is not identical to the value requested. Here's an output listing:

R:    4000 S:   66657 Delta:-62657     [1.0001329183578491, 1.0001329183578491, 1.0001329183578491, 1.0001329183578491, 1.0001329183578491, 1.0001329183578491]
R:    4200 S:    4000 Delta:   200     [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
R:    4400 S:    4200 Delta:   200     [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
R:    4600 S:    4400 Delta:   200     [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

"R:" is the requested ExposureTime, "S:" the reported one, "Delta:" the difference, following by a list of DigitalGains during that capture period.

After a while, the DigitalGain seems to set to a value of 1.0 - for every exposure time requested. That seems supicious to me. Sensors are not able to realize arbitrary exposure times in hardware; the approach taken by libcamera used to be to select a slightly shorter exposure time which the hardware supports and apply an appropriate DigitalGain factor to simulate the requested one.

Obviously, while we can see some remanence of this approach in the first data line, after a while DigitalGain is always reported as 1.0, no matter what exposure time is requested. That seems to be impossible, hardware-wise.

So, what's going on here? For some applications, you want to have control over the raw image data, especially specifying AnalogueGain,DigitalGain and ExposureTime. Only if these values are correct and/or are correctly applied, the sensors the picamera2-lib is supporting can be used as measurement devices.

EDIT: Digging a little bit deeper into this issue. With the following (slightly modified) program

import time

from picamera2 import Picamera2

# different raw modes to test
rawModes   = [{"size":(1332,  990),"format":"SRGGB10"},
              {"size":(2028, 1080),"format":"SRGGB12"},
              {"size":(2028, 1520),"format":"SRGGB12"},
              {"size":(4056, 3040),"format":"SRGGB12"}]

# default resolution and noise mode
mode       = 3
noiseMode  = 0

# creating the camera
picam2 = Picamera2()

# we start with a blank still_configuration
config = picam2.create_still_configuration()

# increase the amount of buffers
config['buffer_count'] = 4

# make sure the queue is *not* used
config['queue']        = True

# select the appropriate raw mode (make sure .sensor always stays empty!
config['raw']          = rawModes[mode]

# this is what we want for preview/LDR capture
config['main']         = {"size":rawModes[mode]['size'],'format':'RGB888'}

# config the noise processing
config['controls']['NoiseReductionMode']  = noiseMode

# enlarge the possible exposure range
config['controls']['FrameDurationLimits'] =  (100, 32_000_000)

picam2.configure(config)
picam2.set_controls({'AnalogueGain':1.0})

for xTime in range(3000, 100_000,+1):

    picam2.start(show_preview=False)
    picam2.set_controls({'ExposureTime':xTime})

    for it in range(0,10):
        time.sleep(0.1)

        request  = picam2.capture_request()
        dummy    = request.make_buffer("raw")
        metadata = request.get_metadata()
        request.release()

        ExposureTime = metadata['ExposureTime']
        AnalogueGain = metadata['AnalogueGain']
        DigitalGain  = metadata['DigitalGain']

    print(f'R:{xTime:>8} S:{ExposureTime:>8} Delta:{xTime-ExposureTime:>6}     A:{AnalogueGain:1.5f} D:{DigitalGain:1.6f}')

    picam2.stop()

I get the following output

....
R:    3026 S:    3000 Delta:    26     A:1.00000 D:1.008667
R:    3027 S:    3000 Delta:    27     A:1.00000 D:1.009000
R:    3028 S:    3000 Delta:    28     A:1.00000 D:1.009333
R:    3029 S:    3028 Delta:     1     A:1.00000 D:1.000142
R:    3030 S:    3028 Delta:     2     A:1.00000 D:1.000472
R:    3031 S:    3028 Delta:     3     A:1.00000 D:1.000802
....

Again, "R:" is the requested ExposureTime, "S:" the reported one, "Delta:" the difference of both. "A:" is the AnalogueGain and "D:" the DigitalGain.

It seems that one has to wait at least 10 frames until libcamera settles for a ExposureTime/DigitalGain-setting. But if 3028 is an exposure value realizable by the hardware (that seems to indicate the above log), why is that exposure value realized by DigitalGain = 1.009333 and ExposureTime = 3000? Of course, the product yields: exposure value = 3027,99. Close, but not exactly the value requested.

The overall behaviour is like this

The user seems to never get the requested ExposureTime.

The ideal situation would be that DigitalGain is setable by user. This was the case in the old picamera-lib.

Obviously, within the libcamera context, this is no longer possible, as an automatic process adjusts the DigitalGain to simulate any exposure value not entertained directly by the hardware. Probably fine for the average user, but why can't we have something similar to the color gains in picamera2? That is, the automatic is active once the camera has started; but if the user sets DigitalGain, this value is used for capturing from now on. Setting DigitalGain to zero switches on the automatic again. I think something along these lines would facilitate a lot of more serious applications of the RP-hardware than the current situation.

In the current situation, there even seems to be no way for the user, even if he knows the hardware values, to set this value. Contrary, as described above, the user gets a nearby value with one of the largest DigitalGain applied. Why is this bad? Because DigitalGain not equal to one leads to additional quantization noise, for starters.

[davidplowman]

Hi, thanks for the report and for the script to reproduce it.

Yes, there is something unfortunate happening here. When the AGC/AEC is running in auto mode, it has a notion of a "stable region", which defaults to a value of 2%. What this means, then, is that if the algorithm wants to make a change that is less than 2% of the effective total exposure, it actually doesn't bother.

Unhappily, this bit of code is getting triggered even when you've set a fixed exposure and gain. Only when you've made a change of 2% from what it was previously, does it update properly.

I'll post a fix for this, though these things always take a little while. In the meantime, you can work around the problem as follows. Find the tuning file that you're using and edit it. Find the first occurrence of "metering_modes", and just above it add the line:

                      "stable_region": 0.0,

This should cause it to update correctly when there is any change at all, though note that it would affect the "auto" mode as well (though I don't think this particularly matters).

[cpixip]

thanks for looking into this. I tried your suggestion, but it seems to have no effect. I even tried to select a very different exposure time before I request the exposure time to actually force the AGC to "move", but again, no effect. Here's this program:

import time

from picamera2 import Picamera2

# different raw modes to test
rawModes   = [{"size":(1332,  990),"format":"SRGGB10"},
              {"size":(2028, 1080),"format":"SRGGB12"},
              {"size":(2028, 1520),"format":"SRGGB12"},
              {"size":(4056, 3040),"format":"SRGGB12"}]

# default resolution and noise mode
mode       = 3
noiseMode  = 0

tuning = Picamera2.load_tuning_file('/home/rdh/cineRaw/imx477_mod.json')

# creating the camera
picam2 = Picamera2(tuning=tuning)

# we start with a blank still_configuration
config = picam2.create_still_configuration()

# increase the amount of buffers
config['buffer_count'] = 4

# make sure the queue is used
config['queue']        = True

# select the appropriate raw mode (make sure .sensor always stays empty!
config['raw']          = rawModes[mode]

# this is what we want for preview/LDR capture
config['main']         = {"size":rawModes[mode]['size'],'format':'RGB888'}

# config the noise processing
config['controls']['NoiseReductionMode']  = noiseMode

# enlarge the possible exposure range
config['controls']['FrameDurationLimits'] =  (100, 32_000_000)

picam2.configure(config)
picam2.set_controls({'AnalogueGain':1.0})

for xTime in range(3026, 100_000,+1):

    picam2.start(show_preview=False)

    # preconditioning
    picam2.set_controls({'ExposureTime':10000})

    for it in range(0,10):
        time.sleep(0.1)    

        request  = picam2.capture_request()
        dummy    = request.make_buffer("raw")
        metadata = request.get_metadata()
        request.release()

    # actually setting my exposure time
    picam2.set_controls({'ExposureTime':xTime})

    for it in range(0,10):
        time.sleep(0.1)

        request  = picam2.capture_request()
        dummy    = request.make_buffer("raw")
        metadata = request.get_metadata()
        request.release()

    ExposureTime = metadata['ExposureTime']
    AnalogueGain = metadata['AnalogueGain']
    DigitalGain  = metadata['DigitalGain']

    print(f'R:{xTime:>8} S:{ExposureTime:>8} Delta:{xTime-ExposureTime:>6}     A:{AnalogueGain:1.5f} D:{DigitalGain:1.6f}')

    picam2.stop()

So it seems that the user never can achieve a setting with DigitalGain = 1.0.

I attach the tuning file used in the test and modified according to the above suggestion here:

imx477_mod.json

[davidplowman]

I had another look here, and yes, there is still some stuff to talk about. When looking at it yesterday I ran into this whole stable region issue, which was genuinely stopping me from seeing what was really going on. Anyway, with the "stable_region" modification, perhaps you could try running this scipt. It's a mangled version of your original one, and takes a few minutes to run.

from picamera2 import Picamera2

picam2 = Picamera2()

raw = {'size': (4056, 3040)}
main = {'size': raw['size'], 'format': 'RGB888'}
controls = {'AnalogueGain': 1.0, 'FrameRate': 10, 'NoiseReductionMode': 0}
config = picam2.create_still_configuration(main=main, raw=raw, controls=controls, buffer_count=4)
picam2.configure(config)

picam2.start(show_preview=False)
last_exp = 0

for xTime in range(4200, 5000, 1):
    picam2.set_controls({'ExposureTime':xTime})

    for it in range(10):
        metadata = picam2.capture_metadata()

    exp = metadata['ExposureTime']
    if exp != last_exp:
        print(xTime, ":", exp, metadata['AnalogueGain'], metadata['DigitalGain'])
    last_exp = exp

It tries out every exposure time from 4200 up to 5000ms, and looks when the exposure delivered by the sensor changes. We expect this to change every time the exposure goes up by the time it takes to scan one more row of pixels. Here's what I got:

4200 : 4200 1.0 1.0
4229 : 4228 1.0 1.0001013278961182
4258 : 4257 1.0 1.0002013444900513
4315 : 4314 1.0 1.000165581703186
4343 : 4342 1.0 1.000032901763916
4372 : 4371 1.0 1.0001307725906372
4400 : 4400 1.0 1.0
4429 : 4428 1.0 1.0000967979431152
4458 : 4457 1.0 1.0001922845840454
4515 : 4514 1.0 1.0001581907272339
4543 : 4542 1.0 1.0000314712524414
4572 : 4571 1.0 1.0001250505447388
4600 : 4600 1.0 1.0
4629 : 4628 1.0 1.000092625617981
4658 : 4657 1.0 1.0001840591430664
4715 : 4714 1.0 1.000151515007019
4743 : 4742 1.0 1.0000301599502563
4772 : 4771 1.0 1.0001198053359985
4800 : 4800 1.0 1.0
4829 : 4828 1.0 1.0000888109207153
4858 : 4857 1.0 1.0001764297485352
4915 : 4914 1.0 1.0001453161239624
4943 : 4942 1.0 1.0000288486480713
4972 : 4971 1.0 1.000114917755127

We can deduce that there are 7 rows of pixels every 200us. So the exposure jumps are every 28 or 29us, though I also notice that some of them (such as 4284) are missing. We read back from the sensor what exposure time it used, so I'm not sure why it seemed not to like those values (another minor mystery).

There isn't really a way in libcamera to find out exactly what exposure values are precisely achievable. I suppose you would set an exposure value, and look at the digital gain that comes back when the exposure has taken effect, and then divide what you asked for by the digital gain. But you'll of course end up with a fractional number of microseconds, which the API doesn't let you request. So there is a microsecond or so of uncertainty in what's going on.

Maybe the API should accept and report floating point exposure times, or maybe use units of nanoseconds, I'm not quite sure what the solution there would be. In this particular case, it's a bit of a coincidence that every 200us does give us an exposure time that works exactly, but obviously this will vary between sensors.

[davidplowman]

Just to clarify here, the aim is to get a fully processed (not raw) output image from the ISP, but where the user has requested a specific exposure time, and an analogue gain of 1.0, and a digital gain of 1.0. Is that correct? I'm just starting to wonder if I've understood things properly...!

[cpixip]

Well, in scientific applications (think astronomy) as well as in other more funny stuff (think telecine), people would rather get a raw image where no analog and digital multipliers are applied.

Analog gain raises the noise floor, and digital gain leads to quantization errors. That's why someone occationally wants to have control of both of these multiplicators. As the raw image is the basis of any derived format, say jpg, I think such a control is also of value for these derived image formats.

Now, as far as I understand the imaging pipeline, both of these multipliers are already applied at the linear raw data level.

I am rather sure that the analog gain is applied in-sensor, before the ADC is sampling the pixel intensities. I guess that the digital gain is also applied in-sensor, but of course only possibly after the ADC. Digital gain could however be applied somewhere else in the imaging pipeline. It would be an important information if, for example, the digital multiplier would not be applied to the raw image, only afterwards at some later stage.

Your program above yields exactly the same values on my RP4 - provided I include my modded tuning file. With the standard tuning file, all DigitalGain-values are = 1.0.

In fact, your program already solved my little personal problem: namely finding exposure times where I can be sure that a digital multiplier is not applied to the footage (that is: DigitalGain = 1.0). The formula 4000 + n*200 seems to do the trick for the HQ sensor. Thanks!

Generally speaking, some things are still irritating for me:

1. without the tuning file trick, the DigitalGain-value returned stays at 1.0 in your test program. Is that this gain value used when requesting an exposure of 4029, or is it rather the 1.000106 needed to approximate the requested exposure value from the one chosen by the sensor (4028)? Can I trust the metadata?
2. Looking deeper into this: if I would request an exposure value of 4028 (which the sensor can handle) the sensor (or libcamera?) chooses instead 4000 plus a DigitalGain = 1.006667. That is slightly suboptimal in terms of image quality.
3. At what point is the DigitalGain applied to the sensor data? I always thought that both analog and digital gain are applied before the raw data is output. I am wrong here?

It would be helpful if you can shed some light into this. Anyway, thanks so much for the clarification so far!

[davidplowman]

The digital gain is not applied in the sensor. You only get it in the fully processed ISP output. No digital gain at all is applied to the raw image. Though ironically, I'm thinking that you can get a better estimate of the real exposure time by taking the the exposure you asked for and dividing it by the digital gain - this might get you round the problem where the reported exposure time is rounded down to microseconds!

Analogue gain is applied to the raw image because the sensor does it. Nearly all sensors allow an analogue gain of exactly 1.0, so far as I know. (The imx708 is weird in that some of its modes have a minimum analogue gain of 1.12, no idea why!)

I've posted a fix to libcamera for the stable region problem, so that will appear "in due course".

[cpixip]

The digital gain is not applied in the sensor. You only get it in the fully processed ISP output.

Uh, that is great information!

Though ironically, I'm thinking that you can get a better estimate of the real exposure time by taking the the exposure you asked for and dividing it by the digital gain...

Ok, I see your point. For the ISP output. The raw file sees the exposure value reported in the metadata, if I understood the situation correctly.

Anyway, that clarified most of the points I was worrying about. Thanks a lot for your help!

[davidplowman]

Yes, that's all correct. The reported exposure time (and analogue gain) is what's been applied to the raw image, though as noted, the exposure time is rounded down to a whole microsecond (which in retrospect I don't particularly love, but that's how it is). The reported digital gain is what the ISP applied during its processing to produce the final output image.

Sorry if that all took such a long time, but hopefully things are clearer now!!