[OTHER] Exposure Bracket Function

[Bra1nsen]

hey guys. thank you for your dedication towards developing picamera2 further

[Exposure Bracketing]

from picamera2 import *
from time import sleep
from fractions import Fraction

# This script captures exposures with varying shutter time. 
# The frame rate needs to be longer than the exposure or it won't work. 
# The capture takes as long as the frame rate, so reducing the frame rate saves time for quick exposures.

with picamera2.Picamera2() as camera:

        camera.resolution = (4056,3040)
        #camera.framerate = Fraction(1, 2)
        camera.iso = 100
        camera.exposure_mode = 'off'
        camera.awb_mode = 'off'
        camera.awb_gains = (1.8,1.8)

        config = camera.create_still_configuration()
        camera.configure(config)

        camera.set_controls({"ExposureTime": 114, "AnalogueGain": 1})
        camera.start()
        camera.capture_file('0.jpg')
        print(camera.capture_metadata())

        camera.stop()

        camera.set_controls({"ExposureTime": 300, "AnalogueGain": 0.5})
        camera.start()
        camera.capture_file('1.jpg')
        print(camera.capture_metadata())

        camera.stop()

        camera.set_controls({"ExposureTime": 500, "AnalogueGain": 0.5})
        camera.start()
        camera.capture_file('2.jpg')
        print(camera.capture_metadata())

        camera.stop()

        camera.close()

Any suggestions for improving my python code in terms of smartness. All setting should remain constant expect for exposure time/shutter speed. For example FocusFoM, ColourGains, Colourcorrectionsmatrix, Lux seems to change in my series.

The idea is to generate a HDR Image via multi exposure fusion algorthmn. Use case for solar power prediction.

[davidplowman]

Hi, just a few things to comment on:

1. Setting properties with camera.iso = 100 is an idiom from PiCamera, not Picamera2. You need to set these values using (for example) camera.set_controls(...). In fact there is no ISO control in libcamera, you just set the (analogue) gain directly.
2. Same goes for those awb_gains, that's a PiCamera thing, in Picamera2 it's camera.set_controls({"ColourGains": (1.8, 1.8)}).
3. In a couple of places you set the analogue gain to 0.5. Actually values under 1.0 aren't valid, I expect it reports "AnalogueGain": 1.0 instead?
4. The ColourGains and ColourCorrectionMatrix should stop changing, I'm guessing that you're not setting the colour gains correctly (point 2 above.)
5. The FocusFoM and Lux values are merely outputs of the system measured from the image (the former is a local contrast measure but it depends on the pixel levels, the latter is an estimate of how bright a scene is). They shouldn't affect processing if you've fixed the AEC/AGC and AWB.

[Bra1nsen]

1. Picamera2 Library doc is great, but some questions still remain.

   config = camera.create_still_configuration(main={"size": (2048, 1536)})
   camera.configure(config)

   Where do I find all possible settings one can use in camera.create_still_configration() ? I read something about different image formats (XBGR888, RGB888, YUV420), where RGB888 works with 24 bits per pixel? I thought that the maximal possible would be 12bit raw format..

2. What else could one set/config here? Iam looking for a complete list of attributes :)

camera.set_controls({"ExposureTime": 200, "AnalogueGain": 1, "ColourGains": (1.8,1.8)})

3. yes auto sets AnalogueGain to 1.0

4. How can one print/export Lux/FocusFoM Values for example?

[davidplowman]

Hi again, let me try to answer a few of these:

1. Picamera2 Library doc is great, but some questions still remain.

config = camera.create_still_configuration(main={"size": (2048, 1536)})
camera.configure(config)

Where do I find all possible settings one can use in camera.create_still_configration() ? I read something about different image formats (XBGR888, RGB888, YUV420), where RGB888 works with 24 bits per pixel? I thought that the maximal possible would be 12bit raw format..

Settings divide into configuration settings (chapter 4 of the manual) and controls. We allow controls to be "associated" with a configuration for convenience, but controls are in general things that can be changed at runtime.

You're right that 12-bit raw is the "most" you can get from the sensor, but the ISP can process this into numerous different formats - RGB, ARGB, YUV etc. The diagram in section 4.2 attempts to illustrate this.

2. What else could one set/config here? Iam looking for a complete list of attributes :)

The exhaustive list of controls is supposed to be in Appendix C of the manual, only I seem not to have written that yet. Oops - I'll get on with that. But check out section 5.1 - using picam2.camera_controls gives you a good list of all the controls that are available, hopefully that will help.

camera.set_controls({"ExposureTime": 200, "AnalogueGain": 1, "ColourGains": (1.8,1.8)})

3. yes auto sets AnalogueGain to 1.0
4. How can one print/export Lux/FocusFoM Values for example?

You can find them in the image metadata (metadata["Lux"] and metadata["FocusFoM"]). Note that capture_file actually returns the metadata directly, so you don't need a separate capture_metadata after.

[Bra1nsen]

Thanks david, rlly grateful for your advices. are you drinking coffee - can I spend you one via paypal :)?

4. I would like to save Lux as variable in my python script.

   x = camera.capture_metadata[metadata["Lux"])
   print(x)

   also tried caputure_file(metadata["Lux"])

TypeError: 'module' object is not subscriptable

[davidplowman]

Sorry, I was a bit too brief in my previous reply! You could do something like:

metadata = camera.capture_metadata()
print(metadata["Lux"])

Or this would be even better as you'd get the metadata for the frame you just captured, not the frame after (which possibly also incurs a short delay):

metadata = camera.capture_file('0.jpg')
print(metadata["Lux"])

[Bra1nsen]

thanks worked out =)!

Lux values are merely outputs of the system measured from the image (the former is a local contrast measure but it depends on the pixel levels, the latter is an estimate of how bright a scene is

I couldnt find the formula: how Lux is calculated/measured (IMX477 Doc.). Iam curious about the function, can you recommend some pdf where I find these informations? I mean Luminance is a measure of the amount of light falling on a surface., so there has to be a formula calculating Lux with camera specs (shutter opening time, sensitivity of the CMOS, analog-to-digital units) and image data or?

[davidplowman]

The lux measurement is a kind of intermediate value that our camera control algorithms want, but it's useful enough that we calculate it once and "publish" it in the image metadata. It gets calculated here. It's actually most important to the AWB algorithm which uses it to choose a "prior" distribution for the candidate illuminants (e.g. "daylight" is more likely when the lux value is high).

It's a simple ratio calculation. When we calibrate the camera we measure the lux level of a scene (according to a light meter) and the average pixel level for a given exposure and gain, and then deduce an estimated lux level while the camera is running using the ratios of the exposure, gain and pixel levels compared to what we measured at calibration time.

It has some limitations. Most obviously, on the HQ cam, we don't know how the aperture is set which could throw the number off completely. If I recall correctly, we calibrated it for the aperture being "mostly open" so it shouldn't be too bad if that's the configuration you're using.

[davidplowman]

I was thinking about closing this issue - I think the original questions here are answered and of course we've been having a long follow-up discussion elsewhere!

[Bra1nsen]

Yes absolutely =)!

[Bra1nsen]

hey david - Iam so extremly grateful for all of your technical assistance. I'm working on high pressure to get everything ready. There is just one more problem. I would like to push the camera metadata to influxdb.

ExposureTime_1 = 50
ExposureTime_2 = 100
ExposureTime_3 = 150
ExposureTime_4 = 250
ExposureTime_5 = 400
ExposureTime_6 = 650
ExposureTime_7 = 1050
ExposureTime_8 = 1700
ExposureTime_9 = 2750

def exposure_bracket(ExposureTime):

    with picamera2.Picamera2(tuning=tuning) as camera: 
        config = camera.create_still_configuration(main={"size": (1014, 760),"format": "RGB888"}, raw={"format": "SRGGB12", "size" : (2032, 1520)})
        camera.configure(config)
        camera.set_controls({"ExposureTime": ExposureTime, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime}.tga")
        metadata = camera.capture_metadata()
        lux = metadata["lux"]
        sens_temp = metadata["SensorTemperature"]
        print(lux)
        print(sens_temp)
        camera.stop()

        camera.close()

#CAPTURE
p1c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_1])
p2c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_2])
p3c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_3])
p4c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_4])
p5c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_5])
p6c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_6])
p7c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_7])
p8c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_8])
p9c_ = multiprocessing.Process(target=exposure_bracket, args=[ExposureTime_9])  
#capture_and_transform
p1c_.start()
p1c_.join()
p1s_.start()
p2c_.start()
p2c_.join()
p2s_.start()
...

I guess due to the dynamic image-naming the Key-Erros (e.g. KeyError: 'SensorTemperature' ) occurs... Or is it maybe something else?

Do you have a clever idea?

[davidplowman]

Hi, I don't have access to a Pi right now, but there are a few things I would change here.

I don't really see any benefit in using the multi-processing module for the captures given that everything actually happens one after the other. (I assume the references to p1s etc. are just typos?) Am I right in thinking that this forks the current process, then the camera does some more stuff, which the original process (which forks again...?) does not know about, or maybe it does and you have two processes fighting over the camera? It may just be that I don't understand it very well, but it makes me a bit nervous!!

I also wouldn't configure the camera every time. It might be harmless, but it does de- and re-allocate tons of memory so it's always safer to avoid it if you can. Then I'd probably repeat only the code from camera.set_controls to camera.stop for each exposure.

Also, I think camera.capture_file actually returns the metadata you want so you can do metadata = camera.capture_file(...). Otherwise, calling capture_metadata again actually gives you the metadata for the next frame. It might also mean you have to wait for the next frame too, so it just slows everything up.

I don't really see why the SensorTemperature key would be missing, therefore no particular reason to suppose that any of the above changes would fix it. Maybe print out metadata if it's failing, and we can see what's in it. I can also try this out once I'm back at Pi Towers on Monday.

[Bra1nsen]

does de- and re-allocate tons of memory

Ah ok good to know. Since it is important to make a prognosis from the images obtained, every second counts in the processing.

The idea was that as soon as the first image is captured, the image transformation for the respective image can begin, while the next image can be captured in parallel. The alternative would be to first capture all images and then transform all images.

[Bra1nsen]

#!/usr/bin/python3

import datetime
import os
import multiprocessing
import time
import shutil

from solarmeter import *
from datetime import timezone, timedelta, datetime
from multiprocessing import Process
from time import sleep
from picamera2 import *

tuning = Picamera2.load_tuning_file("imx477.json")

#-----------------------------------------------------------------------
#FIBONACCI

t1= time.perf_counter()
ExposureTime_1 = 50
ExposureTime_2 = 100
ExposureTime_3 = 150
ExposureTime_4 = 250
ExposureTime_5 = 400
ExposureTime_6 = 650
ExposureTime_7 = 1050
ExposureTime_8 = 1700
ExposureTime_9 = 2750

#-----------------------------------------------------------------------

dt = datetime.now(timezone.utc)
utc_time = dt.replace(tzinfo=timezone.utc)
utc_timestamp = int(utc_time.timestamp())
utc_timestamps = str(utc_timestamp)

SS = datetime.now(timezone.utc).second
MM = datetime.now(timezone.utc).minute
HH = datetime.now(timezone.utc).hour
DD = datetime.now(timezone.utc).day
MM = datetime.now(timezone.utc).month
YYYY = datetime.now(timezone.utc).year

YYYYs = str(YYYY)
MMs = str(MM)
DDs = str(DD)

os.chdir("/home/pi/NN/proc")

def exposure_bracket():

    with picamera2.Picamera2(tuning=tuning) as camera: 
        config = camera.create_still_configuration(main={"size": (1014, 760),"format": "RGB888"}, raw={"format": "SRGGB12", "size" : (2032, 1520)})
        camera.configure(config)
        camera.set_controls({"ExposureTime": ExposureTime_1, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_1}.tga")
        metadata = camera.capture_metadata()
        lux1 = metadata["lux"]
        sens_temp = metadata["SensorTemperature"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_2, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_2}.tga")
        metadata = camera.capture_metadata()
        #lux2 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_3, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_3}.tga")
        metadata = camera.capture_metadata()
        #lux3 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_4, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_4}.tga")
        metadata = camera.capture_metadata()
        #lux4 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_5, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_5}.tga")
        metadata = camera.capture_metadata()
        #lux5 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_6, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_6}.tga")
        metadata = camera.capture_metadata()
        #lux6 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_7, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_7}.tga")
        metadata = camera.capture_metadata()
        #lux7 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_8, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_8}.tga")
        metadata = camera.capture_metadata()
        #lux8 = metadata["lux"]
        camera.stop()

        camera.set_controls({"ExposureTime": ExposureTime_9, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
        camera.start()
        camera.capture_file(f"{ExposureTime_9}.tga")
        metadata = camera.capture_metadata()
        #lux9 = metadata["lux"]
        camera.stop()

        camera.close()          

#imagetransformation
def fish2skyplane(ExposureTime):
    os.system(f"/home/pi/fish2skyplane/fish2skyplane/package/./fish2skyplane -s 185 -r 393 -c 574 335 -m 64 -w 640 -p -0.0116 -0.0012 -0.00194 0.7044 {ExposureTime}.tga")

#save images as utc_timeseries --> (utc_1,utc_2,utc_3,..)
def upload_skydrive(ExposureTime):
    shutil.move(f"/home/pi/NN/proc/{ExposureTime}_sky.tga", f"/home/pi/mnt/skydrive/DATASETS/tuning/{MMs}/{DDs}/{utc_timestamps}_{ExposureTime}.tga")

#protect sd card from overloading (64gb)    
def remove_img_SD(ExposureTime):
    os.remove(f"/home/pi/NN/proc/{ExposureTime}.tga")

#CAPTURE EXPOSURE TIMESERIES
p1c_ = multiprocessing.Process(target=exposure_bracket)
#SOLARMETER TARGET W/m²
s1 = multiprocessing.Process(target=digitalize)
#FISH2SKY
p1s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_1])
p2s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_2])
p3s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_3])
p4s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_4])
p5s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_5])
p6s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_6])
p7s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_7])
p8s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_8])
p9s_ = multiprocessing.Process(target=fish2skyplane, args=[ExposureTime_9])
#UPLOAD_GDRIVE
p1 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_1])
p2 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_2])
p3 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_3])
p4 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_4])
p5 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_5])
p6 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_6])
p7 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_7])
p8 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_8])
p9 = multiprocessing.Process(target=upload_skydrive, args=[ExposureTime_9])
#DELETE FROM SD
pa = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_1])
pb = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_2])
pc = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_3])
pd = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_4])
pe = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_5])
pf = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_6])
pg = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_7])
ph = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_8])
pi = multiprocessing.Process(target=remove_img_SD, args=[ExposureTime_9])

#EXECUTION 
p1c_.start()
p1c_.join()
s1.start()
p1s_.start()
p2s_.start()
p3s_.start()
p4s_.start()
p5s_.start()
p6s_.start()
p7s_.start()
p8s_.start()
p9s_.start()
p9s_.join()
p1.start()
p2.start()
p3.start()
p4.start()
p5.start()
p6.start()
p7.start()
p8.start()
p9.start()
p9.join()
pa.start()
pb.start()
pc.start()
pd.start()
pe.start()
pf.start()
pg.start()
ph.start()
pi.start()

t2= time.perf_counter()
print(f"Finished in {t2-t1} seconds")

it does finish in ~ 11s which is still okay (just lost 1,5s)

but still getting the key errors, would be great if you could take a look, as soon as you have time. In any case, I wish you a relaxing Sunday.

[davidplowman]

Can you print out metadata when it fails? Then we can have a look to see exactly what's there. Also don't forget that you should be able to do metadata = camera.capture_file(...) instead of the separate metadata = camera.capture_metadata(). Also, because you're inside a with, you shouldn't need camera.close() (though it is harmless). I'll give it all a try tomorrow.

[Bra1nsen]

yea printing metadata works fine! But as as soon as I comment out lux = metadata["Lux"] the whole script is going crazy ;D

[davidplowman]

I'm not quite sure what's going wrong for you. I took your script, deleted all the stuff after the captures (which I can't run anyway), and rolled them up into a loop, just so that I was editing less code. Here's what I ended up with:

import time
from picamera2 import *

tuning = Picamera2.load_tuning_file("imx477.json")

t1 = time.perf_counter()

exposures = [50, 100, 150, 250, 400, 650, 1050, 1700, 2750]

def exposure_bracket():
    with Picamera2(tuning=tuning) as camera: 
        config = camera.create_still_configuration(main={"size": (1014, 760),"format": "RGB888"}, raw={"format": "SRGGB12", "size" : (2032, 1520)})
        camera.configure(config)

        for exp in exposures:
            camera.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
            camera.start()
            metadata = camera.capture_file(f"{exp}.tga")
            lux = metadata["Lux"]
            sens_temp = metadata["SensorTemperature"]
            camera.stop()
            print("Exposure:", exp, "lux:", lux, " sensor temp:", sens_temp)

exposure_bracket()

t2 = time.perf_counter()
print(f"Finished in {t2-t1} seconds")

For me this code works perfectly. It completes in just over 7s with no errors. I also tried running the exposure_bracket function using the multiprocessing module, and that too seems to work fine.

Are you able to try this?

I'll also lock this thread so that the discussion can continue on the issue that's still open, if that's OK. Thanks!