Save and load auto white balance settings

[BobrG]

Required Info
Camera Model	D435
Firmware Version	pyrealsense2.version = "2.44.0.3073"
Operating System & Version	Ubuntu 18.04.3 LTS
Kernel Version (Linux Only)	5.4.0-58-generic
Platform	PC
SDK Version
Language	python

Issue Description

In our project we want to have an ability to save settings set by auto white balance and load them afterwards to accurately set previous white balance state in new conditions. We tried to do this using the following approach with librealsense for python:

# auto white balance 
rgb_camera.set_option(rs.option.enable_auto_white_balance, True)
sleep(8)
rgb_camera.set_option(rs.option.enable_auto_white_balance, False)
wb_value = rgb_camera.get_option(rs.option.white_balance)
# snap frame

# set white balance settings 
rgb_camera.set_option(rs.option.white_balance, wb_value)
# snap frame

Unfortunately, this approach gives incorrect results: auto white balance

manually set white balance

We would be most grateful for your support.

[MartyG-RealSense]

Hi @BobrG As the current value of RGB auto-exposure cannot be read whilst auto-exposure is enabled - as described in https://github.com/IntelRealSense/librealsense/issues/7267#issuecomment-688295655 - my expectation would be that the same is true for trying to read the RGB auto white-balance value (though I do not have a documentation or reference source to confirm this).

In the same way that setting a manual exposure value automatically disables auto-exposure, setting a manual white-balance value will disable auto white-balance automatically.

[BobrG]

Hi @MartyG-RealSense Thank you for your answer! Actually, I wanted to know is it possible to copy the parameters that were set by a device after auto-white-balance property was activated, save this parameters and apply them after the auto-white-balance setting was turned off? In our setup we want to tune camera parameters for specific lighting conditions using auto-white-balance, fix and use this parameters for other lighting conditions for a long period of time but in case our RealSense device requires reconnection from the machine it sets all the parameters to the default so we have to repeat the tunning again. We would like to have an ability to save and load this parameters to address the issue.

[MartyG-RealSense]

There is a camera metadata parameter called RS2_FRAME_METADATA_WHITE_BALANCE, so potentially you could read the white-balance metadata a single time after auto white-balance is activated and store the retrieved value in a custom-named variable, such as last_white_balance for example.

I could not find a specific Python example for retrieving RGB white-balance from metadata, but a script provided in the link below may be modifiable for that purpose.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras#section-2-4-manual-vs-auto-exposure

I believe that it would need the line sensor = profile.get_device().query_sensors()[0] to have the '0' in the end brackets changed to '1' to access the RGB sensor instead of the depth sensor.

The ir definition to access the infrared frames could potentially be changed from this:

ir = frames.get_infrared_frame(1)

to:

color = frames.get_color_frame

References to ir. in the next couple of lines at the bottom of the script would then have to be changed to color.

And of course. rs.frame_metadata_value.actual_exposure would have to be changed to retrieve the white_balance metadata instead. For example, rs.frame_metadata_value.white_balance

---

An alternative approach would be to pick a fixed white-balance value and create a json camera configuration file to store the value in, and then load that json file after the reset to apply the custom value to the white-balance setting.

[BobrG]

@MartyG-RealSense I have tried your first proposed approach, but it frames.supports_frame_metadata(rs.frame_metadata_value.white_balance) returns False and as I understood in my case device supports only metadata parameters described here https://github.com/IntelRealSense/librealsense/blob/master/unit-tests/live/options/test-rgb-options-metadata-consistency.py ; Also, I didn't find how can I load the metadata that was set for specific state.

regarding the second option with json file -- do you mean that I should follow the approach from this file https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-rs400-advanced-mode-example.py ?

[MartyG-RealSense]

There is a Chinese-language article at the link below where another RealSense user was using Python to check the status of metadata parameters such as white balance and also found that it could not be read.

https://www.codetd.com/article/8653498

I have attached an English translation of the article below as a PDF document.

white-balance-metadata.pdf

In regard to loading a json: whilst python-rs400-advanced-mode-example.py can be used to load a json, a simpler approach may be the Python script shared by a RealSense user at https://github.com/IntelRealSense/librealsense/issues/6075#issue-582592683 that allows the loading of a specifically named json filename, such as 'Custom.json'.

Another approach to loading a json with a specific filename is at https://github.com/IntelRealSense/librealsense/issues/6902#issuecomment-793433287

[BobrG]

@MartyG-RealSense I have tried to check json files that can be saved via realsense-viewer, I suppose the situation is save for python wrapper. I restore scanner parameters, turn off the auto white balance mode and save the first json file. Then I set auto white balance on, some tuning happens but after I save new json file I see that it is equal to the previous one, no parameter was changed. So I am afraid that saving and loading json files through Python won't work as well, or I am incorrect?

[MartyG-RealSense]

I tested json saving and loading thoroughly with the RealSense Viewer and also had problems with setting the status of Auto White-Balance. So let's try setting the status with Python code instead. This code accesses the RGB sensor (index number '1') instead of the depth sensor (index number '0') and sets enable_auto_white_balance to false.

---

import pyrealsense2 as rs
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
profile = pipeline.start(config)
color_sensor = profile.get_device().query_sensors()[1]
color_sensor.set_option( rs.option.enable_auto_white_balance, False)

[BobrG]

@MartyG-RealSense yep, this code sets the auto white balance mode off, however this is not what I have troubles with. I need to save the white balance value which was set by auto white balance. However this value doesn't change in the metadata nor in the json no matter is auto mode on or off. It only changes when I set it with set_option but I don't want to do it by hand while there is this auto mode.

[MartyG-RealSense]

I extensively researched the subject further but the available options to create such a mechanism seem to me to be limited. The best idea that I had was to use the json serialize function to write a json file containing the current settings (similar to how the RealSense Viewer can export a json during active streaming) and then load that json file back in to your program to apply the snapshot of the configuration that was taken at the moment of serialization.

The SDK's Python example Advanced Mode program python-rs400-advanced-mode-example.py has a code block about this:

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-rs400-advanced-mode-example.py#L71-L74

[MartyG-RealSense]

Hi @BobrG Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[BobrG]

@MartyG-RealSense after following the approach with saving and loading json file with camera parameters, we didn't manage to restore camera state with white balance which is set by auto-white-balance setting. I wanted also to add here that it is not possible to obtain the same image by manual tuning the white balance parameter, not by set_option method, not through advance_mode and json, not by realsense-viewer. White balance tuning gives either purple, blue, or orange image or smth in between but it is not similar to an image obtained after turning on the auto white balance option.

Regarding this issue, we now have another question -- can we get some clarification, what do such parameters from json as aux-param-colorcorrection* mean for the camera? we found that they might affect the white balance manual tuning but we cannot understand how can we tune them to improve the quality of image.

[MartyG-RealSense]

Color correction can be used to remove the IR dot pattern on the D415 camera model, as described in the section of Intel's camera tuning guide linked to below.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-the-left-color-camera

Color correction is typically not referred to other than for the purposes of IR pattern removal.

In general, Advanced Mode parameters are not documented by Intel because they interact with each other in complex ways and so Intel chose to control them with machine learning algorithms instead. RealSense users are free to perform experimentation with Advanced Mode settings to see how it affects the image though.

[BobrG]

@MartyG-RealSense and can we get description for all the parameters of the advanced mode? We want maybe to understand what else can be tuned to obtain a correct white balance manually.

[MartyG-RealSense]

There are no plans to create descriptions for the Advanced Mode parameters. The vast majority are never used manually by RealSense users.

As a general rule, decreasing the white balance setting below its default value of '4600' shifts the color tinting of the RGB image towards the blue range, whilst increasing it above the default brightens the image and makes the colors more intense (for example, a pale red becoming an intense red like a rosy apple).

Below are examples of low white balance (blue tinted) and high white balance (warm colors).

[BobrG]

Okay, and why is it then auto-white-balance can achieve the correct white balance quality and manual setting gives this blue or pink colors depending on decreasing or increasing?

[BobrG]

@MartyG-RealSense and what other parameters from advanced mode can possibly contribute in the quality of white balance?

[MartyG-RealSense]

If you are experiencing a pink shade when manually increasing white balance above the default (for example, 5260 like the image below) then I wonder whether it is because of low lighting conditions in the location that the camera is in. This image was taken around 2.45 pm.

Here is the same scene at 2.45 pm correctly colored with auto white-balance enabled.

As the white balance setting's value does not visibly update when auto white-balance is enabled, it is not possible to see what values are updating to produce the correctly colored image when auto white-balance is enabled.

As Advanced Mode is part of the stereo depth options, as far as I am aware the RGB image would not be affected by changes to Advanced Mode, and so white balance would not be affected as it is an RGB option.

You can though alter the RGB image if you change RGB settings such as Hue, Saturation and Gamma. It has the most effect if manual white balance is being used, though the image can still be changed by a small amount with these settings when auto white-balance is enabled.

[pkarpyshev]

Hello @MartyG-RealSense! I am working with @BobrG on the issue, and I would like to clarify what problem we're facing. On our rig, we use a white calibration board for white balance setting. The lighting in the room is fully controlled by us, it has no windows or other auxiliary light sources. We have chosen an area on the image on which we were evaluating the Color of the board as seen by the RealSense. After the automatic white balance setting, the R, G and B values were very close to each other (not more than 10 points difference between channels), that meant that the RealSense managed to achieve good white. After that, we iterated over all values of manual white balance while evaluating the very same area on its color, using the distance to the grey line (r=g=b) as a metric to evaluate the quality of setting. However, even the best result during this iteration was much worse than the automatic one (more than 30 points difference in one of the colors). Because of this, we are looking for the ways to improve the results by changing other available options. The problem with automatic white balance is that it cannot be saved between sessions, and it is crucial for our experiment that all the settings remain exactly the same. Is there any information on what other parameters are changed during the automatic white balance setting, that we can also alter to get the same results manually?

[MartyG-RealSense]

Hi @pkarpyshev I researched your question deeply. Whilst I could not find a conclusive answer, my best guess from a careful analysis of the SDK code and official documentation is that when auto white-balance is enabled, it is checking the auto white-balance temperature (measured in Kelvins) to determine the appropriate color temperature value to use.

Auto white-balance temperature is a metadata attribute that can be accessed with RS2_FRAME_METADATA_AUTO_WHITE_BALANCE_TEMPERATURE

Line 67 of a Python pyrealsense2 script at the link below provides an example of accessing this metadata attribute.

https://www.programmersought.com/article/72064349875/

To quote line 67:

auto_white_balance_temperature = frame_metadata_value.auto_white_balance_temperature

[pkarpyshev]

Dear @MartyG-RealSense, We have investigated this command, and it seems that this parameter only shows if the auto white balance is on. When turned on, the metadata value is 1, and when turned off it is 0, so no info on the actual value of white balance. Here are the jupyter notebook screenshots:

With the autobalance on:

With autobalance off:

Regarding the color via autobalance and manual balance:

This image was taken with auto white balance on:

And this one with the manual white balance optimized in a way described above, with the smallest distance to the grey line (r=g=b):

These are the regions we were investigating for optimization:

Using our metric, we have achieved the distance 18.99 using the optimization of manual white balance setting, and after the automatic white balance setting the same metric was equal to 13.37.

Are we doing something wrong, or is there another way to extract the values or save them?

[MartyG-RealSense]

I do not have advice to offer about how to make your own method better than it already is. However, I found a different way to save the white balance settings whilst auto white-balance is enabled and then restore them.

Whilst the RGB stream is active and auto white-balance is enabled, you can save a snapshot of the current general camera settings (not just the white balance) to a custom json camera configuration text file.

If auto white-balance is turned off and then the pipeline is stopped then the json file can be loaded in before the pipeline is started again, updating the camera settings to the ones stored in the json file that was loaded in. When the RGB stream is enabled whilst auto white-balance is disabled, this also updates the manual white balance to the value that was stored in the json snapshot when it was saved whilst auto white-balance was enabled. You can then re-enable auto white-balance.

The above process is testable in the RealSense Viewer.