Output PNGs are green

[y2k0999]

I'm using raw images captured on a Hikrobot machine vision camera in bayer rg8 pixel format, then converting them to dng and adding exif data myself. The dng images look totally normal, but the output I get from running them this model is all green. Is there a parameter I can change somewhere or something else I can do to get the output images back to normal?

Your project is amazing by the way!

[teboli]

Dear y2k0999,

I do not have your images but it sounds like you have not applied white balance to the images prior restoration. You should multiply the red and blue channels of your images, preferably prior to restoration in the Bayer images, with the white balance coefficients that come with the EXIF metadata.

Our script should automatically read the white balance values and apply them here: https://github.com/Jamy-L/Handheld-Multi-Frame-Super-Resolution/blob/main/handheld_super_resolution/utils_dng.py#L147 . Perhaps none of the if statements are valid for your images for any reason?

Best,

Thomas

[y2k0999]

Wow thanks for the quick response! I checked and my input hits both of the for loops in there. I was looking at this code for a while today and was thinking it could be a bug related to CFA or something that could be related to my exif data. I'll also share a sample of the output. Please note that I cropped this to keep the size small and had to convert to jpg because github doesn't support dng, but the colors are unchanged.

ExifTool Version Number : 12.70 File Name : 0.dng Directory : . File Size : 5.0 MB File Modification Date/Time : 2023:12:07 17:45:34+08:00 File Access Date/Time : 2023:12:07 17:46:15+08:00 File Inode Change Date/Time : 2023:12:07 17:45:34+08:00 File Permissions : -rw-rw-r-- File Type : DNG File Type Extension : dng MIME Type : image/x-adobe-dng Exif Byte Order : Little-endian (Intel, II) Subfile Type : Full-resolution image Image Width : 2448 Image Height : 2048 Bits Per Sample : 8 Compression : Uncompressed Photometric Interpretation : Color Filter Array Image Description : {"shape": [2048, 2448]} Make : HIKVISION Camera Model Name : MVCA050GC12 Strip Offsets : 352 Orientation : Horizontal (normal) Samples Per Pixel : 1 Rows Per Strip : 2048 Strip Byte Counts : 5013504 X Resolution : 1 Y Resolution : 1 Resolution Unit : None Software : tifffile.py CFA Repeat Pattern Dim : 2 2 CFA Pattern 2 : 0 1 1 2 DNG Version : 1.4.0.0 DNG Backward Version : 1.3.0.0 CFA Pattern : [Red,Green][Green,Blue] Image Size : 2448x2048 Megapixels : 5.0

[teboli]

Hello,

This is indeed odd. Can you provide the full stacktrace of exiftool on one of your .dng image please? It looks like there are some missing in your current metadata if I believe the current stacktrace you shared. Here is what I get with the "exiftool" command line on one of the .dng image of the Samsung G8 sequence of the companion demo:

xifTool Version Number : 12.42 File Name : im_00.dng Directory : . File Size : 24 MB File Modification Date/Time : 2020:10:01 09:50:24-04:00 File Access Date/Time : 2023:10:25 13:59:47-04:00 File Inode Change Date/Time : 2023:10:25 13:59:47-04:00 File Permissions : -rwxrwxrwx File Type : DNG File Type Extension : dng MIME Type : image/x-adobe-dng Exif Byte Order : Little-endian (Intel, II) Subfile Type : Full-resolution image Image Width : 4032 Image Height : 3024 Bits Per Sample : 16 Compression : Uncompressed Photometric Interpretation : Color Filter Array Image Description : Make : samsung Camera Model Name : SM-G950F Strip Offsets : (Binary data 25847 bytes, use -b option to extract) Orientation : Unknown (9) Samples Per Pixel : 1 Rows Per Strip : 1 Strip Byte Counts : (Binary data 15119 bytes, use -b option to extract) X Resolution : 72 Y Resolution : 72 Planar Configuration : Chunky Resolution Unit : inches Software : samsung/dreamltexx/dreamlte:9/PPR1.180610.011/G950FXXSADTI1:user/release-keys Modify Date : 2020:09:30 15:02:01 CFA Repeat Pattern Dim : 2 2 CFA Pattern 2 : 1 0 2 1 Copyright : Exposure Time : 1/2924 F Number : 1.7 ISO : 50 Date/Time Original : 2020:09:30 15:02:01 Focal Length : 4.2 mm TIFF-EP Standard ID : 1 0 0 0 DNG Version : 1.4.0.0 DNG Backward Version : 1.1.0.0 Unique Camera Model : SM-G950F-samsung-samsung CFA Plane Color : Red,Green,Blue CFA Layout : Rectangular Black Level Repeat Dim : 2 2 Black Level : 0 0 0 0 White Level : 1023 Default Scale : 1 1 Default Crop Origin : 8 8 Default Crop Size : 4016 3008 Color Matrix 1 : 0.6748046875 -0.0888671875 -0.1123046875 -0.50390625 1.40625 0.0673828125 -0.2197265625 0.4951171875 0.4833984375 Color Matrix 2 : 1.24609375 -0.517578125 -0.1650390625 -0.4130859375 1.412109375 0.1162109375 -0.0654296875 0.322265625 0.693359375 Camera Calibration 1 : 0.9541015625 0 0 0 1 0 0 0 0.9814453125 Camera Calibration 2 : 0.9541015625 0 0 0 1 0 0 0 0.9814453125 As Shot Neutral : 0.46875 1 0.75390625 Baseline Exposure : 0 Calibration Illuminant 1 : D65 Calibration Illuminant 2 : Standard Light A Active Area : 0 0 3024 4032 Forward Matrix 1 : 0.7158203125 0.0234375 0.2255859375 0.2578125 0.7421875 0 0.044921875 -0.56640625 1.346679688 Forward Matrix 2 : 0.60546875 0.091796875 0.2666015625 0.150390625 0.8369140625 0.013671875 -0.0576171875 -0.90625 1.7890625 Opcode List 2 : GainMap, GainMap, GainMap, GainMap Opcode List 3 : Noise Profile : 0.000180710882 0 0.000180710882 0 0.000180710882 0 Aperture : 1.7 CFA Pattern : [Green,Red][Blue,Green] Image Size : 4032x3024 Megapixels : 12.2 Shutter Speed : 1/2924 Focal Length : 4.2 mm Light Value : 14.0

Our code expects to have some of these information to preprocess the raw images as we expect. Hope it may help!

Best,

Thomas

[y2k0999]

Hi,

I provided the full output from exiftool. Maybe the issue lies in my input data missing certain exif tags. The DNGs I made were created myself, as the adobe DNG tool doesn't handle raw images that are just a flattened tensor of pixel values. I guess it expects some header from common raw image formats and doesn't allow for manual input of metadata.

Best

[teboli]

Hi,

Yup what you say is most likely what could explain the issue. In case of missing tags, you can hardcode white balance weights such as 2.1 for the red channel, 1 for the green channel and 1.9 for the blue channel. These are the mean values of the Darmstadt raw denoising dataset as reported by Brooks et al. in https://arxiv.org/abs/1811.11127. In case of missing white balance weights, these values would definitely help to remove the greenish tint your images reveal.

Best,

Thomas

[y2k0999]

Wow that fixed it! That would've taken me so long to find on my own. The white_balance variable was originally = [0,1,0,0]. I got the colors looking normal enough with [1,1,1,1].

Thanks!