python isp result different with matlab code

[oneTaken]

Hey, thanks for your python implementation of the isp code.

I tried with the python isp code, and I found the result is really different with the matlab result.

I set the icrf_index and the pattern_index the same.

Does the python implementation is the same as the matlab?

[GuoShi28]

@oneTaken Sorry for the late reply. It's actually different. For python code, I refer to other paper and complete the isp process in CBDNet, including color transformation, white balance and etc. If you want to totally follow the experiment setting of CBDNet using python, these ISP released codes can be easily modified to original CBDNet. You can also refer to other papers and I holp these code can also help you to reproduce other works. If there are other questions and errors, please feel free to contact me. Thank you!

[oneTaken]

@GuoShi28 Thanks for your reply. So, it's a choice to choose which way. Can I ask why you change the implement function? Is the original ways before is not good enough? I find the new python implement results of noise is strange.

[GuoShi28]

@oneTaken , The Python Code contains a more complete process for ISP process. I'm not sure this format can definitely obtain better results (For sRGB, I have obtained a better result. For RAW, I am still testing and may further update my code if some mistake observed). Actually, you can reproduce Matlab code by a simple change the Python code. I add many commits in the code and hope readers can understand more easily through reading them.

[GuoShi28]

Also you can refer to these materials for more details.

[JimmyChame]

Hi，I use the python code to convert the RGB image to RAW and invert the process. I found some zipper effect (color-mixing artifact) show up. I guess some problems may happen in the demosaic process. Did you meet this problem when you were testing? @GuoShi28

[GuoShi28]

I use the same demosaic algorithm with Matlab code. Yes, there should contain the color-mixing artifact, since the mosaic process and demosaic process are not totally inverse and some information lost in this process. Thus I set ground-truth not directly the sRGB image but isp results of the sRGB image (See the code below:)

# -------- ISP PROCESS --------------------------
        # Step 5 : White Balance
        wb_mask = self.WB_Mask(img_mosaic_noise, pattern, 1/self.fr_now, 1/self.fb_now)
        img_mosaic_noise = img_mosaic_noise * wb_mask
        img_mosaic_noise = np.clip(img_mosaic_noise, 0, 1)
        img_mosaic_gt = gt_img_mosaic * wb_mask
        img_mosaic_gt = np.clip(img_mosaic_gt, 0, 1)
        # Step 4 : Demosaic
        img_demosaic = self.Demosaic(img_mosaic_noise, pattern=self.pattern)
        img_demosaic_gt = self.Demosaic(img_mosaic_gt, pattern=self.pattern)
        # Step 3 : from Cam to XYZ
        img_IXYZ = self.CAM2XYZ(img_demosaic, M_xyz2cam=self.M_xyz2cam)
        img_IXYZ_gt = self.CAM2XYZ(img_demosaic_gt, M_xyz2cam=self.M_xyz2cam)
        # Step 2 : frome XYZ to RGB
        img_IL = self.XYZ2RGB(img_IXYZ)
        img_IL_gt = self.XYZ2RGB(img_IXYZ_gt)
        # Step 1 : tone mapping
        img_Irgb = self.CRF_Map(img_IL, index=icrf_index)
        img_Irgb_gt = self.CRF_Map(img_IL_gt, index=icrf_index)

        return img_Irgb_gt, img_Irgb

I have two suggestions: (1) sent the color artifact image on Github. I can check whether my understanding is correct. (2) You can train the network using this code on color denoising to check if the model can handle the real-world denoising problem. Since I simple train a small network using this python code on sRGB denoising task and find it could work. If you have other questions or find mistakes in my code, please feel free to contact me. @JimmyChame

[JimmyChame]

Thank you for your careful reply ! Here are some examples of my tests. I run the ISP test 2. The first subfigure is the original image, the middle one is the result of inverse ISP. Some color-mixing artifacts appear in the textual areas like water.

[GuoShi28]

(1) I treat this as the drawback of demosaicking now. And if I find some other reason or mistake for this problem, I will update my code and let you know. (2) I using this training pairs to train the network and find that I can works. I still suggest you can use a small network to train and see the denoising results.

[BaiYu0120]

(1) I treat this as the drawback of demosaicking now. And if I find some other reason or mistake for this problem, I will update my code and let you know. (2) I using this training pairs to train the network and find that I can works. I still suggest you can use a small network to train and see the denoising results.

To solve the issues, how about add the delta to original groundtruth images. Noisy_final = (Noisy_inverse_ISP - GT_inverse_ISP) + GT_original