search

yzxing87 / Invertible-ISP

[CVPR2021] Invertible Image Signal Processing
MIT License
338 stars 39 forks source link

Data generalization and incorrect highlight color #12

Open SuTanTank opened 2 years ago

SuTanTank commented 2 years ago

From my understand, the trained model provides an invertible function that can convert between Raw and RGB image. The GT RGBs used in training are not the JPEGs straight out of camera but generated by Rawpy. So the model simulates the process of Rawpy, which is relatively simple comapred to the ISP inside the real camera(e.g. most likely no local tone mapping is used, neither the camera manufacturer's proprietary color profile). Therefore, the trained model only applies to a specific RAW to Jpeg process, with fixed ISP parameters. White balance, which is most likely different between photos, is conducted in preprocess, but other ISP tunable parameters like tone curve, 3D LUT, color temperature-related CCM, lens shading are left for the network to simulate, which leads to my first question:

  1. How does this method perform for Jpeg that is not generated by Rawpy? Do we need to know how the Jpeg is generated before we reconstruct the raw?

I tried the pretrained model with NIKON data, and found the simulated Raw to Jpeg process cause visible hightlight color shifting, as shown in the top left corner:

gt_pred_a0341-dgw_002_00000

  1. Is that incorrect highlight color a bug or known issue?
  2. Do you have any idea to preserve the highlight color? One suggestion may be training a 3DLUT to match color. 3DLUT is somewhat invertible also.
yzxing87 commented 2 years ago

Hi, thanks for the issue. We choose to take rawpy ISP as a reference since we have trouble finding other eligible paired datasets before submission. Here are my thoughts on your questions.

  1. This method is trained to simulate one kind of ISP at a time. Thus, after training, we need to know which ISP we need to invert to reconstruct the raw.
  2. I think the incorrect highlight color is due to some numerical issue. But I am not sure --- I will follow up when I can come to a firm conclusion.
  3. My guess is that if it is indeed a failure case of our method, 3DLUT would also face the same highlight issue. Still, let me check with the problem first.
LinQinLiang commented 1 year ago

我猜测这是由于在数据处理的过程中缺少tonemap这个操作