What role does ISP play?

[TwilightArchon]

Thanks for your work! I'm wondering, in data preparation, "For sRGB noise removal, we utilized the ISP/inverse ISP pipelines from CBDNet and clean images from DIV2K, and synthesized noisy images based on Poisson-Gaussian noise with the fixed noise level." But what role does ISP and inverse ISP play, since it's not found anywhere in the paper? Do you use the knowledge of ISP during training or also during the actual denoising of real life images? Is the good real world denoising results come from ISP and not the network structure of CLIP?

[alwaysuu]

Thanks for your question. The ISP pipeline is used to transfer the synthetic noisy images in the raw domain into the sRGB domain for the subsequent supervised training. The ISP model is taken from CBDNet and has a gap with real sRGB images. Moreover, we used a fixed level of Poisson plus Gaussian noise in the raw domain to synthesize noisy images, rather than a range of noise levels used in CBDNet. Such a setting further enlarges the gap between training and testing. See Section 4.1 for details. We tried to randomly initialize the RN50 encoder and trained the network, but the denoising performance was inferior to the frozen RN50 of CLIP (about 1dB), which verified the effectiveness of the proposed method.

[TwilightArchon]

Thank you so much for your answer! I'm still a bit confused about some details in the paper.

1. Why use inverse ISP to get raw data and apply Possion-Gaussian noise? If CLIP is generalizable, shouldn't it generalize to real world noise if trained on synthetic noise? If we train on possion-gaussian noise added during RAW, how do we know that it's not overfitting the noise pattern?
2. Can I have the pretrained model please? I just sent the request yesterday for the authorization to google doc. Thanks a lot!

[alwaysuu]

1. Ideally, we wish a generalizable denoiser trained on e.g., Gaussian noise, can handle any other noises, e.g., real noisy sRGB images from mobile phones, but this is difficult to achieve. We thus consider a simpler task setting, in which the training data distribution does not deviate significantly from the test data distribution. As a result, we chose to simulate noisy sRGB images for training and then test on real-world sRGB image datasets. Our paper utilized frozen RN50 of CLIP to achieve a good generalization. If you randomly initialize the RN50 or finetune it, the corresponding denoising performance will decline.
2. The pre-trained models have been provided in this repo.

[alwaysuu]

I will close this issue. Feel free to reopen it if you have any further questions.

[TwilightArchon]

Sorry for the late reply. I didn't see the email coming in.

Thank you so much for the explanation! I understand the problem now.

Also, yes I saw the pretrained model in the repo, which is a link to google drive, but it requires authorization. Can you please grant me an authorization? Thank you!

[alwaysuu]

You can access these files through the link now.

[TwilightArchon]

It works now, thank you!