Pretrained model

[XenonLamb]

Can you upload a pretrained pytorch model, or the code to load the matlab models provided by the original authors? I find it hard to reproduce their results using this pytorch code (typically, a lot of artifacts appear in the reconstruction) Thanks!

[IDKiro]

im sorry but no. the model is missing.

can you detail the 'artifacts'?

[IDKiro]

since you do not reply, i'll close the issue

[XenonLamb]

Sorry for the late reply. By "artifacts" I mean after training, I tested the model on SIDD validation set. The resulting images has some color shift, and contains blocky and other irregular artifacts in almost every image, including the flat ones.

[XenonLamb]

Also I noticed that the pytorch model is different from the matlab one given by the authors:

1. Why do we share the weights of 3 conv layers in the middle of the FCN network?
2. In the unet, inc should have one more conv2d, conv2 should have one more conv2d, and the authors used 2x2 stride 2 conv2d followed by 1x1 stride 1 conv2d for downsampling instead of average pooling.

[IDKiro]

Sorry for the late reply. By "artifacts" I mean after training, I tested the model on SIDD validation set. The resulting images has some color shift, and contains blocky and other irregular artifacts in almost every image, including the flat ones.

The 'artifacts' does exist and I observed it half year ago. The main reason for this is that the quality of the RENOIR dataset is not high enough. For example, the brightness of some areas of the image in the dataset is mismatched. In contrast, the SSID dataset is a very high-quality dataset. Training a UNet on the SSID can achieve a much higher PSNR than CBDNet (at least 0.5db)

In addition, the noise generation method in the CBDNet paper is not the same as the official repo. Especially the official python code, which is written based on the algorithms in the UIP paper.

And maybe there is some error in the method of generating noise in my repo.

If you are interested in how to generate realistic noise, you can read UIP papers and official repo.

Some materials:

https://github.com/GuoShi28/CBDNet/blob/master/SomeISP_operator_python/isp_implement.md https://arxiv.org/pdf/1811.11127.pdf https://github.com/google-research/google-research/tree/master/unprocessing

[XenonLamb]

The 'artifacts' does exist and I observed it half year ago. The main reason for this is that the quality of the RENOIR dataset is not high enough. For example, the brightness of some areas of the image in the dataset is mismatched. In contrast, the SSID dataset is a very high-quality dataset. Training a UNet on the SSID can achieve a much higher PSNR than CBDNet (at least 0.5db)

Thanks for providing such information! I also observed superior performance by training an edsr-like model with SIDD instead of using CBDNet.

[XenonLamb]

But there's another thing worth noticing: I tried to adjust the model of this pytorch repo to the structure provided by CBDNet Repo and loaded all the network parameters from their .mat checkpoint. The wierd thing is that when I did testing, the resulting images look almost identical with the noisy one. Using their matconvnet code, the resulting image seem just fine.

[IDKiro]

Also I noticed that the pytorch model is different from the matlab one given by the authors:

1. Why do we share the weights of 3 conv layers in the middle of the FCN network?
2. In the unet, inc should have one more conv2d, conv2 should have one more conv2d, and the authors used 2x2 stride 2 conv2d followed by 1x1 stride 1 conv2d for downsampling instead of average pooling.

1. I don't remember, I wrote the code based on the paper.
2. There should be no specific downsampling method mentioned in the original paper. Different downsampling methods tend to have same performance. I haven't written a neural network in MATLAB, but I think what you mentioned (2x2 stride 2 conv2d followed by 1x1 stride 1 conv2d) is just the avgpooling in MATLAB, and you can see if these convolution kernels are constant.

In fact, there is nothing special about CBDNet's network structure (FCN+UNet). In my experiments, the nuances of the network structure did not make any difference. And the FCN does not improve the performance, but it does provide the feature of adjusting the denoising intensity.

[IDKiro]

But there's another thing worth noticing: I tried to adjust the model of this pytorch repo to the structure provided by CBDNet Repo and loaded all the network parameters from their .mat checkpoint. The wierd thing is that when I did testing, the resulting images look almost identical with the noisy one. Using their matconvnet code, the resulting image seem just fine.

I cannot answer the question of model conversion. Since this repo was written at least six months ago, I have forgotten many details, so I recommend that you just try your best.

[IDKiro]

CBDNet is indeed a very important job, but I recommend that you study the new top jobs on the DND dataset.