Reproduce Results in the Paper

[TwilightArchon]

Hello! I'm using the image in the paper as an input to test on the real world denoising of the model, but above shown that the model is not performing correctly, as the first image shows the original image, and second shows the denoised image.

I'm wondering is this because I commented out the load pretrain encoder, since it cause an error.

If so, can you please provide with the pretraiend encoder please? I did not find it anywhere in the repo or in the google drive link. Thank you!

[alwaysuu]

The provided pre-trained model (full model) already contains the frozen RN50 encoder, so it should work without loading the RN50 during the model initialization, and my local test is normal. What changes did you make during the experiment?

[TwilightArchon]

I deleted the place that require me to load the diffusion/RN50 net. Also, I changed the file path, and that's all.

[alwaysuu]

That is waired. Did you use the right checkpoint from the sRGB folder for sRGB noise removal?

[alwaysuu]

Any updates? I utilized the denoiser trained on Gaussian std=15 to remove sRGB noise and obtained the same result as yours. Currently, our method trained on Gaussian std=15 cannot directly handle real sRGB noisy images, e.g., SIDD, which is complicated and exhibits a significant domain gap.

[TwilightArchon]

Hi there! I just downloaded the pretrained model from google doc again, from the sRGB folder. I tried again, but the results were similar. Is there a possibility that you accidentally put the pretrained model on gaussian std=15 into the sRGB folder? I think I have the same setting with you in all other parts. Thank you!

[alwaysuu]

I checked the md5 of the uploaded files and they are identical to my local files. I also checked the entire github code in another Linux server, and they run normally. Can you try the following things to debug:

1. Check the md5 value of your downloaded checkpoint for sGRB noise. Mine is 20af4b12502b442e5da392fbc5b6d849
2. Try test_synthetic_denoising.py to see if the code can work for synthetic noise
3. Check your torch environment. I used torch1.10 (or torch 1.11)+python3.9. The torch version may matter
4. Can you paste the full code of your test_real_denoising_sRGB.py here so I can double-check.

[TwilightArchon]

Hello! Thanks for your reply!

1. I checked, and it is exactly the same as yours
2. Yes, synthetic noise with std=15 gaussian seems to work fine.
3. I used version of torch = 2.2.2 and python = 3.11, is it the issue?
4. Sure! But the code seems to be not compatible with the files to be uploaed. I'll paste the code down below. import argparse import numpy as np import os import sys import torch print(torch.version) print(torch.version.cuda) print(torch.cuda.is_available()) from PIL import Image sys.path.append('F:\work\CLIPDenoising') import torch.nn.functional as F import utils from glob import glob from scipy.ndimage import convolve from tqdm import tqdm

from basicsr.models.archs.CLIPDenoising_arch import CLIPDenoising

os.environ['CUDA_VISIBLE_DEVICES'] = '1'

parser = argparse.ArgumentParser(description='Synthetic Color Denoising')

parser.add_argument('--input_dir', default='F:\work/video_to_images/fly_clipped', type=str, help='Directory of validation images')

args = parser.parse_args()

def proc(tar_img, prd_img):
PSNR = utils.calculate_psnr(tar_img, prd_img) SSIM = utils.calculate_ssim(tar_img, prd_img) return PSNR, SSIM

network arch

''' type: CLIPDenoising inp_channels: 3 out_channels: 3 depth: 5 wf: 64 num_blocks: [3, 4, 6, 3] bias: false model_path: /data0/cj/model_data/ldm/stable-diffusion/RN50.pt

aug_level: 0.025 '''

model_restoration = CLIPDenoising(inp_channels=3, out_channels=3, depth=5, wf=64, num_blocks=[3,4,6,3], bias=False, model_path=None, aug_level=0.025) checkpoint = torch.load('./denoising/pretrained_model/net_g_latest.pth') load_result = model_restoration.load_state_dict(checkpoint['params'])

model_restoration.cuda() model_restoration.eval() ##########################

factor = 32

testsets = ['CC', 'PolyU', 'SIDD_val','fly']

for testset in testsets:

if testset == 'CC':

noises = sorted(glob(os.path.join(args.input_dir, testset, '*real.png') ))

cleans = sorted(glob(os.path.join(args.input_dir, testset, '*mean.png') ))

index = -9

elif testset == 'PolyU':

noises = sorted(glob(os.path.join(args.input_dir, testset, '*real.JPG') ))

cleans = sorted(glob(os.path.join(args.input_dir, testset, '*mean.JPG') ))

index = -9

elif testset == 'SIDD_val':

cleans = sorted(glob(os.path.join(args.input_dir, 'SSID/SIDD_test_PNG', 'GT/*.png')))

noises = sorted(glob(os.path.join(args.input_dir, 'SSID/SIDD_test_PNG', 'noisy/*.png')))

cleans=sorted(glob(os.path.join(args.input_dir, '.jpg'))) noises=sorted(glob(os.path.join(args.input_dir, '.jpg'))) print(args.input_dir) print(cleans) print(noises) psnr_list = []; ssim_list = [] for noise, clean in tqdm(zip(noises, cleans)):

with torch.no_grad():
    torch.cuda.ipc_collect()
    torch.cuda.empty_cache()
    img_clean = utils.load_img(clean)

    img = np.float32(utils.load_img(noise))/255.

    img = torch.from_numpy(img).permute(2,0,1)
    input_ = img.unsqueeze(0).cuda()

    # Padding in case images are not multiples of 8
    h,w = input_.shape[2], input_.shape[3]
    H,W = ((h+factor)//factor)*factor, ((w+factor)//factor)*factor
    padh = H-h if h%factor!=0 else 0
    padw = W-w if w%factor!=0 else 0
    input_ = F.pad(input_, (0,padw,0,padh), 'reflect')

    restored = model_restoration(input_)

    # Unpad images to original dimensions
    restored = restored[:,:,:h,:w]

    restored = torch.clamp(restored,0,1).cpu().detach().permute(0, 2, 3, 1).squeeze(0).numpy()
    restored = (restored * 255.0).round().astype(np.uint8)

    restored_image = Image.fromarray(restored)
    output_path = os.path.join('F:/work/CLIPDenoising/output', 'restored_' + os.path.basename(noise))
    restored_image.save(output_path)

    psnr, ssim = proc(img_clean, restored)

    psnr_list.append(psnr); ssim_list.append(ssim)

print('dataset:{}, psnr:{:.2f}, ssim:{:.3f}'.format(['fly'], sum(psnr_list)/len(psnr_list), sum(ssim_list)/len(ssim_list)))

[alwaysuu]

Thanks for the response. Well, I cannot find something wrong in the code. I also checked the github code under torch2.3 and python3.10, and it runs normally. Currently, the only thing I can suspect is your input data. Can you try the following image in PNG format?

If this still doesn't work, I will keep this issue open and see if someone else will encounter this.

[TwilightArchon]

Hey there! I just tried these two pictures, and they worked! I have no idea why though, because the screenshot in jpg doesn't work. Thanks for the reply!