DCNv2+Cuda11

[nonick2k23]

Is there a way to update your DCNv2 code to support Cuda 11? Cuda 10 is very old and newer GPUs don't work with it.

Since your DCNv2 code is modified, only you can update it...

[Algolzw]

You can try this repo: https://github.com/jinfagang/DCNv2_latest

[nonick2k23]

Your DCNv2 contains the following functions:

• DCN_sep(DCNv2)
• FlowGuidedDCN(DCNv2)
• InsideFlowGuidedDCN(DCNv2):
• _DCNv2Pooling(Function):

Those functions are not present in any updated version of DCNv2, so the other versions do not work...

[Algolzw]

Your DCNv2 contains the following functions:

• DCN_sep(DCNv2)
• FlowGuidedDCN(DCNv2)
• InsideFlowGuidedDCN(DCNv2):
• _DCNv2Pooling(Function):

Those functions are not present in any updated version of DCNv2, so the other versions do not work...

Yes, but the basic DCNv2 files are the same. So could you just replace the compiled dcn_v2.py with our dcn_v2.py?

[nonick2k23]

I got it to compile and work..

but when I infer on ntire21 burstsr dataset using the weights you provide, I receive 33PSNR (in contrary to 48 you wrote in the paper)

[Algolzw]

If you use the same testing code as in the test_real.py?

And please note the PSNR should be AlignedPSNR since the outputs and the ground truths are not aligned in real-world dataset.

[nonick2k23]

Yes. I've used test_real and provided it the arguments:

1. burst_sr validaton set - 882 bursts
2. bsrt_large_realworld.pth

These are the results:

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 882/882 [06:49<00:00, 2.15it/s] avg PSNR: 32.646484 avg SSIM: 0.730539 avg LPIPS: 0.212212 avg time: 0.327914

[Algolzw]

Could you also try the tiny model? As

python test_real.py --n_GPUs 1 --model BSRT --model_level S --swinfeature --batch_size 1 --burst_size 14 --patch_size 80 --pre_train ../train_log/bsrt/real_models/bsrt_tiny/bsrtbest_epoch.pth --root /data/dataset/ntire21/burstsr/real

[nonick2k23]

I do not have "bsrtbest_epoch.pth" file. What I have is:

1. bsrt_large_realworld.pth
2. bsrt_small_realworld.pth

Those were downloaded from here: https://drive.google.com/file/d/1Bv1ZwoE3s8trhG--wjB0Yt6WJIQPpvsn/view

Results on "small":

avg PSNR: 33.421289 avg SSIM: 0.741668 avg LPIPS: 0.205523 avg time: 0.205678

[Algolzw]

Ya, I see. It seems strange because, in my experience, the model can achieve 40+ dB even using a single image. Could you provide the command you used for testing? I'm not sure if it is caused by the different versions of DCN module.

[nonick2k23]

This is the command line: (test_real is unmodified)

python test_real.py --n_GPUs 1 --model BSRT --model_level S --swinfeature --batch_size 1 --burst_size 14 --patch_size 80 \
                    --pre_train /home/alguser/research/superresolution/BSRT/code/real/bsrt/weights/bsrt_small_realworld.pth \
                    --root /home/alguser/research/superresolution/data/ntire21/burstsr_dataset

[Algolzw]

The command seems indeed correct.

Then I have no idea about it -_-. Maybe you can save the inputs, output, and GT images (by uncommenting the code in test_real.py - line 84-98) and see what happens.

You can also test the synthetic dataset. And if you found that this is indeed caused by the DCN, you may need to retrain the model (usually a week).

[nonick2k23]

Can you detail the training process you have done in the paper?

Did you train on real data only, or did you train on synthetic then on real data?

How did you train and achieve 48+ on burst_sr dataset real case?

I'll train at my server and hopefully I can reproduce the results...

[Algolzw]

I train the model on synthetic images and then finetune it on real data.

Maybe you can use our pretrained model and fine-tune it (also first on synthetic then on real data).

[nonick2k23]

Small update, synthetic weights provide results on par with the ones in the paper on synthetic data.

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 300/300 [00:48<00:00,  6.15it/s]
avg PSNR: 42.721505
avg SSIM: 0.970805
avg LPIPS: 0.031344
 avg time: 0.145964

Which means the updated DCNv2 seems fine, and your weights of the synthetic model is fine. It seems that your real-world weights are not the ones you used in the competition/paper.

Can you dig around in your files and find the correct model weights? Maybe look for the file "bsrt/real_models/bsrt_tiny/bsrt_best_epoch.pth" under your logs folder?

Thanks.

[Algolzw]

OK, seems there are some problems in the code/weights, I will check again and update it.

[nonick2k23]

hope you can upload the new version of the code or explain the changes needed when you can

Thanks

[Algolzw]

Hi, I found the problem is caused by the new version of [torch.lstsq].

The function torch.lstsq(B, A) of old version (torch1.6) changes to torch.linalg.lstsq(A, B), which means you only need to change the inputs order from c = torch.lstsq(ir.t(), iq.t()) to c = torch.linalg.lstsq(iq.t(), ir.t()) at "match_colors" function (line 65).

[qtct]

I got it to compile and work..

but when I infer on ntire21 burstsr dataset using the weights you provide, I receive 33PSNR (in contrary to 48 you wrote in the paper)

Hi,I want to ask you how you compile deformable convolutions, is it the way the author proposed?

[nonick2k23]

Hi, I found the problem is caused by the new version of [torch.lstsq].

The function torch.lstsq(B, A) of old version (torch1.6) changes to torch.linalg.lstsq(A, B), which means you only need to change the inputs order from c = torch.lstsq(ir.t(), iq.t()) to c = torch.linalg.lstsq(iq.t(), ir.t()) at "match_colors" function (line 65).

Deprecated functions behave the same - until removed in future release.

I've ran your suggestion anyway and still received PSNR of 32.5:

avg PSNR: 32.643951 avg SSIM: 0.729982 avg LPIPS: 0.212303 avg time: 0.329302

Are you sure you are using the weight file found at your github page? I believe the issue lies there...

[nonick2k23]

I got it to compile and work.. but when I infer on ntire21 burstsr dataset using the weights you provide, I receive 33PSNR (in contrary to 48 you wrote in the paper)

Hi,I want to ask you how you compile deformable convolutions, is it the way the author proposed?

Yes - Combine the modified python file with the updated DCN and compile with cuda11 and pytorch 1.6+

You will get warnings along the way but it will compile successfully.

[Algolzw]

Hi, I found the problem is caused by the new version of [torch.lstsq]. The function torch.lstsq(B, A) of old version (torch1.6) changes to torch.linalg.lstsq(A, B), which means you only need to change the inputs order from c = torch.lstsq(ir.t(), iq.t()) to c = torch.linalg.lstsq(iq.t(), ir.t()) at "match_colors" function (line 65).

Deprecated functions behave the same - until removed in future release.

I've ran your suggestion anyway and still received PSNR of 32.5:

avg PSNR: 32.643951 avg SSIM: 0.729982 avg LPIPS: 0.212303 avg time: 0.329302

Are you sure you are using the weight file found at your github page? I believe the issue lies there...

Hi,

I am sure this is the correct weight. I have tested it on my server and the final PSNR is 48, which is exactly the same as the paper.

Can you provide your match_colors function? And can you check if the PWC weights are also well-downloaded?

[nonick2k23]

def match_colors(im_ref, im_q, im_test, ksz, gauss_kernel):
    """ Estimates a color transformation matrix between im_ref and im_q. Applies the estimated transformation to
        im_test
    """
    gauss_kernel = gauss_kernel.to(im_ref.device)
    bi = 5

    # Apply Gaussian smoothing
    im_ref_mean = apply_kernel(im_ref, ksz, gauss_kernel)[:, :, bi:-bi, bi:-bi].contiguous()
    im_q_mean = apply_kernel(im_q, ksz, gauss_kernel)[:, :, bi:-bi, bi:-bi].contiguous()

    im_ref_mean_re = im_ref_mean.view(*im_ref_mean.shape[:2], -1)
    im_q_mean_re = im_q_mean.view(*im_q_mean.shape[:2], -1)

    # Estimate color transformation matrix by minimizing the least squares error
    c_mat_all = []
    for ir, iq in zip(im_ref_mean_re, im_q_mean_re):
        #c = torch.lstsq(ir.t(), iq.t())
        c = torch.linalg.lstsq(iq.t(), ir.t())
        c = c.solution[:3]
        c_mat_all.append(c)

    c_mat = torch.stack(c_mat_all, dim=0)
    im_q_mean_conv = torch.matmul(im_q_mean_re.permute(0, 2, 1), c_mat).permute(0, 2, 1)
    im_q_mean_conv = im_q_mean_conv.view(im_q_mean.shape)

    err = ((im_q_mean_conv - im_ref_mean) * 255.0).norm(dim=1)

    thresh = 20

    # If error is larger than a threshold, ignore these pixels
    valid = err < thresh

    pad = (im_q.shape[-1] - valid.shape[-1]) // 2
    pad = [pad, pad, pad, pad]
    valid = F.pad(valid, pad)

    upsample_factor = im_test.shape[-1] / valid.shape[-1]
    valid = F.interpolate(valid.unsqueeze(1).float(), scale_factor=upsample_factor, mode='bilinear', align_corners=False)
    valid = valid > 0.9

    # Apply the transformation to test image
    im_test_re = im_test.view(*im_test.shape[:2], -1)
    im_t_conv = torch.matmul(im_test_re.permute(0, 2, 1), c_mat).permute(0, 2, 1)
    im_t_conv = im_t_conv.view(im_test.shape)

    return im_t_conv, valid

Line 65 changed as you stated above. PWCNet is fine (Downloaded from your website) Weights are fine (downloaded from your website as well)

[Algolzw]

Looks correct. Then it's really strange because I only changed this function and the psnr increases from ~32 to 48.

Can you save the output images and compared with my results? I provide the first 5 SR images here:

bsrt.zip

[Algolzw]

You can just uncomment these lines:

[nonick2k23]

can you attach the weights file and the pwcnet file you have used for these results?

I need to eliminate these 2 variables, hopefully i'll find the bug in my code.

If the results are the same I'll compare your output and mine after this attempt.

Thanks!

[Algolzw]

Sure no problem! Here is the weight I used, which should be the same as I provided in this repo.

bsrt_small_realworld.pth.zip

[nonick2k23]

Same PSNR, 33.5

[Algolzw]

You can compare the produced results with mine, to see if the SR images look good. If so, the problem might be the evaluation codes (i.e., in calculating psnr...).

[nonick2k23]

I didn't change any code except DCNv2 (which seems to work fine) and line 65 as you suggested. So the results should be identical for me and for you, but it is not the case.

Can you attach the PWCNet weights you have used?

Thanks.

[Algolzw]

Hi, I use exactly the same PWCNet weight I provided, I only have 1 weight file so it must not be the problem.

[Algolzw]

I provide all my used weights here: https://github.com/Algolzw/BSRT/releases/tag/weights

[nonick2k23]

Thanks!

I managed to find the issue. It was an incorrect spynet weights file.

This is the correct one to be used: https://github.com/JingyunLiang/VRT/releases/download/v0.0/spynet_sintel_final-3d2a1287.pth

Results: [using large model weights on real data]

avg PSNR: 48.575360
avg SSIM: 0.985441
avg LPIPS: 0.021139
avg time: 0.307091

Results: [using small model weights on real data]

avg PSNR: 48.498242
avg SSIM: 0.985456
avg LPIPS: 0.021455
avg time: 0.197311

[Algolzw]

Very nice to hear! Thank you for providing these results!