[REQ] Image "restoration" tools

[MarcoRavich]

It could be very interesting to have those cool tools in ML, even to make comparisons !

• https://github.com/tducvu/PIC16B-project
• https://github.com/animalecs/deep_koalarization
• https://github.com/jantic/DeOldify
• https://github.com/shatadal88/Simple-Old-photo-restore-
• https://github.com/flynneastwood/VintagePhotoRestaurator
• https://github.com/awarischool/photo-colorization
• https://github.com/Sarthakh/Automatic-Image-Colorization-Using-Auto-encoders-without-Image-Compression
• https://github.com/AmitNikhade/colorvibes
• https://github.com/Saaketh-J/NewColor
• https://github.com/KorneliuszDyszczynski/ColorizingPhotosNeuralNetwork
• https://github.com/Oleg007003/ColorizationPicturesNetwork
• https://github.com/krisbitney/ImgBuff
• https://github.com/srijag2700/Colorization
• https://github.com/tejasbana/Colorizing_images
• ...

Hope that inspires !

[shirooo39]

I've just added (an extemely simple, rough, and unfinished) implementation of Real-ESRGAN

I'll... think about adding some more later

[MarcoRavich]

Just tested, and looks quite good:

Do you think is possible to adapt it for videos too ? (something like: input video -> FFMPEG decoding -> still images -> Real-ESRGAN -> FFMPEG concat/encoding -> restored/upscaled video)

Thanks !

[shirooo39]

I think that could be possible, but I know nothing about ffmpeg. there's already ffmpeg in my notebook (that I took frome somewhere else) and maybe you can use it to extract each frames, use Real-ESRGAN to upscale them all, and then use ffmpeg again to put them all back together.

though I can't imagine how long that'd take

[MarcoRavich]

Well, the main "problem" could be more the disk space than rendering time...

Anyway, just found this interesting notebook by @tg-bomze that may inspire: https://colab.research.google.com/github/tg-bomze/ENTAR/blob/master/ENTAR_Eng.ipynb

[MarcoRavich]

Another nice find (with colab-demo): Training and testing codes for USRNet, DnCNN, FFDNet, SRMD, DPSR, MSRResNet, ESRGAN, BSRGAN, SwinIR by @cszn

As always, hope that inspire !!

[MarcoRavich]

Clean SwinIR wrapper by @Lin-Sinorodin: https://github.com/Lin-Sinorodin/SwinIR_wrapper

[MarcoRavich]

Very interesting find: HINet - Half Instance Normalization Network for Image Restoration by @megvii-model

With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8 times and 2.9 times speedup respectively. For image deblurring, we get comparable performance with 22.5% of its MACs and 3.3 times speedup on REDS and GoPro datasets. For image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4 times speedup.

Some "real-world" tests by Selur:

Mode: Deblur GoPro

Mode: Deblur REDS

Mode: denoise

Mode: derain

;)

[MarcoRavich]

According to @Lin-Sinorodin:

If you are willing to try it out yourself, it seems from their code that it is pretty straightforward. If you look at the code on this page, It seems that something like this can be a pretty clean and short wrapper:

import cv2
import torch

from basicsr.models import create_model
from basicsr.train import parse_options
from basicsr.utils import FileClient, imfrombytes, img2tensor, padding

# define model
opt = parse_options(is_train=False)
model = create_model(opt)

img_lq = cv2.imread(path, cv2.IMREAD_COLOR)
img_lq = cv2.cvtColor(img_lq, cv2.COLOR_BGR2RGB)

img_lq_tensor = torch.from_numpy(img_lq.transpose(2, 0, 1)).float()

# get high res img
img_hq: np.array = model.my_single_image_inference(img_lq_tensor , output_path)

The main difference is adding a function similar to the inference function that return the image instead of saving it:

def my_single_image_inference (self, img) -> np.array:
    self.feed_data(data={'lq': img.unsqueeze(dim=0)})

    if self.opt['val'].get('grids', False):
        self.grids()

    self.test()

    if self.opt['val'].get('grids', False):
        self.grids_inverse()

    visuals = self.get_current_visuals()
    sr_img = tensor2img([visuals['result']])

    # REPLACE THIS
    # imwrite(sr_img, save_path)

   # WITH THIS
   return sr_img

It's just in a glance, and I probably missed something but the main idea is here. This way it's should be also available to merge this into a more high-level API, that supports both SwinIR and the paper you referenced. Seems like a cool project, and I'll be happy to contribute.

Hope it's helpful, Lin