Denoising with GAN
Introduction
Animation movie companies like Pixar and Dreamworks render their 3d scenes using a technique called Pathtracing which enables them to create high quality photorealistic frames. Pathtracing involves shooting 1000’s of rays into a pixel randomly(Monte Carlo) which will then hit the objects in the scene and based on the reflective property of the object the rays reflect or refract or get absorbed. The colors returned by these rays are averaged to get the color of the pixel and this process is repeated for all the pixels. Due to the computational complexity it might take 8-16 hours to render a single frame.
We are proposing a neural network based solution for reducing 8-16 hours to a couple of seconds using a Generative Adversarial Network. The main idea behind this proposed method is to render using small number of samples per pixel (let say 4 spp or 8 spp instead of 32K spp) and pass the noisy image to our network, which will generate a photorealistic image with high quality.
Demo Video Link
Table of Contents
Installation
To run the project you will need:
Running
Once you have all the depenedencies ready, do the folowing:
Download the dataset extract it to a folder named 'dataset' (ONLY if you want to train, not needed to run).
Extract the CKPT files to a folder named 'Checkpoints'
Run main.py -- python3 main.py
Go to the browser, if you are running it on a server then [ip-address]:8888, if you are on your local machine then localhost:8888
Dataset
We picked random 40 images from pixar movies, added gaussian noise of different standard deviation, 5 sets of 5 different standard deviation making a total of 1000 images for the training set. For validation we used 10 images completely different from the training set and added gaussian noise. For testing we had both added gaussian images and real noisy images.
Hyperparameters
- Number of iterations - 10K
- Adversarial Loss Factor - 0.5
- Pixel Loss Factor - 1.0
- Feature Loss Factor - 1.0
- Smoothness Loss Factor - 0.0001
Results
3D rendering test data:
Real noise images:
CT-Scan:
Improvements
- Increase the num of iteration to 100K.
- Train the network for different noises.
- Make it work on a real-time app.