Cloud-Removal-in-Satellite-Images-using-Conditional-Generative-Adversarial-Networks

Affiliation

Photogrammetry and Remote Sensing Department, Indian Institute of Remote Sensing, ISRO, Dehradun

April 2020 - July 2020

Summary

Cloud cover in the earth's atmosphere is a major issue in temporal optical satellite image processing. Clouds, thick or thin, cover the earth features in a satellite image and hide important information. The objective of this project is to develop a deep learning based automated pipeline to remove cloud cover from optical satellite imagery. Conditional GANs architecture is used to learn the mapping of the cloudy satellite image to it's cloud-free counterpart. A novel augmented and computationally efficient training approach is suggested.

Modules

Built With/Libraries Used/Network Architectures

• Keras (TF Backend)
• Numpy
• Matplotlib
• PatchGAN
• U-Net

Dataset

Sentinel-2 is a part of the opensource earth-observation Copernicus mission of the ESA. It provides high temporal resolution multispectral data (13 bands) at 10m spatial resolution. False Color Composite Images (Green, Red and NIR bands) are used. The study site is chosen based availability of phenological features (crops/agriculture). Removing cloud cover in Sentinel-2 satellite images using only optical data and a novel augmented training approach using conditional GANs.

Currently the training dataset is augmented from a single pair of cloudy and cloud-free images labelled as True and False image and used as Training and Test Data respectively.

The initial augmentation is performed only by rotation of 0, 90, 180 and -90 degrees which gives us 4 pairs, say 1,2,3 and 4.These four pairs are then stacked iteratively 10 times as: [1,2,3,4,1,2,3,4,1,2]. Later on, more data augmentation was implemented by performing skew operations on the cloudy and cloud-free images, and then perform the rotation operations. Now there are 4 pairs of original (unskewed) images, say, [1, 2, 3, 4] and 4 pairs of the skewed images, say [1', 2', 3', 4'] which are stacked as [1,2,3,4,1',2',3',4',1,1'].

Training Data Drive Link

Skew Augmented Training Data Drive Link

Model and Training

pix-2-pix GAN model is used for learning the mapping of the cloudy image to it's cloud-free counterpart. It follows a supervised conditional vector based training approach. The model is trained on the cloudy and cloud-free image pairs which are dated two days apart. pix-2-pix model follows a pixel to pixel image restoration approach and generates a new pixel cloud-free for every cloudy pixel.

Trained Model Drive Link

Architectures

Discriminator Architecture (PatchGAN)

Generator Architecture (U-Net)

Model Performance and Analyzing results on Skewed Dataset

Various Parameters for evaluation per 100 epochs are as follows in terms of PC,PSNR and SSIM on train and validation data:

Spikes in Skew Dataset Losses

The spikes in the generator loss graph can be due to a few reasons such as the image dataset might have a few big contrasts due to augmenting the dataset and the Generator might have been stuck in a mode collapse(local minima) which when resolved resulted in very high spikes in the loss as the descriminator was quite good trained till then.

Comparison of Result on the two datasets

Correlation

PC on original data	PC on skewed data

SSIM

PSNR

Contribution

Please feel free to raise issues and fix any existing ones. Further details can be found in our code of conduct.

While making a PR, please make sure you:

• [ ] Always start your PR description with "Fixes #issue_number", if you're fixing an issue.
• [ ] Briefly mention the purpose of the PR, along with the tools/libraries you have used. It would be great if you could be version specific.
• [ ] Briefly mention what logic you used to implement the changes/upgrades.
• [ ] Provide in-code review comments on GitHub to highlight specific LOC if deemed necessary.
• [ ] Please provide snapshots if deemed necessary.
• [ ] Update readme if required.

Credits

• This study available as a book chapter. DOI

Please cite this work as:

Maniyar, C., & Kumar, A. (2021). Generative Adversarial Network for Cloud Removal from Optical Temporal Satellite Imagery. In Soft Computing for Problem Solving (pp. 481-491). Springer, Singapore.