Semantic-Guided-Low-Light-Image-Enhancement
This is the official Pytorch implementation for our paper "Semantic-Guided Zero-Shot Learning for Low-Light Image/Video Enhancement"
Updates
- 2021.11.27: The figure for the model architecture has been uploaded.
- 2021.11.10: This work is accepted and will be presented at WACV 2022.
- 2021.10.8: An easy way to calculate mPA and mIOU score is available at this repository
- 2021.10.6: A sample enhanced low-light video in GIF format has been released.
- 2021.10.5: A sample low-light video and its enhanced result is available at: [Low-Light] [Enhanced]
- 2021.10.5: My new repository contains a collection of low-light enhancement methods. Hope you will find it helpful.
- 2021.10.4: The arxiv link is available at http://arxiv.org/abs/2110.00970
Abstract
Low-light images challenge both human perceptions and computer vision algorithms. It is crucial to make algorithms robust to enlighten low-light images for computational photography and computer vision applications such as real-time detection and segmentation tasks. This paper proposes a semantic-guided zero-shot low-light enhancement network which is trained in the absence of paired images, unpaired datasets, and segmentation annotation. Firstly, we design an efficient enhancement factor extraction network using depthwise separable convolution. Secondly, we propose a recurrent image enhancement network for progressively enhancing the low-light image. Finally, we introduce an unsupervised semantic segmentation network for preserving the semantic information. Extensive experiments on various benchmark datasets and a low-light video demonstrate that our model outperforms the previous state-of-the-art qualitatively and quantitatively. We further discuss the benefits of the proposed method for low-light detection and segmentation.
Model Architecture
Click the following link to see the model architecture in pdf format.
Sample Results
1. Low-Light Video Frames
From left to right, and from top to bottom: Dark, Retinex [1], KinD [2], EnlightenGAN [3], Zero-DCE [4], Ours.
## 2. Low-Light Images (Real-World)
From left to right, and from top to bottom: Dark, PIE [5], LIME [6], Retinex [1], MBLLEN [7], KinD [2] , Zero-DCE [4], Ours
# Get Started
## 1. Requirements
* CUDA 10.0
* Python 3.6+
* Pytorch 1.0+
* torchvision 0.4+
* opencv-python
* numpy
* pillow
* scikit-image
## 2. Prepare Datasets
### Testing Dataset
- Downloaded the official testing images from [BaiduYun](https://pan.baidu.com/s/1pbz0YlOmLqxnazeKrRRUQg) with password `8itq`.
- Move the unzipped file into `data/test_data/`.
- You can also put custom datasets or your own low-light images in this folder for testing. Example: `data/test_data/yourDataset/`
### Training Dataset
- Downloaded the official training images from [BaiduYun](https://pan.baidu.com/s/19ez3dM079WksPRB0Xw98kg) with password `n93t`.
- Move the unzipped file into `data/train_data/`.
NOTE: If you don't have BaiduYun account, you can download both the training and the testing dataset via [Google Drive](https://drive.google.com/drive/folders/1RIQsP5ap5QU7LstHPknOffQZeqht_FCh)
## 3. Training
For model training, run in terminal:
```
python train.py --snapshots_folder weight_folder --pretrain_dir pretrained_weight_file
```
For example, if your folder to save weights are `weight`, and your most recent weight file are at`weight/Epoch99.pth`, run in terminal:
```
python train.py --snapshots_folder weight/ --pretrain_dir weight/Epoch99.pth
```
## 4. Testing
**NOTE: Please delete all readme.txt in the `data` folder to avoid model inference error.**
For model testing, run in terminal
```
python test.py --input_dir your_input_folder --weight_dir pretrained_weight_file --test_dir your_output_folder
```
For example, if your input folder is named `test_dataset`, pretrained weight is at `weight/Epoch99.pth`, and your result image will be saved in `test_output`, you can run in terminal:
```
python test.py --input_dir test_dataset --weight_dir weight/Epoch99.pth --test_dir test_output
```
## 5. Testing on Videos
For model testing on videos (MP4 format), run in terminal:
```
bash test_video.sh
```
There are five hyperparameters in `demo/make_video.py`for video testing. See the following explanation.
- `video_path`: path of the low-light video
- `image_lowlight_folder`: path of the low-light images
- `image_folder`: path of the enhanced images
- `save_path`: path of the enhanced video
- `choice`: whether converting video to image, or image to video
# Hyperparameters
| Name | Type | Default |
|----------------------|-------|--------------------|
| lowlight_images_path | str | data/train_data/ |
| lr | float | 1e-3 |
| weight_decay | float | 1e-3 |
| grad_clip_norm | float | 0.1 |
| num_epochs | int | 100 |
| train_batch_size | int | 6 |
| val_batch_size | int | 8 |
| num_workers | int | 4 |
| display_iter | int | 10 |
| snapshot_iter | int | 10 |
| scale_factor | int | 1 |
| snapshots_folder | str | weight/ |
| load_pretrain | bool | False |
| pretrain_dir | str | weight/Epoch99.pth |
| num_of_SegClass | int | 21 |
| conv_type | str | dsc |
| patch_size | int | 4 |
| exp_level | float | 0.6 |
# TODO List
- [x] List (important) hyperparameters
- [x] Addres model input size issue
- [x] Upload Pretrained Weight
- [x] Rewrite training and testing argparse in a option.py
- [x] Rewrite training as a class
- [x] Rewrite testing as a class
- [x] Upload Testing Dataset
- [x] Upload Arxiv Link
- [x] Testing on Video
- [x] Upload BibTeX
- [x] Modify Readme file
- [x] Upload Model Architecture
- [ ] Provide Online Demo
# Others
Please reach zhengsh@kean.edu if you have any questions. This repository is heavily based upon [Zero-DCE](https://github.com/Li-Chongyi/Zero-DCE). Thanks for sharing the code!
# Citations
Please cite the following paper if you find this repository helpful.
```
@inproceedings{zheng2022semantic,
title={Semantic-guided zero-shot learning for low-light image/video enhancement},
author={Zheng, Shen and Gupta, Gaurav},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages={581--590},
year={2022}
}
```
# References
[1] Wei, Chen, et al. "Deep retinex decomposition for low-light enhancement." arXiv preprint arXiv:1808.04560 (2018).
[2] Zhang, Yonghua, Jiawan Zhang, and Xiaojie Guo. "Kindling the darkness: A practical low-light image enhancer." Proceedings of the 27th ACM international conference on multimedia. 2019.
[3] Jiang, Yifan, et al. "Enlightengan: Deep light enhancement without paired supervision." IEEE Transactions on Image Processing 30 (2021): 2340-2349.
[4] Guo, Chunle, et al. "Zero-reference deep curve estimation for low-light image enhancement." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[5] Fu, Xueyang, et al. "A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation." IEEE Transactions on Image Processing 24.12 (2015): 4965-4977.
[6] Guo, Xiaojie, Yu Li, and Haibin Ling. "LIME: Low-light image enhancement via illumination map estimation." IEEE Transactions on image processing 26.2 (2016): 982-993.
[7] Lv, Feifan, et al. "MBLLEN: Low-Light Image/Video Enhancement Using CNNs." BMVC. 2018.