Manipulating Attributes of Natural Scenes via Hallucination

Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem

This is the implementation of Manipulating Attributes of Natural Scenes via Hallucination (Accepted for publication in ACM Transactions on Graphics, 2019). See our website for more information about the model!

Bibtex

@ARTICLE{attribute_hallucination,
author={L. {Karacan} and Z. {Akata} and A. {Erdem} and E. {Erdem}},
journal={ACM Transactions on Graphics},
title={Manipulating Attributes of Natural Scenes via Hallucination},
year={2019},
volume={},
number={},
pages={1-1},
month={},}

Requirements

• Linux or OSX
• NVIDIA GPU + CUDA
• We tested our code on python2.7 and python3.6
• You need to install Tkinter module to use interactive scene editing tool.

Dataset

• We arranged dataset from ADE20K and Transient Attributes datasets.
• For our work, we curated a new dataset, which we call ALS18K, using images and annotations from ADE20K and Transient Attributes datasets.
• You can download the ALS18K dataset from this link
• Please refer to the project pages of the aforementioned works for more details.
  • Laffont et al., Transient Attributes for High-Level Understanding and Editing of Outdoor Scenes, SIGGRAPH 2014
  • Zhou et al., Scene Parsing through ADE20K Dataset, CVPR 2017

Coarse Model Training

• We follow similar multiscale strategy with Pix2pixHD model in our code. Some parts of code adapted from Pix2pixHD's official code.
• Download "sceneparsing" models from the link.

python train_coarse.py --img_root ./data/ADE20K_TA_Dataset/  --save_filename ./model/sgn_coarse --scene_parsing_model_path ./sceneparsing/resnet34_dilated8/ --batch_size 16  --num_epoch 100

Enhancer Model Training

python train_enhancer.py --img_root ./data/ADE20K_TA_Dataset/ --coarse_model ./model/sgn_coarse_G_latest  --save_filename ./model/sgn_hd --scene_parsing_model_path ./sceneparsing/resnet34_dilated8/ --batch_size 8 --num_epoch 100 --isEnhancer

Test Coarse Model

• Download pre-trained models from the link.

python test.py --img_root ./data/ADE20K_TA_Dataset/ --model_path ./model/sgn_coarse_G_latest --save_dir ./results

Test Enhancer Model

python test.py --img_root ./data/ADE20K_TA_Dataset/ --model_path ./model/sgn_enhancer_G_latest --save_dir ./resultsHD --isEnhancer

Interactive Scene Editing Demo

• Install Tkinter module.
• Photo style transfer and semantic segmentation codes taken directly from the following works and adapted to editing tool.
  • Zhou et al., Scene Parsing through ADE20K Dataset, CVPR 2017
  • Li et al., A Closed-form Solution to Photorealistic Image Stylization, ECCV 2018
  • Mechrez et al.,Photorealistic Style Transfer with Screened Poisson Equation,BMVC 2017
  • Yoo et al.,Photorealistic Style Transfer via Wavelet Transforms, ICCV 2019
• If you do not want to try the codes to adapt, you can download semantic_segmentation_pytorch, fastphoto, photorealism, WCT2 link
• Place those codes under the editing_tool folder.

cd editing_tool
python main.py --model_path ./pretrained_models/sgn_enhancer_G_latest --isEnhancer --image_size 512