deep-photo-styletransfer-tf

This is a pure Tensorflow implementation of Deep Photo Styletransfer, the torch implementation could be found here

This implementation support L-BFGS-B (which is what the original authors used) and Adam in case the ScipyOptimizerInterface incompatible when Tensorflow upgrades to higher version.

This implementation may seem to be a little bit simpler thanks to Tensorflow's automatic differentiation

Additionally, there is no dependency on MATLAB thanks to another repository computing Matting Laplacian Sparse Matrix. Below is example of transferring the photo style to another photograph.

Disclaimer

This software is published for academic and non-commercial use only.

Setup

Dependencies

Tensorflow
Numpy
Pillow
Scipy
PyCUDA (used in smooth local affine, tested on CUDA 8.0)

It is recommended to use Anaconda Python, since you only need to install Tensorflow and PyCUDA manually to setup. The CUDA is optional but really recommended

Download the VGG-19 model weights

The VGG-19 model of tensorflow is adopted from VGG Tensorflow with few modifications on the class interface. The VGG-19 model weights is stored as .npy file and could be download from Google Drive or BaiduYun Pan. After downloading, copy the weight file to the ./project/vgg19 directory

Usage

Basic Usage

You need to specify the path of content image, style image, content image segmentation, style image segmentation and then run the command

python deep_photostyle.py --content_image_path <path_to_content_image> --style_image_path <path_to_style_image> --content_seg_path <path_to_content_segmentation> --style_seg_path <path_to_style_segmentation> --style_option 2

Example:

python deep_photostyle.py --content_image_path ./examples/input/in11.png --style_image_path ./examples/style/tar11.png --content_seg_path ./examples/segmentation/in11.png --style_seg_path ./examples/segmentation/tar11.png --style_option 2

Other Options

--style_option specifies three different ways of style transferring. --style_option 0 is to generate segmented intermediate result like torch file neuralstyle_seg.lua in torch. --style_option 1 uses this intermediate result to generate final result like torch file deepmatting_seg.lua. --style_option 2 combines these two steps as a one line command to generate the final result directly.

--content_weight specifies the weight of the content loss (default=5), --style_weight specifies the weight of the style loss (default=100), --tv_weight specifies the weight of variational loss (default=1e-3) and --affine_weight specifies the weight of affine loss (default=1e4). You can change the values of these weight and play with them to create different photos.

--serial specifies the folder that you want to store the temporary result out_iter_XXX.png. The default value of it is ./. You can simply mkdir result and set --serial ./result to store them. Again, the temporary results are simply clipping the image into [0, 255] without smoothing. Since for now, the smoothing operations need pycuda and pycuda will have conflict with tensorflow when using single GPU

Run python deep_photostyle.py --help to see a list of all options

Image Segmentation

This repository doesn't offer image segmentation script and simply use the segmentation image from the torch version. The mask colors used are also the same as them. You could specify your own segmentation model and mask color to customize your own style transfer.

Examples

Here are more results from tensorflow algorithm (from left to right are input, style, torch results and tensorflow results)

Acknowledgement

This work was done when Yang Liu was a research intern at Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, under the supervision of Prof. Mingli Song and Yongcheng Jing.
Our tensorflow implementation basically follows the torch code.
We use martinbenson's python code to compute Matting Laplacian.

Citation

If you find this code useful for your research, please cite:

@misc{YangPhotoStyle2017,
  author = {Yang Liu},
  title = {deep-photo-style-transfer-tf},
  publisher = {GitHub},
  organization={Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies},
  year = {2017},
  howpublished = {\url{https://github.com/LouieYang/deep-photo-styletransfer-tf}}
}

Contact

Feel free to contact me if there is any question (Yang Liu lyng_95@zju.edu.cn).

LouieYang / deep-photo-styletransfer-tf

readme