This repository contains source code necessary to reproduce some of the main results in the paper:
Nguyen A, Clune J, Bengio Y, Dosovitskiy A, Yosinski J. (2017). "Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space". Computer Vision and Pattern Recognition.
If you use this software in an academic article, please consider citing:
@inproceedings{nguyen2017plug,
title={Plug \& Play Generative Networks: Conditional Iterative Generation of Images in Latent Space},
author={Nguyen, Anh and Clune, Jeff and Bengio, Yoshua and Dosovitskiy, Alexey and Yosinski, Jason},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
year={2017},
organization={IEEE}
}
For more information regarding the paper, please visit www.evolvingai.org/ppgn
This code is built on top of Caffe. You'll need to install the following:
caffe/python
folder in settings.py is correctYou will need to download a few models to run the examples below. There are download.sh
scripts provided for your convenience.
cd nets/generator/noiseless && ./download.sh
cd nets/caffenet && ./download.sh
cd nets/placesCNN && ./download.sh
cd nets/lrcn && ./download.sh
Settings:
download.sh
scripts run correctly.The main sampling algorithm is in sampler.py. We provide two Python scripts for sampling conditioned on classes and sampling conditioned on captions to which you can pass various command-line arguments to run different experiments. The basic idea is to sample from the joint model p(x,y) which decomposes into a prior p(x) model (given by the G and E) and a condition p(y|x) model. Here, we provide the pre-trained networks for the Noiseless Joint PPGN-h model (Sec 3.5 in the paper). We show examples conditioning on classes, hidden neurons, and captions by using different condition networks.
We provide here 5 different examples as a starting point. Feel free to fork away to produce even cooler results!
1_class_conditional_sampling.sh: Sampling conditioning on the class "junco" (output unit #13 of the CaffeNet DNN trained on ImageNet dataset). This script produces a sampling chain for a single given class.
./1_class_conditional_sampling.sh 13
produces this result:
A sampling chain conditioning on class "junco" starting from a random code (top left)
2_class_conditional_sampling_many.sh: We can also run a long sampling chain between different classes.
./2_class_conditional_sampling_many.sh <epsilon1>
with different epsilon1
(multiplier for the image prior component) produces a chain with different styles of samples:1e-5 | 1e-3 | 1e-1 |
---|---|---|
![]() |
![]() |
![]() |
Default | More abstract style | Ignoring class gradient |
3_hidden_conditional_sampling.sh: Instead of conditioning on a class, it is possible to condition on a hidden neuron i.e. performing Multifaceted Feature Visualization or synthesizing a set of inputs that highly activate a given neuron to understand what features it has learned to detect.
./3_hidden_conditional_sampling.sh 196
produces a set of images for a conv5 neuron #196 previously identified as a "face detector" in DeepVis toolbox:
30 samples generated by conditioning on a "face detector" conv5 neuron. It is interesting that the face detector neuron even fires for things that do not look like a face at all (e.g. the yellow house in the center)
Running the above longer could can produce many other types of faces.
4_hidden_conditional_sampling_placesCNN.sh: One can repeat the example above but with an arbitrary neuron in a different condition network. Here, we visualize the conv5 neuron #182 in the AlexNet DNN trained on MIT Places205 dataset. This neuron has been previously identified as a "food detector" in Zhou et al [2].
./4_hidden_conditional_sampling_placesCNN.sh 182
produces this result:
30 random samples that highly activate a "food detector" conv5 neuron.
5_caption_conditional_sampling.sh: We can also replace the image classifier network in previous examples with a pre-trained image captioning network to form a text-to-image model without even re-training anything. The image captioning model in this example is the LRCN model in Donahue et al (2015) [1].
You would need to use the Caffe provided here and update the path to Caffe accordingly in settings.py
The list of words supported are here
Running ./5_caption_conditional_sampling.sh a_church_steeple_that_has_a_clock_on_it
produces this result:
Note that we often obtain mixed results with this particular text-to-image model. For some words, it works pretty well, but for others it struggles to produce reasonable images. While the language space in this model still needs further exploration, as a starting point, here are some sentences that produce reasonable images.
6_class_conditional_sampling_from_real_image.sh: One can also initialize the sampling from a real image (here, images/volcano.jpg).
./6_class_conditional_sampling_from_real_image.sh 980
produces this result:
7_inpainting.sh: One can also perform "inpainting" i.e. predicting the missing pixels given the observed ones.
./7_inpainting.sh
produces this result:
In each pair, the left is a real image with a random 100x100 patch masked out. The right is the result of PPGNs filling in the patch.
Here are a few (crazy?) ideas that one could play with PPGNs:
Learning What and Where to Draw
Reed et al. (2016), by conditioning on a region of the last heatmap of a fully convolutional classification network or a semantic segmentation network.Note that the code in this repository is licensed under MIT License, but, the pre-trained condition models used by the code have their own licenses. Please carefully check them before use.
If you have questions/suggestions, please feel free to email, tweet to @anh_ng8 or create github issues.
[1] Donahue et al. "Long-term Recurrent Convolutional Networks for Visual Recognition and Description". CVPR 2015
[2] Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. "Object detectors emerge in deep scene cnns". ICLR 2015.