search

Evolving-AI-Lab / ppgn

Code for paper "Plug and Play Generative Networks"
MIT License
540 stars 102 forks source link

readme

Plug and Play Generative Networks

This repository contains source code necessary to reproduce some of the main results in the paper:

Nguyen A, Clune J, Bengio Y, Dosovitskiy A, Yosinski J. (2017). "Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space". Computer Vision and Pattern Recognition.

If you use this software in an academic article, please consider citing:

@inproceedings{nguyen2017plug,
  title={Plug \& Play Generative Networks: Conditional Iterative Generation of Images in Latent Space},
  author={Nguyen, Anh and Clune, Jeff and Bengio, Yoshua and Dosovitskiy, Alexey and Yosinski, Jason},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2017},
  organization={IEEE}
}

For more information regarding the paper, please visit www.evolvingai.org/ppgn

1. Setup

Installing software

This code is built on top of Caffe. You'll need to install the following:

Downloading models

You will need to download a few models to run the examples below. There are download.sh scripts provided for your convenience.

Settings:

2. Usage

The main sampling algorithm is in sampler.py. We provide two Python scripts for sampling conditioned on classes and sampling conditioned on captions to which you can pass various command-line arguments to run different experiments. The basic idea is to sample from the joint model p(x,y) which decomposes into a prior p(x) model (given by the G and E) and a condition p(y|x) model. Here, we provide the pre-trained networks for the Noiseless Joint PPGN-h model (Sec 3.5 in the paper). We show examples conditioning on classes, hidden neurons, and captions by using different condition networks.

Examples

We provide here 5 different examples as a starting point. Feel free to fork away to produce even cooler results!

1_class_conditional_sampling.sh: Sampling conditioning on the class "junco" (output unit #13 of the CaffeNet DNN trained on ImageNet dataset). This script produces a sampling chain for a single given class.

A sampling chain conditioning on class "junco" starting from a random code (top left)

2_class_conditional_sampling_many.sh: We can also run a long sampling chain between different classes.

1e-5 1e-3 1e-1
Default More abstract style Ignoring class gradient

3_hidden_conditional_sampling.sh: Instead of conditioning on a class, it is possible to condition on a hidden neuron i.e. performing Multifaceted Feature Visualization or synthesizing a set of inputs that highly activate a given neuron to understand what features it has learned to detect.

30 samples generated by conditioning on a "face detector" conv5 neuron. It is interesting that the face detector neuron even fires for things that do not look like a face at all (e.g. the yellow house in the center)

Running the above longer could can produce many other types of faces.

4_hidden_conditional_sampling_placesCNN.sh: One can repeat the example above but with an arbitrary neuron in a different condition network. Here, we visualize the conv5 neuron #182 in the AlexNet DNN trained on MIT Places205 dataset. This neuron has been previously identified as a "food detector" in Zhou et al [2].

30 random samples that highly activate a "food detector" conv5 neuron.

5_caption_conditional_sampling.sh: We can also replace the image classifier network in previous examples with a pre-trained image captioning network to form a text-to-image model without even re-training anything. The image captioning model in this example is the LRCN model in Donahue et al (2015) [1].

Note that we often obtain mixed results with this particular text-to-image model. For some words, it works pretty well, but for others it struggles to produce reasonable images. While the language space in this model still needs further exploration, as a starting point, here are some sentences that produce reasonable images.

6_class_conditional_sampling_from_real_image.sh: One can also initialize the sampling from a real image (here, images/volcano.jpg).

7_inpainting.sh: One can also perform "inpainting" i.e. predicting the missing pixels given the observed ones.

In each pair, the left is a real image with a random 100x100 patch masked out. The right is the result of PPGNs filling in the patch.

Using your own condition models

3. Ideas

Here are a few (crazy?) ideas that one could play with PPGNs:

  1. One can generate objects in a specific region of the image, similar to Learning What and Where to Draw Reed et al. (2016), by conditioning on a region of the last heatmap of a fully convolutional classification network or a semantic segmentation network.
  2. Plugin a better image captioning model e.g. the Show and Tell
  3. Synthesize a Music Video by sampling images conditioned on lyrics.
  4. There are more and crazier ideas to do with PPGNs, feel free to reach out if you want to chat.

4. Licenses

Note that the code in this repository is licensed under MIT License, but, the pre-trained condition models used by the code have their own licenses. Please carefully check them before use.

5. Questions?

If you have questions/suggestions, please feel free to email, tweet to @anh_ng8 or create github issues.

References

[1] Donahue et al. "Long-term Recurrent Convolutional Networks for Visual Recognition and Description". CVPR 2015

[2] Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. "Object detectors emerge in deep scene cnns". ICLR 2015.