TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures

We tackle the problem of texture synthesis in the setting where many input images are given and a large-scale output is required. We build on recent generative adversarial networks and propose two extensions in this paper. First, we propose an algorithm to combine outputs of GANs trained on a smaller resolution to produce a large-scale plausible texture map with virtually no boundary artifacts. Second, we propose a user interface to enable artistic control. Our quantitative and qualitative results showcase the generation of synthesized high-resolution maps consisting of up to hundreds of megapixels as a case in point.

Video

Watch our video on Youtube:

High Resolution Results

Some of our results can be viewed interactively on EasyZoom:

Code

The TileGAN application consists of two independent processes, the server and the client. Both can be run locally on your machine or you can choose to run the server on a remote location, depending on your hardware setup. All network operations are performed by the server process, which sends the result to the client for displaying.

Download our pre-trained networks

• Download the network(s) to the location of your server (this can be your local machine or a remote server)
  • The Scream (Edvard Munch)
  • Un dimanche après-midi (Georges Seurat)
  • Starry Night (Vincent Van Gogh)
  • Extract the .zip to ./data. There should be a separate folder for each dataset in ./data (e.g. ./data/vangogh) containing a *_network.pkl, a *_descriptors.hdf5, a *_clusters.hdf5 and a *_kmeans.joblib file.

Setup server

• Install requirements from requirements-pip.txt
• Install hnswlib:

  git clone https://github.com/nmslib/hnswlib.git
  cd hnswlib/python_bindings
  python setup.py install

  • Run python tileGAN_server.py
  • The server process will start, then tell you the IP to connect to.

Setup client

• Install Qt for Python. The easiest way to do this is using conda: conda install -c conda-forge qt pyside2
• Install requirements from requirements-pip.txt
• Run python tileGAN_client.py XX.XX.XX.XX (insert the IP from the server). If you're running the server on the same machine as the application, you can omit the IP address or use 'localhost'.

Using your own data/network (optional)

• Train a network on own your data using Progressive Growing of GANs
• run create_dataset path_to_pkl num_latents t_size num_clusters network_name (expected to take between 10 and 60 minutes depending on the specified sample size)
  • path_to_pkl the path to the trained network pickle of your ProGAN network
  • num_latents the size of the database entries (a good number would be 50K to 300K)
  • t_size the size of the output descriptors (an even number somewhere around 12 and 24)
  • num_clusters the number of clusters (approx. 8-16)
  • network_name the name you want to assign your network
• run server and client and load network in UI from the drop down menu. First time the network is loaded, an ANN index is created (expected to take <5mins depending on sample size)

Using our application

Paper

available on arXiv or ACM

Authors

Anna Frühstück, Ibraheem Alhashim, Peter Wonka Contact: anna.fruehstueck (at) kaust.edu.sa

Citation

If you use this code for your research, please cite our paper:

@article{Fruehstueck2019TileGAN,
  title      = {{TileGAN}: Synthesis of Large-Scale Non-Homogeneous Textures},
  author     = {Fr\"{u}hst\"{u}ck, Anna and Alhashim, Ibraheem and Wonka, Peter},
  journal    = {ACM Transactions on Graphics (Proc. SIGGRAPH) },
  issue_date = {July 2019},
  volume     = {38},
  number     = {4},
  pages      = {58:1-58:11},
  year       = {2019}
}

Acknowledgements

Our project is based on ProGAN. We'd like to thank Tero Karras et al. for their great work and for making their code available.