Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval

PyTorch | Arxiv | Project

PyTorch implementation of our D2S model for zero-shot sketch-based image retrieval:
Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval
Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados and Yi-Zhe Song
CVPR, 2019

Retrieval Results

Sketchy

TU-Berlin

QuickDraw

Prerequisites

• Linux (tested on Ubuntu 16.04)
• NVIDIA GPU + CUDA CuDNN
• 7z

  sudo apt-get install p7zip-full

  Getting Started

  Introduction

  We took the first steps to move towards practical zero shot sketch based image retrieval systems (see the paper for more detail). To this end, we have used Quick Draw! to curate the sketches and for the images we would like to thanks Flickr API for such an amazing API.

The structure of this repo is as follows:

1. Installation
2. Getting the data
3. How to train models
4. At last how to test and evaluate

Installation

• Clone this repository

  git clone https://github.com/sounakdey/doodle2search.git
  cd doodle2search

• Install the requirements (not checked)

  pip3 install -r requirements.txt

  Download datasets

• Sketchy
• TU-Berlin

  bash download_datasets.sh

• QuickDraw-Extended sketches and images

Train

Finally we are ready to train. Magical words are:

python3 src/train.py sketchy_extended --data_path <mention the data path of the dataset>

The first argument is the dataset name, which you can replace it with tuberlin_extend or quickdraw_extend. You can check the options.py for changing a lot of the options such dimension size, different models, hyperparameters, etc.

Test

Sketchy

python3 src/test.py sketchy_extend --data_path <mention the data path of the dataset> --load <path of the trained models>

Citation

@InProceedings{Dey_2019_CVPR,
    author = {Dey, Sounak and Riba, Pau and Dutta, Anjan and Llados, Josep and Song, Yi-Zhe},
    title = {Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}

Conclusion

Thank you and sorry for the bugs!

Author

• Sounak Dey (@SounakDey)
• Pau Riba (@PauRiba)
• Anjan Dutta (@AnjanDutta)