Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention

Overview

This repository contains the implementation of Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention.

In this work, we develop a zero-shot fine-grained recognition with the ability to localize attributes using a dense attribute-based attention and embedding mechanism.

Prerequisites

To install all the dependency packages, please run:

pip install -r requirements.txt

Data Preparation

1) Please download and extract information into the ./data folder. We include details about download links as well as what are they used for in each folder within ./data folder.

2) [Optional] For DeepFashion dataset, we partition seen/unseen classes and training/testing split via:

python ./extract_feature/extract_annotation_DeepFashion.py                          #create ./data/DeepFashion/annotation.pkl

We have included the result file by default in the repository. Similarly, we have also included the attribute semantics from GloVe model for all datasets which are computed by:

python ./extract_feature/extract_attribute_w2v_DeepFashion.py                               #create ./w2v/DeepFashion_attribute.pkl
python ./extract_feature/extract_attribute_w2v_AWA2.py                              #create ./w2v/AWA2_attribute.pkl
python ./extract_feature/extract_attribute_w2v_CUB.py                               #create ./w2v/CUB_attribute.pkl
python ./extract_feature/extract_attribute_w2v_SUN.py                               #create ./w2v/SUN_attribute.pkl

3) Please run feature extraction scripts in ./extract_feature folder to extract features from the last convolution layers of ResNet as region features for attention mechanism:

python ./extract_feature/extract_feature_map_ResNet_101_DeepFashion.py                      #create ./data/DeepFashion/feature_map_ResNet_101_DeepFashion_sep_seen_samples.hdf5
python ./extract_feature/extract_feature_map_ResNet_101_AWA2.py                     #create ./data/AWA2/feature_map_ResNet_101_AWA2.hdf5
python ./extract_feature/extract_feature_map_ResNet_101_CUB.py                      #create ./data/CUB/feature_map_ResNet_101_CUB.hdf5
python ./extract_feature/extract_feature_map_ResNet_101_SUN.py                      #create ./data/SUN/feature_map_ResNet_101_SUN.hdf5

These scripts create hdf5 files which contain image features and data splits for training and evaluation.

Training and Evaluation

1) We provide separate jupyter notebooks for training and evaluation on all four datasets in ./notebook folder:

./notebook/DAZLE_DeepFashion.ipynb
./notebook/DAZLE_AWA2.ipynb
./notebook/DAZLE_CUB.ipynb
./notebook/DAZLE_SUN.ipynb

Pretrained Models

Since the training process is not resource-intensive, most experiments can be produced within 30mins.

• If you need the pretrained models, please reach out to me via huynh.dat@northeastern.edu

Citation

If this code is helpful for your research, we would appreciate if you cite the work:

@article{Huynh-DAZLE:CVPR20,
  author = {D.~Huynh and E.~Elhamifar},
  title = {Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention},
  journal = {{IEEE} Conference on Computer Vision and Pattern Recognition},
  year = {2020}}

References

We adapt our dataloader classes from the following project: https://github.com/edgarschnfld/CADA-VAE-PyTorch