SPN4CIR: Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives (ACM MM 2024)

Table of Contents

• Overview
• Requirements
• Checkpoints and Datasets
• Pre-Process
• CLIP4CIR
• TGCIR
• BLIP4CIR
• BLIP24CIR
• ZSCIR
• Citation
• Acknowledgement

Overview

The Composed Image Retrieval (CIR) task aims to retrieve target images using a composed query consisting of a reference image and a modified text. Advanced methods often utilize contrastive learning as the optimization objective, which benefits from adequate positive and negative examples. However, the triplet for CIR incurs high manual annotation costs, resulting in limited positive examples. Furthermore, existing methods commonly use in-batch negative sampling, which reduces the negative number available for the model. To address the problem of lack of positives, we propose a data generation method by leveraging a multi-modal large language model to construct triplets for CIR. To introduce more negatives during fine-tuning, we design a two-stage fine-tuning framework for CIR, whose second stage introduces plenty of static representations of negatives to optimize the representation space rapidly. The above two improvements can be effectively stacked and designed to be plug-and-play, easily applied to existing CIR models without changing their original architectures. Extensive experiments and ablation analysis demonstrate that our method effectively scales positives and negatives and achieves state-of-the-art results on both FashionIQ and CIRR datasets. In addition, our methods also perform well in zero-shot composed image retrieval, providing a new CIR solution for the low-resources scenario.

Checkpoints and Datasets

You can find all checkpoints and generated data at this huggingface repo: https://huggingface.co/BUAADreamer/SPN4CIR

For ease of reproduction, we have also uploaded the original FashionIQ and CIRR datasets (including images) to this huggingface repo: https://huggingface.co/datasets/BUAADreamer/cir_dataset

The checkpoints of the first stage model are taken from the repo of the original paper.

Requirements

• Prepare python 3.8.13 cuda 12.2 environment
• Install python3 environment

pip3 install -r requirements.txt

• Download FashionIQ and CIRR datasets from corresponding websites and merge them with data we provided like the following structure:

project_base_path
└───  tgcir
      | train.py
      | ...

└───  clip4cir
      | train.py
      | ...

└───  blip4cir
      | train.py
      | ...

└───  blip24cir
      | train.py
      | ...

└───  zscir
      | ...

└───  data # ckpts of the first stage
      └─── tgcir
      └─── clip4cir
      └─── blip4cir
      └─── blip24cir

└───  mm_data # generated caption data
      └─── fiq
      └─── cirr
      └─── zs

└───  checkpoints # ckpts of the second stage
      └─── fiq_clip
      └─── cirr_clip
      └─── fiq_blip
      └─── cirr_blip
      └─── fiq_blip2
      └─── cirr_blip2
      └─── fiq_tgcir
      └─── cirr_tgcir

└───  fashionIQ_dataset
      └─── captions
            | cap.dress.test.json
            | cap.dress.train.json
            | cap.dress.val.json
            | cap.extend_*.train.json
            | ...

      └───  images
            | B00006M009.jpg
            | ...

      └─── image_splits
            | split.dress.test.json
            | split.dress.train.json
            | split.dress.val.json
            | ...

      | optimized_images.json

└───  cirr_dataset  
       └─── train
            └─── 0
                | train-10108-0-img0.png
                | ...
            ...

       └─── dev
            | dev-0-0-img0.png
            | ...

       └─── test1
            | test1-0-0-img0.png
            | ...

       └─── cirr
            └─── captions
                | cap.rc2.test1.json
                | cap.rc2.train.json
                | cap.rc2.val.json
                | cap.rc2.train.extend_*.json

            └─── image_splits
                | split.rc2.test1.json
                | split.rc2.train.json
                | split.rc2.val.json

      | optimized_images.json

Pre-Process

CLIP4CIR

TGCIR

BLIP4CIR

BLIP24CIR

ZSCIR

Citation

@article{feng2024improving,
  title={Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives},
  author={Feng, Zhangchi and Zhang, Richong and Nie, Zhijie},
  journal={arXiv preprint arXiv:2404.11317},
  year={2024}
}

Acknowledgement

About code, our project is based on CLIP4Cir. Some of our code are learned from TG-CIR, SPRC, Candidate-Reranking-CIR.

About data, we train and evaluate on two CIR dataset FashionIQ and CIRR. We use LLaVA to do caption generation and Unicom to do image pair match.

Thanks for their great jobs! If you need to use a particular part of our code, please cite the relevant papers.