search

ZhangXu0963 / NPC

The code of the paper "Negative Pre-aware for Noisy Cross-modal Matching" in AAAI 2024.
12 stars 2 forks source link

readme

Introduction

This is a PyTorch implementation for the AAAI 2024 paper "Negative Pre-aware for Noisy Cross-Modal Matching". Our method NPC is built on top of the CLIP in PyTorch for end-to-end Image-text Matching.

In Step 1, we calculate the negative impact of each sample via the siamese model A' of base model A.

In Step 2, we train the base model A with the re-weight samples and memory bank.

The proposed NPC achieves much better accuracy (higher R@1) and higher robustness (lower variance among R@1).

Requirements

pip install requirments.txt

Data Preparation

Split Dataset

We conducted experiments on three datasets: MSCOCO, Flickr30K, and CC120K. We followed SCAN to split image-text pairs in MSCOCO and FLickr30K into training, validation and testing sets.

Construct Noisy Datasets

We constructed noise by randomly shuffling some captions of the images.

You can obtain your noisy dataset using construct_noise.py. And we also integrated this part of the code in data.py. The noisy dataset will be automatically constructed and saved during training when it doesn't exist.

Since there are around 3%-20% incorrect annotations existing in the real-world dataset Conceptual Captions, we did not create noisy samples manually.

Frozen Memory Bank

We provide the frozen Memory Bank that appeared in the paper for the datasets with different noise ratios. If you want to update them during training, please use get_memorybank.py.

Note!

If you want to use your own noisy dataset for training, the Memory Bank should also be rebuilt. You can construct the noise dataset by construct_noise.py, and obtain the Memory Bank by get_memorybank.py.

Download Link

The final data directory tree should be:

(Fix a mistake: the path has been defined as dataset/${DATASET_NAME}/annotations/frozen_memory_bank before, and the correct one should be dataset/${DATASET_NAME}/annotations/memory_bank. Thanks for correction！)

├── dataset/
├── ${DATASET_NAME}/
|    ├── annotations/
|    |   ├── memory_bank/
|    |   |   ├── ${noise_ratio}_mbank_img_idx.npy
|    |   |   ├── ${noise_ratio}_mbank_txt_idx.npy
|    |   |   └── ...
|    |   └──scan_split/
|    |       ├── ${noise_ratio}_noise_train_caps.txt #samples use for training. ${noise_ration} is in {0, 0.2, 0.4, 0.6}
|    |       ├── train_caps.txt # the same as `0_noise_train_caps.txt`
|    |       ├── train_ids.txt 
|    |       ├── dev_caps.txt #samples use for validation
|    |       ├── dev_ids.txt 
|    |       ├── test_caps.txt #samples use for testing
|    |       ├── test_ids.txt 
|    |       └── ...
|    └── images/ # all images in MSCOCO (or Flickr30K, CC120K)
└── ...

Models and Evaluation

You can download the models fine-tuned using NPC(ours) and CLIP(our baseline) from this link.

Save the models in folder ./pre-trained_models, and evaluate the models via the following command. For example, evaluate the models trained on MSCOCO with 60% noise.

python main_NPC.py --eval --resume /AAAI24-NPC/pre-trained_models/npc_coco_60.pt --dataset_root /AAAI24-NPC/dataset/MSCOCO --dataset coco
python main_CLIP.py --eval --resume /AAAI24-NPC/pre-trained_models/clip_coco_60.pt --dataset_root /AAAI24-NPC/dataset/MSCOCO --dataset coco