Interact, Embed, and EnlargE (IEEE): Boosting Modality-specific Representations for Multi-Modal Person Re-identification

We provide the codes for reproducing result of our paper Interact, Embed, and EnlargE (IEEE): Boosting Modality-specific Representations for Multi-Modal Person Re-identification.

Installation

Basic environments: python3.6, pytorch1.8.0, cuda11.1.
Our codes structure is based on Torchreid. (More details can be found in link: https://github.com/KaiyangZhou/deep-person-reid , you can download the packages according to Torchreid requirements.)

# create environment
cd AAAI2022_IEEE/
conda create --name ieeeReid python=3.6
conda activate ieeeReid

# install dependencies
# make sure `which python` and `which pip` point to the correct path
pip install -r requirements.txt

# install torch and torchvision (select the proper cuda version to suit your machine)
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge

# install torchreid (don't need to re-build it if you modify the source code)
python setup.py develop

Get start

You can use the setting in im_r50_softmax_256x128_amsgrad_RGBNT_ieee_part_margin.yaml to get the results of full IEEE.
```
python ./scripts/mainMultiModal.py --config-file ./configs/RGBNT_ieee_part_margin.yaml --seed 40
```

Details

The details of our Cross-modal Interacting Module (CIM) and Relation-based Embedding Module (REM) can be found in .\torchreid\models\ieee3modalPart.py. The design of Multi-modal Margin Loss(3M loss) can be found in .\torchreid\losses\multi_modal_margin_loss_new.py.

Ablation study settings.

You can control these two modules and the loss by change the corresponding codes.

1) Cross-modal Interacting Module (CIM) and Relation-based Embedding Module (REM)

# change the code in .\torchreid\models\ieee3modalPart.py

class IEEE3modalPart(nn.Module):
   def __init__(···
   ):
       modal_number = 3
       fc_dims = [128]
       pooling_dims = 768
       super(IEEE3modalPart, self).__init__()
       self.loss = loss
       self.parts = 6

       self.backbone = nn.ModuleList(···
       )

      # using Cross-modal Interacting Module (CIM)
       self.interaction = True
       # using channel attention in CIM
       self.attention = True

       # using Relation-based Embedding Module (REM)
       self.using_REM = True

       ···

2) Multi-modal Margin Loss(3M loss)

# change the code in .\configs\your_config_file.yaml

# using Multi-modal Margin Loss(3M loss), you can change the margin by modify the parameter of "ieee_margin".
···
loss:
 name: 'margin'
 softmax:
   label_smooth: True
 ieee_margin: 1
 weight_m: 1.0
 weight_x: 1.0
···

# using only CE loss
···
loss:
 name: 'softmax'
 softmax:
   label_smooth: True
 weight_x: 1.0
···

Dataset

RGBNT201 & reconstructed cross-modal datasets & Market1501 multi-modal version:

Google Drive Link: https://drive.google.com/drive/folders/1EscBadX-wMAT56_It5lXY-S3-b5nK1wH?usp=sharing Please contact with Zi Wang (email address: ziwang1121@foxmail.com).

ziwang1121 / IEEE

readme