the data structure and reproducing the experiments and results

[jpainam]

Hi, in your readme, you show how the directory should like that. But their implementation in data/datasets/images are different. For example, you asked to create Occluded_Duke folder, but you are referring P-DukeMTMC-reid in this code. Samed applied for occluded_reID.

https://github.com/hh23333/PVPM/blob/1bcdb5d85ef46283c0893b3e60aeef318d512233/torchreid/data/datasets/image/pduke_reid.py#L20

Look like, they must be all be under Market-1501-v15.09.15. I understand this can be easily fixed for anyone who want to reproduce your result. But it can be misleading. Just pointing that out my help.

Here my final directory structure, for any interested

PVPM_experiments/data/Market-1501-v15.09.15
├── ICME2018_Occluded-Person-Reidentification_datasets
│   ├── P-DukeMTMC_reid
│   └── Occluded_REID
├── bounding_box_train
└── Partial-REID_Dataset

[chenxiangzZ]

i also find this problem, anyway, do you reproduce this work? i have a difference in the result,shall we discuss?

[jpainam]

You are right, there is a big gap between the results reported in the paper and the results i get when trained on two partial or occluded persion reid datasets. these are the results i get

=> Final test
##### Evaluating occlusion_reid (target) #####
Extracting features from query set ...
Done, obtained 1000-by-2048 matrix
Extracting features from gallery set ...
Done, obtained 1000-by-2048 matrix
Speed: 0.2468 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 53.4%
CMC curve
Rank-1  : 61.1%
Rank-3  : 70.6%
Rank-5  : 76.0%
Rank-10 : 82.8%
Rank-20 : 87.8%
##### Evaluating partial_reid (target) #####
Extracting features from query set ...
Done, obtained 300-by-2048 matrix
Extracting features from gallery set ...
Done, obtained 300-by-2048 matrix
Speed: 0.2373 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 64.7%
CMC curve
Rank-1  : 67.0%
Rank-3  : 74.3%
Rank-5  : 83.0%
Rank-10 : 89.3%
Rank-20 : 94.3%
Checkpoint saved to "savedir/model.pth.tar-30"
Elapsed 1:29:34

what did you get?

[chenxiangzZ]

i think i have the same result like you, there are results i get on two datasets:

Evaluating partial_reid (target)

Done, obtained 300-by-2048 matrix Speed: 0.1105 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 65.3% CMC curve Rank-1 : 67.0% Rank-3 : 77.3% Rank-5 : 84.0% Rank-10 : 90.7% Rank-20 : 93.7% Checkpoint saved to "../PVPM/model.pth.tar-20"

Evaluating Occluded reid(target)

Results mAP: 54.2% CMC curve Rank-1 : 61.2% Rank-3 : 72.9% Rank-5 : 77.3% Rank-10 : 85.3% Rank-20 : 90.2%

is something ignored in the command? i really hope the auther give us the solution!

[chenxiangzZ]

哈哈，都是中国人，要不加个QQ更好沟通，417545906

[hh23333]

@jpainam @chenxiangzZ Hi, About the data structure, I set root=PVPM_experiments/data, and there was no error report when I reproduced the experiment. For the experiment result, I reproduced it on my own machine, with the following parameters

--load-pose --root /media/hh/disc_d/datasets -s market1501 -t occlusion_reid partial_reid --save-dir /media/hh/disc_e/deep_person_reid_experiment/market_pose_steptrain_graphmatching_newatt_reproduce -a pose_p6s --gpu-devices 0 --fixbase-epoch 30 --open-layers pose_subnet --new-layers pose_subnet --transforms random_flip --optim sgd --lr 0.02 --stepsize 15 25 --staged-lr --height 384 --width 128 --batch-size 32 --start-eval 5 --eval-freq 5 --load-weights /media/hh/disc_e/deep_person_reid_experiment/market_PCB_sgd_384_losssum_EA3/model.pth.tar-60 --train-sampler RandomIdentitySampler --reg-matching-score-epoch 0 --graph-matching --max-epoch 30 --part-score

and I got a similiar result with the paper report:

Evaluating occlusion_reid (target)

Extracting features from query set ... Done, obtained 1000-by-2048 matrix Extracting features from gallery set ... Done, obtained 1000-by-2048 matrix Speed: 0.0989 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 60.0% CMC curve Rank-1 : 66.5% Rank-3 : 77.8% Rank-5 : 82.7% Rank-10 : 88.8% Rank-20 : 93.4%

Evaluating partial_reid (target)

Extracting features from query set ... Done, obtained 300-by-2048 matrix Extracting features from gallery set ... Done, obtained 300-by-2048 matrix Speed: 0.0948 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 71.1% CMC curve Rank-1 : 76.3% Rank-3 : 83.7% Rank-5 : 86.0% Rank-10 : 93.0% Rank-20 : 95.7% Checkpoint saved to "/media/hh/disc_e/PVPM_reproduce/model.pth.tar-20"

There are some things I need to know: 1.What is the result of PCB baseline on your own device. 2.Did you use the same environment I mentioned in README 3.How did you get the pose heatmaps

[jpainam]

@hh23333 The PCB model you shared didn't work. So, i had to train my own PCB but using your code.

1. The result i get with the PCB baseline on my device

   ##### Evaluating market1501 (source) #####
   Extracting features from query set ...
   Done, obtained 3368-by-12288 matrix
   Extracting features from gallery set ...
   Done, obtained 15913-by-12288 matrix
   Speed: 0.0270 sec/batch
   Computing distance matrix with metric=euclidean ...
   Computing CMC and mAP ...
   ** Results **
   mAP: 78.1%
   CMC curve
   Rank-1  : 92.8%
   Rank-3  : 96.3%
   Rank-5  : 97.2%
   Rank-10 : 98.1%
   Rank-20 : 98.8%
   Checkpoint saved to "market_PCB/model.pth.tar-60"
   Elapsed 7:36:13

2. My environment is different, i used

   Python version 3.7
   Pytorch version 1.5.1
   CUDA 10.1

3. I used the pose heatmaps you shared in the readme. I didn't produce my own pose heatmaps

These are my parameters too

--load-pose --root /home/fstu1/datasets/market1501 
-s market1501 -t occlusion_reid partial_reid --save-dir savedir 
-a pose_p6s --gpu-devices 0 --fixbase-epoch 30 
--open-layers pose_subnet --new-layers pose_subnet 
--transforms random_flip 
--optim sgd --lr 0.02 --stepsize 15 25 --staged-lr 
--height 384 --width 128 --batch-size 32 --start-eval 20 
--eval-freq 10 --load-weights market_PCB/model.pth.tar-60 
--train-sampler RandomIdentitySampler --reg-matching-score-epoch 0 
--graph-matching --max-epoch 30 --part-score

I'll try to run using your own environment with Python 2.7 and see. But do you thing python and pytorch versions can affect the training?

Thank you for your reply

[hh23333]

@jpainam The only reason I can think of is that the versions of python are different, which would make some operation get different results, for example, 2/3 get 0 in python and 0.66666... in python3. But, I am also surprised that using different versions will have such a big impact on the results. Looking forward to your experiment results in python2.

[chenxiangzZ]

@jpainam The only reason I can think of is that the versions of python are different, which would make some operation get different results, for example, 2/3 get 0 in python and 0.66666... in python3. But, I am also surprised that using different versions will have such a big impact on the results. Looking forward to your experiment results in python2.

@hh23333 @jpainam Thanks for your reply！！！ sadly,I reproduce the code with python2.7 and pytorch1.0,but results seems not changed, results as follows:

1. pretrain PCB:

python scripts/main.py --root ../PVPM_experiments/data \
 -s market1501 -t market1501\
 --save-dir ../market_PCB\
 -a pcb_p6 --gpu-devices 0 --fixbase-epoch 0\
 --open-layers classifier fc\
 --new-layers classifier em\
 --transforms random_flip\
 --optim sgd --lr 0.02\
 --stepsize 25 50\
 --staged-lr --height 384 --width 192\
 --batch-size 32 --base-lr-mult 0.5

##### Evaluating market1501 (source) #####
Extracting features from query set ...
Done, obtained 3368-by-12288 matrix
Extracting features from gallery set ...
Done, obtained 15913-by-12288 matrix
Speed: 0.0259 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 77.5%
CMC curve
Rank-1  : 92.7%
Rank-3  : 96.5%
Rank-5  : 97.3%
Rank-10 : 98.3%
Rank-20 : 98.8%
Checkpoint saved to "../market_PCB/model.pth.tar-60"

2. PVPM trian：

   python scripts/main.py --load-pose --root ../PVPM_experiments/data \
   -s market1501\
   -t partial_reid occlusion_reid p_duke\
   --save-dir ../PVPM\
   -a pose_p6s --gpu-devices 0\
   --fixbase-epoch 30\
   --open-layers pose_subnet\
   --new-layers pose_subnet\
   --transforms random_flip\
   --optim sgd --lr 0.02\
   --stepsize 15 25 --staged-lr\
   --height 384 --width 128\
   --batch-size 32\
   --start-eval 5\
   --eval-freq 5\
   --load-weights ../market_PCB/model.pth.tar-60\
   --train-sampler RandomIdentitySampler\
   --reg-matching-score-epoch 0\
   --graph-matching \
   --max-epoch 30 \
   --part-score

=> Final test
##### Evaluating occlusion_reid (target) #####
Extracting features from query set ...
Done, obtained 1000-by-2048 matrix
Extracting features from gallery set ...
Done, obtained 1000-by-2048 matrix
Speed: 0.0552 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 52.3%
CMC curve
Rank-1  : 58.5%
Rank-3  : 69.6%
Rank-5  : 75.0%
Rank-10 : 82.4%
Rank-20 : 88.0%
##### Evaluating p_duke (target) #####
Extracting features from query set ...
Done, obtained 2163-by-2048 matrix
Extracting features from gallery set ...
Done, obtained 9053-by-2048 matrix
Speed: 0.0554 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 23.2%
CMC curve
Rank-1  : 39.1%
Rank-3  : 49.4%
Rank-5  : 54.0%
Rank-10 : 61.5%
Rank-20 : 67.7%
##### Evaluating partial_reid (target) #####
Extracting features from query set ...
Done, obtained 300-by-2048 matrix
Extracting features from gallery set ...
Done, obtained 300-by-2048 matrix
Speed: 0.0546 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 64.7%
CMC curve
Rank-1  : 64.3%
Rank-3  : 76.3%
Rank-5  : 83.0%
Rank-10 : 88.3%
Rank-20 : 94.7%
Checkpoint saved to "../PVPM/model.pth.tar-30"

by the way, the pose information of occluded-reid and partial-reid comes from your link,but the pose information of P-duckMTMC made by myself which follows your readme 's guide, the result of this dataset seems terrible ,can you offer P-dukeMTMC 's pose information for us? Look forward to your favourable reply！！

[jpainam]

@chenxiangzZ Thanks. I also changed the issue title to reflect what was discussed here.

[hh23333]

@jpainam @chenxiangzZ In order to find the reason for the low results, I reproduced the experiment on another device(ubuntu16.04, GPU2080Ti, and mine is ubuntu18.04 with GPU1080Ti) following the instruction in README. I find the results is also similiar to what I report in the paper. So, I recommend you to reproduce the experiment with no modification in the code first.

[chenxiangzZ]

@hh23333 I haven't modified the code,just adjust the dataset's location, and i have the low result with @jpainam ,is the github's code same with your computer's? I especially want to reproduce your code！

[hh23333]

emmm...., I directly clone it to the new device. emmm.., Try to train the pvpm with my provided PCB baseline model directly?

[jpainam]

python scripts/main.py --load-pose --root ../PVPM_experiments/data \
 -s market1501\
 -t partial_reid occlusion_reid p_duke\
....

In thes training parameters. the source dataset -s is set to market1501 while the target datasets -t are partial_reid, occlussion_reid or p_duke. To me, this setting looks more like an unsupervised domain adaptation UDA (i.e. trained on a dataset and test on a different datasets). I did write a paper about UDA in person re-id.

So, you confirm that, we trained on market1501 and test on partial_reid, occlusion_reid or p_duke. Because this is what we are doing.

[chenxiangzZ]

it is transfer learning,this is what has been illustrated in the paper ,if you trained on p_duke and test on p_duke, the result will be higher!

[chenxiangzZ]

@hh23333 when i trained on your PCB baseline ，the result improved 👍 `Evaluating occlusion_reid (target)

Extracting features from query set ... Done, obtained 1000-by-2048 matrix Extracting features from gallery set ... Done, obtained 1000-by-2048 matrix Speed: 0.0547 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 59.6% CMC curve Rank-1 : 67.4% Rank-3 : 78.1% Rank-5 : 82.3% Rank-10 : 87.6% Rank-20 : 92.6%

Evaluating p_duke (target)

Extracting features from query set ... Done, obtained 2163-by-2048 matrix Extracting features from gallery set ... Done, obtained 9053-by-2048 matrix Speed: 0.0553 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 29.3% CMC curve Rank-1 : 50.4% Rank-3 : 59.9% Rank-5 : 63.8% Rank-10 : 69.8% Rank-20 : 75.0%

Evaluating partial_reid (target)

Extracting features from query set ... Done, obtained 300-by-2048 matrix Extracting features from gallery set ... Done, obtained 300-by-2048 matrix Speed: 0.0539 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 70.3% CMC curve Rank-1 : 73.7% Rank-3 : 85.7% Rank-5 : 89.0% Rank-10 : 93.3% Rank-20 : 96.3% Checkpoint saved to "../PVPM/model.pth.tar-30" Elapsed 1:16:12

so is there something wrong in the PCB trianing?and can you provide the p-duke 's pose imformation?

[jpainam]

@chenxiangzZ pls, can you show me the results of the PCB baseline evaluation alone on market1501 dataset? These are my results

##### Evaluating market1501 (source) #####
Extracting features from query set ...
Done, obtained 3368-by-12288 matrix
Extracting features from gallery set ...
Done, obtained 15913-by-12288 matrix
Speed: 0.0270 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 78.1%
CMC curve
Rank-1  : 92.8%
Rank-3  : 96.3%
Rank-5  : 97.2%
Rank-10 : 98.1%
Rank-20 : 98.8%
Checkpoint saved to "market_PCB/model.pth.tar-60"
Elapsed 7:36:13

[chenxiangzZ]

@jpainam do you mean the result on auther's PCB baseline，or the baseline trained by myself？ the result trained by myself follows: `

Evaluating market1501 (source)

Extracting features from query set ... Done, obtained 3368-by-12288 matrix Extracting features from gallery set ... Done, obtained 15913-by-12288 matrix Speed: 0.0259 sec/batch Computing distance matrix with metric=euclidean ... Computing CMC and mAP ... Results mAP: 77.5% CMC curve Rank-1 : 92.7% Rank-3 : 96.5% Rank-5 : 97.3% Rank-10 : 98.3% Rank-20 : 98.8% Checkpoint saved to "../market_PCB/model.pth.tar-60" ` and i havent test on auther‘s baseline, if you need ,i will try!’

[jpainam]

@chenxiangzZ I mean the author's PCB baseline. Maybe the author's PCB baseline on market1501 dataset is higher than ours. We got the same mAP: 77.5, rank-1: 92.7 on our PCB baseline. I would like to know, how about the authors PCB baseline results? (Sorry, i haven't setup his environment. That's why I'm asking). Thanks

[mohuang1996]

The PCB model you shared didn't work. When i try to train my own PCB but using your code it works. I don't know why. By the way,can you provide the pose information of P-duckMTMC?I can not find it from your link. Thanks. @hh23333

[chenxiangzZ]

@jpainam when i test on author's PCB baseline ,use the following command: python scripts/main.py --root ../PVPM_experiments/data \ -s market1501 -t market1501\ --save-dir ../market_PCB\ -a pcb_p6 --gpu-devices 0 --fixbase-epoch 0\ --open-layers classifier fc\ --new-layers classifier em\ --transforms random_flip\ --optim sgd --lr 0.02\ --stepsize 25 50\ --load-weights ../market_PCB/Pretrain_PCB_model.pth.tar-60\ --staged-lr --height 384 --width 192\ --batch-size 32 --base-lr-mult 0.5 \ --evaluate

and got the lower result than us :

Results mAP: 73.4% CMC curve Rank-1 : 90.6% Rank-3 : 95.0% Rank-5 : 96.6% Rank-10 : 97.6% Rank-20 : 98.5%

[jpainam]

That's really weird. the authors' PCB baseline has a lower performance than ours. But when using it with PVPM, it increases the final results.

[chenxiangzZ]

@jpainam @hh23333 is something wrong in the trainning of pcb baseline?Can you confirm that the training commands of pcb are valid in the README?

[jpainam]

@chenxiangzZ Yes, the command for training PCB baseline is valid. It worked for me with the following results

##### Evaluating market1501 (source) #####
Extracting features from query set ...
Done, obtained 3368-by-12288 matrix
Extracting features from gallery set ...
Done, obtained 15913-by-12288 matrix
Speed: 0.0270 sec/batch
Computing distance matrix with metric=euclidean ...
Computing CMC and mAP ...
** Results **
mAP: 78.1%
CMC curve
Rank-1  : 92.8%
Rank-3  : 96.3%
Rank-5  : 97.2%
Rank-10 : 98.1%
Rank-20 : 98.8%
Checkpoint saved to "market_PCB/model.pth.tar-60"
Elapsed 7:36:13

[Blueyao17]

The PCB model you shared didn't work. When i try to train my own PCB but using your code it works. I don't know why. By the way,can you provide the pose information of P-duckMTMC?I can not find it from your link. Thanks. @hh23333

Do you have pose information now? Can you share it if you do?

[hh23333]

I apologize for the delayed response. I have recently reviewed the code again and found that the discrepancy is due to a difference in the image sizes used for training and testing in the script I provided. Changing the scale during the training phase from 384x192 to 384x128 should yield the results presented in the paper.

[mohuang1996]

您的信件已将收到，谢谢！