Big gap between my reproduction and paper's results in JAAD

[CrisCloseTheDoor]

Hi I wonder why there's a Big gap between my reproduction and paper's results in JAAD dataset.

here's my results compared to paper's results: Deterministic

my reproduction: MSE_05: 806.880521; MSE_10: 2894.095019; MSE_15: 6527.214091;
paper:  MSE_05: 82 MSE_10: 328 MSE_15: 1049 

CVAE

my reproduction: MSE_05: 95.9; MSE_10: 274.0; MSE_15: 617.9
paper:  MSE_05: 37 MSE_10: 86 MSE_15: 197 

my config is totally follow the code's default setting:

lr=5e-04
bbox_type='cxcywh'
dropout=0.0

Looking forward to your reply, thanks a lot @ChuhuaW

[CrisCloseTheDoor]

By the way, the results in ETH/UCY are as good as paper's. Here're more details when running with deterministic JAAD:

Generating action sequence data
fstride: 1
sample_type: all
subset: default
height_rng: [0, inf]
squarify_ratio: 0
data_split_type: default
seq_type: trajectory
min_track_size: 61
random_params: {'ratios': None, 'val_data': True, 'regen_data': True}
kfold_params: {'num_folds': 5, 'fold': 1}
---------------------------------------------------------
Generating database for jaad
jaad database loaded from *****\JAAD_datasets\data_cache\jaad_database.pkl
---------------------------------------------------------
Generating trajectory data
Split: train
Number of pedestrians: 1355 
Total number of used pedestrians: 980 
JAAD
---------------------------------------------------------
Generating action sequence data
fstride: 1
sample_type: all
subset: default
height_rng: [0, inf]
squarify_ratio: 0
data_split_type: default
seq_type: trajectory
min_track_size: 61
random_params: {'ratios': None, 'val_data': True, 'regen_data': True}
kfold_params: {'num_folds': 5, 'fold': 1}
---------------------------------------------------------
Generating database for jaad
jaad database loaded from *****\JAAD_datasets\data_cache\jaad_database.pkl
---------------------------------------------------------
Generating trajectory data
Split: val
Number of pedestrians: 202 
Total number of used pedestrians: 153 
JAAD
---------------------------------------------------------
Generating action sequence data
fstride: 1
sample_type: all
subset: default
height_rng: [0, inf]
squarify_ratio: 0
data_split_type: default
seq_type: trajectory
min_track_size: 61
random_params: {'ratios': None, 'val_data': True, 'regen_data': True}
kfold_params: {'num_folds': 5, 'fold': 1}
---------------------------------------------------------
Generating database for jaad
jaad database loaded from *****\JAAD_datasets\data_cache\jaad_database.pkl
---------------------------------------------------------
Generating trajectory data
Split: test
Number of pedestrians: 1023 
Total number of used pedestrians: 770 
  0%|          | 0/152 [00:00<?, ?it/s]Number of validation samples: 26
Number of test samples: 122
100%|██████████| 152/152 [05:25<00:00,  2.14s/it]
  0%|          | 0/122 [00:00<?, ?it/s]Train Epoch: 1    Goal loss: 1.8928   Decoder loss: 1.9545    
100%|██████████| 122/122 [01:33<00:00,  1.30it/s]
Test Loss: 3.7269
MSE_05: 850.564338;  MSE_10: 3265.110804;  MSE_15: 7784.713009

Saving checkpoints: metric_epoch_001_MSE15_7784.7130.pth
100%|██████████| 152/152 [05:30<00:00,  2.18s/it]
  0%|          | 0/122 [00:00<?, ?it/s]Train Epoch: 2    Goal loss: 1.7976   Decoder loss: 1.7880    
100%|██████████| 122/122 [01:33<00:00,  1.31it/s]
Test Loss: 3.7658
MSE_05: 832.921553;  MSE_10: 3244.152499;  MSE_15: 7759.023308

Saving checkpoints: metric_epoch_002_MSE15_7759.0233.pth
100%|██████████| 152/152 [05:32<00:00,  2.19s/it]
Train Epoch: 3   Goal loss: 1.7827   Decoder loss: 1.7726    
100%|██████████| 122/122 [01:35<00:00,  1.27it/s]
Test Loss: 3.7486
MSE_05: 828.523306;  MSE_10: 3234.219866;  MSE_15: 7739.036007

....
....

100%|██████████| 152/152 [05:28<00:00,  2.16s/it]
  0%|          | 0/122 [00:00<?, ?it/s]Train Epoch: 48   Goal loss: 1.5696   Decoder loss: 1.5972    
100%|██████████| 122/122 [01:33<00:00,  1.31it/s]
Test Loss: 3.4885
MSE_05: 806.880521;  MSE_10: 2894.095019;  MSE_15: 6527.214091

Saving checkpoints: metric_epoch_048_MSE15_6527.2141.pth
100%|██████████| 152/152 [05:27<00:00,  2.16s/it]
  0%|          | 0/122 [00:00<?, ?it/s]Train Epoch: 49   Goal loss: 1.5709   Decoder loss: 1.5980    
100%|██████████| 122/122 [01:33<00:00,  1.30it/s]
Test Loss: 3.4935
MSE_05: 813.618932;  MSE_10: 2907.024978;  MSE_15: 6545.161632

100%|██████████| 152/152 [05:28<00:00,  2.16s/it]
  0%|          | 0/122 [00:00<?, ?it/s]Train Epoch: 50   Goal loss: 1.5585   Decoder loss: 1.5800    
100%|██████████| 122/122 [01:34<00:00,  1.29it/s]
Test Loss: 3.5223
MSE_05: 912.481662;  MSE_10: 3155.994252;  MSE_15: 6857.372476

[Zhang-Xiaoxue]

Now, I cannot find the train_deterministic.py file in JAAD and PIE datasets.

[JaychC]

@CrisCloseTheDoor Hi, Could you please offer me your checkpoints of deterministic JAAD?