Low minADE and minFDE scores for inference

[chenfengw]

Dear author,

I did the flowing step to test out the pretrained model, but the minADE and FDE score I got is pretty low.

1. ./scripts/preprocessing.sh
2. Download TNT pretrained wight here and unzip to the TNT-Trajectory-Predition directory.
3. python test_tnt.py -rm TNT/best_TNT.pth I added compute_metric=True inside trainer.test() function. The result I got is

   The test result: {'minADE': 96.14342095395128, 'minFDE': 261.1454114112499, 'MR': 1.0};

   which is much lower than expected, but I have follow the procedure as instructed and there is no other code change.

[Henry1iu]

Hi,

Which split are you using for your testing? Also, please check if the normalized been set to True. The pre-trained model is trained under the normalized coordinate. I think you tested it with the trajectory in the original coordinate.

Best regards, Jianbang

[chao-SFD]

Hello, @chenfengw . I did the following step to test out the pretrained model, and the associated coordinates are also converted to the normalized coordinate system. I have the similar result as yours: The test result: {'minADE': 99.6560015459436, 'minFDE': 281.07380034322466, 'MR': 1.0};

And dear auther @Henry1iu , after carefully read the code provided, I found there are two confused questions:

1. In the test dataset, there are no GroundTrue(GT) in the last 3s, and in the preprocessing stage(argoverse_preprocess_v2.py), the Offset of GT will be set to 0: So If TNT is used to make a prediction in this dataset, it might lead a mistake in the stage of computing metrics. Because the predicted value is actually compared to the position of the latest frame. The following figure shows the comparison of "GT" and the predicted results of TNT model in test dataset:

2. If the test data set is changed to val(split=val) dataset, it is found that the prediction values of the model and GT differ greatly. The following figure compares "GT" and the predicted results of TNT model in val dataset:

I don't know if there are other settings I ignored. And I am also longing for your reply. best regards， Chao

[Henry1iu]

Hello, @chenfengw . I did the following step to test out the pretrained model, and the associated coordinates are also converted to the normalized coordinate system. I have the similar result as yours: The test result: {'minADE': 99.6560015459436, 'minFDE': 281.07380034322466, 'MR': 1.0};

And dear auther @Henry1iu , after carefully read the code provided, I found there are two confused questions:

1. In the test dataset, there are no GroundTrue(GT) in the last 3s, and in the preprocessing stage(argoverse_preprocess_v2.py), the Offset of GT will be set to 0: So If TNT is used to make a prediction in this dataset, it might lead a mistake in the stage of computing metrics. Because the predicted value is actually compared to the position of the latest frame. The following figure shows the comparison of "GT" and the predicted results of TNT model in test dataset:
2. If the test data set is changed to val(split=val) dataset, it is found that the prediction values of the model and GT differ greatly. The following figure compares "GT" and the predicted results of TNT model in val dataset:

I don't know if there are other settings I ignored. And I am also longing for your reply. best regards， Chao

Dear Chao,

Thanks for your detailed information. I know where the bug is.

The reason is that: The trajectory completion of my implementation predicted the trajectory offset in my previous version. Recently, I find that predicting the complete trajectory results in a better performance. But, the test function is designed for the offset prediction.

I will fix this bug in testing. BTW, the compute metric function shouldn't be applied to the test set since the ground truth is not available. Please be noted.

Best regards, Jianbang

[Henry1iu]

Hi.

I have corrected the code and checked the performance on val set. It should be ok now. Feel free to check the performance yourself. Please remember that only apply the metric computation on val set.

If there is no further problem, I will close this issue. Thansk!

Best Regards, Jianbang

[chao-SFD]

Hi.

I have corrected the code and checked the performance on val set. It should be ok now. Feel free to check the performance yourself. Please remember that only apply the metric computation on val set.

If there is no further problem, I will close this issue. Thansk!

Best Regards, Jianbang

Dear author,

I have synchronized my local code， and test the pretrained TNT model on val dataset. And my result is: The test result: {'minADE': 1.1289799042637099, 'minFDE': 2.0268777046092445, 'MR': 0.19876868953386104};

And there are somewhat different from the published results (Especially on minADE and minFDE). And this gap still seems unacceptable.

[Henry1iu]

Hi. I have corrected the code and checked the performance on val set. It should be ok now. Feel free to check the performance yourself. Please remember that only apply the metric computation on val set. If there is no further problem, I will close this issue. Thansk! Best Regards, Jianbang

Dear author,

I have synchronized my local code， and test the pretrained TNT model on val dataset. And my result is: The test result: {'minADE': 1.1289799042637099, 'minFDE': 2.0268777046092445, 'MR': 0.19876868953386104};

And there are somewhat different from the published results (Especially on minADE and minFDE). And this gap still seems unacceptable.

Dear user,

As I stated in the readme, it's not the official implementation. I just follow the details described in their publication. I think there are still details hidden behind. The authors refused to provide further detail to me due to the so-called "patent issue". This is the best performance I have ever got. You can try to reproduce the performance yourself if you think it's unacceptable. : )

Good Luck & Cheers Jianbang

[chao-SFD]

Hi. I have corrected the code and checked the performance on val set. It should be ok now. Feel free to check the performance yourself. Please remember that only apply the metric computation on val set. If there is no further problem, I will close this issue. Thansk! Best Regards, Jianbang

Dear author, I have synchronized my local code， and test the pretrained TNT model on val dataset. And my result is: The test result: {'minADE': 1.1289799042637099, 'minFDE': 2.0268777046092445, 'MR': 0.19876868953386104}; And there are somewhat different from the published results (Especially on minADE and minFDE). And this gap still seems unacceptable.

Dear user,

As I stated in the readme, it's not the official implementation. I just follow the details described in their publication. I think there are still details hidden behind. The authors refused to provide further detail to me due to the so-called "patent issue". This is the best performance I have ever got. You can try to reproduce the performance yourself if you think it's unacceptable. : )

Good Luck & Cheers Jianbang

Thank you very much for your patience, and I have gained a lot from your important work. I will try to reproduce the performance.

best regards， Chao