search

Khrylx / AgentFormer

[ICCV 2021] Official PyTorch Implementation of "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting".
https://www.ye-yuan.com/agentformer/
MIT License
257 stars 64 forks source link

About normalization #5

Closed sjtuxcx closed 3 years ago

sjtuxcx commented 3 years ago

Hi, I have noticed that in the code (https://github.com/Khrylx/AgentFormer/blob/cf13e4033ceef7bdcfbf27183842415e7b841d34/model/agentformer.py#L535) the center of both the past trajectory and the future trajectory is used to normalize the input data. However, the future trajectory should not be available in the test. Is this data snooping? Please let me know if there is anything wrong with my understanding. Many thanks!

Khrylx commented 3 years ago

Thanks for spotting this! This is indeed an oversight on my part. I think you could try to normalize with the positions at t=0 (self.data['pre_motion'][[0]]), retrain the model, and rerun the evaluation. I will also try to do this later.

Khrylx commented 3 years ago

I'm retraining with the correct scene_orig. It is worth noting that all the other models in the ablation study also use this scene_orig, so it does not give our method an edge in the ablations.

Mirorrn commented 3 years ago

Hello all,

first of all, thank you for publishing the code, which is very helpful for the community!

I can only partially agree with this, as the calculated anchor point for the proposed normalisation and calculation of the prediction does have an influence on the prediction. For example, the ETH scene has many 1-person trajectories. Considering that the history consists of 8 time steps and the prediction of 12, the anchor point may well contain information about where the person might be in the future. This is not the case with the Univ dataset with many people in one scene.

I have retrained the models with https://github.com/Khrylx/AgentFormer/blob/cf13e4033ceef7bdcfbf27183842415e7b841d34/model/agentformer.py#L535 changed to:

self.data['scene_orig'] = self.data['pre_motion'].view(-1, 2).mean(dim=0) All models are trained exactly like in the given config files. Here are the results for the ETH/UCY data set:

Scene | ADE/FDE ETH | 0.454/0.762 HOTEL | 0.144/0.233 UNIV | 0.252/0.45 Zara1 | 0.18/0.31 Zara2 | 0.14/0.241 Avg | 0.234/ 0.4

I still find the results very good and very similar to the Trajectron++. I think that the result on the dataset is almost perfect and I don't think that the result can be improved by more than 5%.

For the NuScenes dataset I have the problem that I could not reproduce the published results with the proposed (wrong anchor point) using the available script, which i had not for ETH/UTC. I followed all the instructions and downloaded the entire NuScenes dataset and converted it to a file of about 56 MB with the preprocessing script. When I use the model published by you, I get the result in the paper, but unfortunately not when I train it myself. Am I doing something wrong or is there something wrong with the config file? Here are the results for NuScenes-5samples:

Model | ADE\FDE original | 1.73 / 3.56 correct Anchor | 1.87/ 3.92

Again very similar to the Trajectron, but I hope that there is something wrong with the config file and that I can get much better results.

Many greetings, Mirrorn

Khrylx commented 3 years ago

Hi @Mirorrn ,

Thank you for your detailed analysis! I will keep investigating this error, and update the paper with the correct numbers.

One thing you may want to try is to change the anchor point to

self.data['scene_orig'] = self.data['pre_motion'][[-1]].view(-1, 2).mean(dim=0)

So that it will use t=0 as the scene origin.

For the nuScenes dataset, I haven't retrained the model yet, but I will retrain it soon with the correct anchor. It could be there is something wrong with the configs or something is changed during the code cleaning process.

Thanks!

Mirorrn commented 3 years ago

Hi,

unfortunately I have no time this week to test it with t=0 as anchor point.

Many greetings Mirrorn

Khrylx commented 3 years ago

Hi Mirrorn,

I’m actively running experiments and hopefully will have updates soon.

Thanks again for your help!

Khrylx commented 3 years ago

Sorry for the late update. I was busy with a deadline. I have modified the code using the correct scene_orig computation. Indeed, the performance has dropped quite a bit 😢. Still, our current correct model is a bit better than Trajectron++ on nuScenes. I have updated the pretrained models and the numbers both in the repo and the arXiv paper. Thanks again for everyone's participation and help!

Khrylx commented 3 years ago

I will close this issue for now, but if anything else comes up, please let me know.