Evaluation results mismatch raw source file

[zhaone]

Recently I wanna visualize your prediction result but I got a little confused about the correspondence between the idx of preprocessed val data and $ARGO_RAW_DATA/val/data/argo_id.csv. In ArgoTestDataset : https://github.com/uber-research/LaneGCN/blob/7e9b51d18a62f5dafb67a2215eba9053a64aff16/data.py#L382 I download your preprocessed val data and found the prediction paths with argo_id doesn't match the source file $ARGO_RAW_DATA/val/data/argo_id.csv. My dataset source data is download from argo website(version 1.1), I want to know whether the order of preprocessed data matches argo_id? Or did I do something wrong?

[anant-joshi]

I tried this a few days back. There's a mismatch in the argo IDs for sure, but running the preprocessing script myself fixed the issue.

[chenyuntc]

Thanks for point out, will fix it after cvpr deadline

[zhaone]

Hi, may I ask whether you have uploaded newer preprocessed data? For me, due to the memory limitation, processing locally is sticky, even on the servers... Thank you 😄

[leoncesc]

Are other vehicles used during training not only the predicted agent vehicle but also other vehicles?

[zhaone]

Are other vehicles used during training not only the predicted agent vehicle but also other vehicles?

Yes (for loss and optimization), but only the agent vehicle is used for computing metrics.

[Kay1794]

I tried this a few days back. There's a mismatch in the argo IDs for sure, but running the preprocessing script myself fixed the issue.

Hi, thank you for your comments! I tried to preprocess data using the script they provided but still was not able to generated correct evaluation submission file. I was wondering when you preprocessed the data, did you directly run their script?

[chenyuntc]

Sorry for the late response here. I have been distracted since I left Uber.

I check the code, it seems that the idx and argo_idx mismatch because the ArgoVerseForcastingLoader returns files in an inconsistent order (os.listdir), which means it could be different in different machines.

Therefore I update the data.py in this commit. Hope this can fix the bug.

You need to rerun the preprocess_data.py or wait for me to update the preprocessed data. I will update here after the latest data is uploaded.

[Kay1794]

Sorry for the late response here. I have been distracted since I left Uber.

I check the code, it seems that the idx and argo_idx mismatch because the ArgoVerseForcastingLoader returns files in an inconsistent order (os.listdir), which means it could be different in different machines.

Therefore I update the data.py in this commit. Hope this can fix the bug.

You need to rerun the preprocess_data.py or wait for me to update the preprocessed data. I will update here after the latest data is uploaded.

Thank you so much for the reply! I will do as you suggested. Hopefully it'll work :)

[Kay1794]

Sorry for the late response here. I have been distracted since I left Uber.

I check the code, it seems that the idx and argo_idx mismatch because the ArgoVerseForcastingLoader returns files in an inconsistent order (os.listdir), which means it could be different in different machines.

Therefore I update the data.py in this commit. Hope this can fix the bug.

You need to rerun the preprocess_data.py or wait for me to update the preprocessed data. I will update here after the latest data is uploaded.

It worked! I just simply pulled the repo and reran the preprocess_data.py. Thank you so much!

[chenyuntc]

Preprocessed test pickled updated. Let me know if you still have issues.

[chenyuntc]

Note that, I only update preprocessed test split. You need to reprocess val/train if you want to get correct mapping between idx and argo_id in that split.

If you have such results, you are welcome to share them here. I will update them.