argoverse2 tracking benchmark upload error

[lejk8104]

Dear Argo,

I am reaching out to inform you about an issue I encountered while using your valuable service for the Argoverse 2: End-to-End Forecasting Challenge, in which I attempted to evaluate the performance of my tracking model (Evaluate about Tracking section).

In Stderr file log Traceback (most recent call last): File "remote_challenge_evaluation/run_worker.py", line 86, in score_submission eval_out = evaluate(test_annotation_path, File "/home/nperi/Workspace/AV2-Challenge/av2-e2e-prediction-challenge-evalai/evaluation_script/main.py", line 75, in evaluate _, tuned_hota_per_class, _ = evaluate_tracks(prediction, ground_truth, "HOTA", max_range_m, dataset_dir, "") File "/home/nperi/Workspace/Core/av2-api/src/av2/evaluation/tracking/eval.py", line 525, in evaluate utils.annotate_frame_metadata( File "/home/nperi/Workspace/Core/av2-api/src/av2/evaluation/tracking/utils.py", line 55, in annotate_frame_metadata assert len(prediction_frames) == len(label_frames) AssertionError

I came across identical error messages in both the test set and the validation set. I used the "sensor dataset" downloaded for the evaluation, and the count of frames in the pkl file I uploaded - the outcome of len(prediction_frames) - was 23547 and 23574 for the validation and test sets respectively.

In light of this, I am curious if you could provide precise information about the total quantity of the tracking validation and test data.

I look forward to your response and appreciate your help in advance.

[neeharperi]

Hi @lejk8104, The validation set is of length 2404. I have added a print statement with this information to the assert statement to make it easier to debug. Please see our documentation for more information on the expected format, or this sample tracking result for the validation set. Please let me know if you have any further questions.

Neehar

[lejk8104]

Thank you for your kind response. I will try to resolve the issue by checking the sample.

Could you possibly inform me about the deadline for the CVPR2023 workshop Tracking Challenge?

[neeharperi]

All submissions sent before June 11th 11:59 PM EST will be considered for the CVPR 2023 WAD Challenge. The AV2 evaluation server will still be open after this date if you'd like to continue submitting to the leaderboard.

[lejk8104]

I would like to express my gratitude for the sample data you provided, which enabled me to effectively resolve the previous issue I was facing.

Q1. However, I must inform you that a new error has occurred during the server upload process.

our error msg

I attempted to debug this error, but found that the relevant part of the code has been changed.

latest api

Q2. In light of this, could you kindly confirm if the tracking eval code provided in the challenge I am currently participating in differs from the eval code shared on the argo2-api GitHub?

The attached image contains information about the sample data provided by you and my data.

your samples

our sample

our data

As always, I am grateful for your assistance.

[neeharperi]

Q1: It seems that the same issue you had previously is still triggering the same assert statement on L55. Unfortunately I can't see any obvious issues with the screenshots you shared. Can you confirm that both your submission and the sample tracking file I provided evaluate correctly on your local machine? Q2: I can confirm that the evaluation code running on our server is identical to the av2-api

[lejk8104]

I would like to inform you that both the pkl file provided by you and our model's pkl file operate on the Local PC without error.

The two attached files, eval_argo2_tracking_examples and eval_our_tracking_result are each results generated on the Local PC and saved pkl format. Furthermore, the our_label, which used for the validation was created by slightly modifying the LT3D code.

This is the pkl file that is currently not functioning on the evaluation server (Eval AI validation task).

[neeharperi]

I think I've figured out the issue. About two months ago, we pushed a small fix (See Issue) that corrected the sampling rate. Both your tracking results and my tracking results that I shared with you were generated using an older version of the av2_infos_trainval.pkl file. If you pull the latest version of LT3D, or make the small fix described in the above issue, I think you should be able to successfully submit to EvalAI.

The corrected length of the val set should be 4806. I'm regenerating the new av2_infos_trainval.pkl file on my local machine to validate this as well. Please let me know if this solves your issue.

[lejk8104]

By referring to the link you provided and adjusting the sampling cycle from 10 to 5, the test on Eval AI was successfully conducted. Your detailed and kind response was of great assistance in resolving this issue.

Thank you very much.