search

argoverse / av2-api

Argoverse 2: Next generation datasets for self-driving perception and forecasting.
https://argoverse.github.io/user-guide/
MIT License
317 stars 75 forks source link

[online leaderboard] av2 scene flow test data have different array, cannot eval online #210

Closed Kin-Zhang closed 1 year ago

Kin-Zhang commented 1 year ago

Information:

  1. Error message online:

    boolean index did not match indexed array along dimension 0; dimension is 67074 but corresponding boolean dimension is 67073'

  2. API version I used: https://github.com/argoverse/av2-api/commit/64dd83c1429f1d5aaea2e0333d88fa2630e5328c

  3. Dataset version I downloaded through at 01 Jun 2023:

    s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/sensor/val/*" sensor/val
s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/sensor/test/*" sensor/test
# also the mask
s5cmd --no-sign-request cp "s3://argoverse/tasks/3d_scene_flow/zips/*" .

  4. Script I run, note There is no error when I run the local evaluation scripts.

    # generate mask
python3 av2-api/src/av2/evaluation/scene_flow/make_mask_files.py test-mask.zip <data_set_path>  --split test
# run the example submission
python av2-api/src/av2/evaluation/scene_flow/example_submission.py checkpoints/results/test/example <data_set_path> test-mask.zip
# upload to the online leaderboard

I didn't change the code av2-api, so maybe there are some dataset or mask file versions different now and the online leaderboard didn't update?

Kin-Zhang commented 1 year ago

Ha I see the problem is the downloaded mask and run av2-api script is different, I rerun with downloaded mask file and success.

av2-api/src/av2/evaluation/scene_flow/make_mask_files.py