search

tusen-ai / SST

Code for a series of work in LiDAR perception, including SST (CVPR 22), FSD (NeurIPS 22), FSD++ (TPAMI 23), FSDv2, and CTRL (ICCV 23, oral).
Apache License 2.0
788 stars 100 forks source link

Correctness of FSDv2 performance on Argoverse2 #187

Open ArseniuML opened 6 months ago

ArseniuML commented 6 months ago

I run dist_test.sh with Argo2 config and obtained the same metrics that were reported in article (AP and others). However, it seems that test.py loads files from train split:

self.data_infos[0]
{'uuid': '04994d08-156c-3018-9717-ba0e29be8153/315971486059794000', 'sample_idx': '1002000', 'image': {}, 'point_cloud': {'num_features': 4, 'velodyne_path': 'training/velodyne/1002000.bin'}, 'calib': {}, 'pose': {}, 'annos': {'name': array(['Bicycle', 'Bicycle', 'Bicycle', 'Bicycle', 'Bollard', 'Bollard',
       'Boll...,
       'Regular_vehicle'], dtype='<U15'), 'truncated': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0.,..., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), 'occluded': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0,..., 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0]), 'alpha': array([-10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10.,
       -10.,...10., -10., -10., -10., -10.,
       -10.]), 'bbox': None, 'dimensions': array([[0.5       , 1.5       , 1.0864296 ],
       [0.5       , 1.5       , 1.173091...    [1.74      , 4.24284935, 1.65305519]]), 'location': array([[ 2.36707701e+01,  1.80500351e+01, -4.87866102e-02],
       [ 2.19137558e+01, ...6e+01,  3.86977774e+00,  3.84238931e-01]]), 'rotation_y': array([ 0.05450454, -1.97456541,  0.05685537, -1.95257706,  1.53496329,
        1.621...4168644,
       -1.54653023, -1.5323414 ]), 'index': array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17,...1, 62, 63, 64, 65, 66],
      dtype=int32), 'group_ids': None, 'camera_id': None, 'difficulty': None, 'num_points_in_gt': array([  18,   31,   13,   18,    8,    7,    6,   63,    4,    7,  224,
          5,...28,   32,    2,
         19], dtype=int32)}, 'sweeps': []}

why training/velodyne/1002000.bin is being loaded? Was it used for training?

Abyssaledge commented 6 months ago

Hello, thanks for your issue but it is your misunderstanding. We convert Argoverse 2 data into KITTI format. In KITTI format, the 'training' folder actually contains both the train split and the validation split. So, when you test FSDv2 in Argoverse 2 validation split, the loader will load data from 'training/velodyne'.