JialianW / TraDeS

Track to Detect and Segment: An Online Multi-Object Tracker (CVPR 2021)
MIT License
553 stars 108 forks source link

Low performance on the nuscenes validation set #53

Open lhiceu opened 2 years ago

lhiceu commented 2 years ago

Hello @JialianW I test the performance of the model you provided (nuscenes.pth)on the nuscenes validation set following the readme. But I got low performance which is different from the results in the paper.

### Final results ###

Per-class results:
        AMOTA   AMOTP   RECALL  MOTAR   GT  MOTA    MOTP    MT  ML  FAF TP  FP  FN  IDS FRAG    TID LGD
bicycle     0.000   1.495   0.552   0.000   1993    0.000   0.793   27  38  683.2   864 35219   893 236 90  1.07    1.82
bus         0.062   1.420   0.224   0.394   2112    0.088   0.811   9   87  17.5    470 285 1638    4   19  0.70    4.02
car         0.088   0.893   0.114   0.520   58317   0.059   0.486   133 3379    55.1    6582    3158    51646   89  166 2.04    4.35
motorcy     0.000   1.579   0.545   0.000   1977    0.000   0.851   29  24  494.5   819 22430   899 259 118 1.14    2.32
pedestr     0.000   1.394   0.639   0.000   25423   0.000   0.873   486 198 2454.6  10887   143963  9168    5368    1778    0.60    1.73
trailer     0.000   1.614   0.543   0.000   2425    0.000   1.084   26  29  2907.8  869 151552  1108    448 169 1.32    2.16
truck       0.003   1.341   0.093   0.139   9650    0.013   0.939   23  509 21.2    884 761 8756    10  30  0.70    3.08

Aggregated results:
AMOTA   0.022
AMOTP   1.391
RECALL  0.387
MOTAR   0.150
GT  14556
MOTA    0.023
MOTP    0.834
MT  733
ML  4264
FAF 947.7
TP  21375
FP  357368
FN  74108
IDS 6414
FRAG    2370
TID 1.08
LGD 2.78
Eval time: 3056.8s 

I also tested the results from CenterTrack and got good performance. So I ruled out the issues of data preparation and nuScenes dataset API. What anything else should I do to get the same performance as you provided? Thank you.

lhiceu commented 2 years ago

Details about opt.txt :

==> commit hash: b'0443c36\n'
==> torch version: 1.3.1
==> cudnn version: 7603
==> Cmd:
['test.py', 'tracking,ddd', '--exp_id', 'nuScenes_3Dtracking', '--dataset', 'nuscenes', '--pre_hm', '--track_thresh', '0.1', '--gpus', '0', '--inference', '--load_model', '../models/nuscenes.pth', '--clip_len', '2', '--trades']
==> Opt:
  K: 100
  add_05: False
  amodel_offset_weight: 1
  arch: dla_34
  aug_rot: 0
  backbone: dla34
  batch_size: 32
  box_nms: -1
  chunk_sizes: [32]
  clip_len: 2
  custom_dataset_ann_path: 
  custom_dataset_img_path: 
  data_dir: /media/he/Disk/TraDeS/src/lib/../../data
  dataset: nuscenes
  dataset_version: 
  debug: 0
  debug_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/debug
  debugger_theme: white
  deform_kernel_size: 3
  demo: 
  dense_reg: 1
  dep_weight: 1
  depth_scale: 1
  dim_weight: 1
  dla_node: dcn
  down_ratio: 4
  efficient_level: 0
  embedding: False
  eval_val: False
  exp_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd
  exp_id: nuScenes_3Dtracking
  fix_res: True
  fix_short: -1
  flip: 0.5
  flip_test: False
  fp_disturb: 0
  gpus: [0]
  gpus_str: 0
  head_conv: {'hm': [256], 'reg': [256], 'wh': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]}
  head_kernel: 3
  heads: {'hm': 10, 'reg': 2, 'wh': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2}
  hm_disturb: 0
  hm_hp_weight: 1
  hm_weight: 1
  hp_weight: 1
  hungarian: False
  ignore_loaded_cats: []
  inference: True
  input_h: 448
  input_res: 800
  input_w: 800
  keep_res: False
  kitti_split: 3dop
  load_model: ../models/nuscenes.pth
  load_results: 
  lost_disturb: 0
  lr: 0.000125
  lr_step: [60]
  ltrb: False
  ltrb_amodal: False
  ltrb_amodal_weight: 0.1
  ltrb_weight: 0.1
  map_argoverse_id: False
  master_batch_size: 32
  max_age: -1
  max_frame_dist: 3
  model_output_list: False
  msra_outchannel: 256
  nID: -1
  neck: dlaup
  new_thresh: 0.1
  nms: False
  no_color_aug: False
  no_pause: False
  no_pre_img: False
  no_repeat: True
  non_block_test: False
  not_cuda_benchmark: False
  not_idaup: False
  not_max_crop: False
  not_prefetch_test: False
  not_rand_crop: False
  not_set_cuda_env: False
  not_show_bbox: False
  not_show_number: False
  num_classes: 10
  num_epochs: 70
  num_head_conv: 1
  num_iters: -1
  num_layers: 101
  num_stacks: 1
  num_workers: 4
  nuscenes_att: False
  nuscenes_att_weight: 1
  off_weight: 1
  optim: adam
  out_thresh: 0.1
  output_h: 112
  output_res: 200
  output_w: 200
  overlap_thresh: -1
  pad: 31
  pre_hm: True
  pre_img: True
  pre_thresh: 0.1
  print_iter: 0
  prior_bias: -4.6
  public_det: False
  qualitative: False
  reg_loss: l1
  reset_hm: False
  resize_video: False
  resume: False
  reuse_hm: False
  root_dir: /media/he/Disk/TraDeS/src/lib/../..
  rot_weight: 1
  rotate: 0
  same_aug_pre: False
  save_all: False
  save_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking
  save_framerate: 30
  save_img_suffix: 
  save_imgs: []
  save_point: [90]
  save_results: False
  save_video: False
  scale: 0
  seed: 317
  seg: False
  seg_feat_channel: 8
  shift: 0
  show_track_color: True
  skip_first: -1
  tango_color: False
  task: tracking,ddd
  test: False
  test_dataset: nuscenes
  test_focal_length: -1
  test_scales: [1.0]
  track_thresh: 0.1
  tracking: True
  tracking_weight: 1
  trades: True
  trainval: False
  transpose_video: False
  use_kpt_center: False
  use_loaded_results: False
  val_intervals: 10000
  velocity: False
  velocity_weight: 1
  video_h: 512
  video_w: 512
  vis_gt_bev: 
  vis_thresh: 0.3
  weights: {'hm': 1, 'reg': 1, 'wh': 0.1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1, 'cost_volume': 1.0}
  wh_weight: 0.1
  window_size: 7
  zero_pre_hm: False
  zero_tracking: False