NuScenes tracking results evaluation

[jeho-lee]

Thanks for the great work! I had an error while reproducing the tracking results using the NuScenes dataset. Since the official nuscenes tracking eval code is not in your repo, I used the eval code in here.

the command is, python tools/eval_nuscenes.py det_results/SimpleTrack2Hz/results/results_official.json --dataroot ../data/nuscenes/,

where eval_nuscenes.py is from the official nuscenes repo and the results_official.json is downloaded from the Dropbox you provided (nuscenes/validation_2hz/pred.json).

The main function of eval_nuscenes.py is,

And the error message is,

For debugging purposes, I printed out the values in the assert statement, Unachieved thresholds: 7, Duplicate thresholds: 6, Threshold metrics: 33, Total thresholds: 40

Anyone who solved this problem?

[ziqipang]

@jeho-lee Thanks for your question! I haven't encountered this issue before, but let me offer you an alternative so that you could try again.

Please try the code from my PF-Track. More specifically, the part as below. You only need to replace the path, etc, though. Hope this will help!

        from nuscenes import NuScenes
        output_dir = osp.join(*osp.split(result_path)[:-1])

        eval_set_map = {
            'v1.0-mini': 'mini_val',
            'v1.0-trainval': 'val',
        }
        from nuscenes.eval.tracking.evaluate import TrackingEval
        from nuscenes.eval.common.config import config_factory as track_configs

        cfg = track_configs("tracking_nips_2019")
        nusc_eval = TrackingEval(
            config=cfg,
            result_path=result_path,
            eval_set=eval_set_map[self.version],
            output_dir=output_dir,
            verbose=True,
            nusc_version=self.version,
            nusc_dataroot=self.data_root
        )
        metrics = nusc_eval.main()
        # record metrics
        metrics = mmcv.load(osp.join(output_dir, 'metrics_summary.json'))
        print(metrics)

[jeho-lee]

Thank you for the quick reply :) I've tried the code that you provided from PF-Track, but I got the same error message. Actually, it seems the PF-Track's eval code is also from the official nuscenes repo, which is exactly the same as the code that I tried first.

By the way, I'll further investigate how to solve the error, and I guess the package version issue might be the problem. If you don't mind, could you give me the package version list (pip or conda) of your environment?

[jeho-lee]

I found the solution from here. The numpy version and nuscenes-devkit version compatibility is the problem, and as soon as I upgrade the nuscenes package version to the latest, the problem is solved.