Evaluate prediction accuracy with nuScenes

xmfcx commented 3 months ago

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I've agreed with the maintainers that I can plan this task.

Description

Autoware currently uses rule-based prediction modules with simple linear models. We aim to assess the performance of these existing prediction modules using the nuScenes dataset's Prediction Challenge to better understand their accuracy and reliability before integrating learning-based prediction algorithms.

Similar previous work:

https://github.com/autowarefoundation/autoware.universe/issues/565

Purpose

The purpose of this evaluation is to benchmark Autoware's current prediction capabilities against the nuScenes dataset. This will provide a clear baseline of performance and highlight areas for improvement as we transition to more sophisticated learning-based prediction algorithms.

Possible approaches

Utilize the nuscenes prediction dataset and follow the guidelines provided in the nuscenes prediction tutorial.
Analyze the performance of existing linear prediction models in Autoware against the benchmark data.
Document the findings and identify potential areas where machine learning models could offer significant improvements.

Definition of done

[ ] Download the nuScenes prediction dataset from the nuScenes download page.
[ ] Setup the environment and necessary dependencies to run the prediction tutorial notebook.
[ ] Create a new repository for an adapter tool to facilitate the integration of Autoware with the nuScenes dataset.
[ ] Run the existing Autoware prediction models on the nuScenes dataset using the adapter tool.
[ ] Collect and document the performance metrics from the evaluations.
[ ] Prepare a step-by-step document detailing the process to reproduce the evaluations, including any configurations and tools used.
[ ] Prepare a final report summarizing the findings and outlining recommendations for the integration of learning-based models.

sezan92 commented 3 months ago

I have started working on it. I will try to update in this issue regularly. Right now I am trying to understand the notebook. my next plan is to generate some sample predictions and run the evaluation pipeline. then i would like to use autoware for prediction

sezan92 commented 2 months ago

Update 2024/04/21

[x] I have kind of read the overall idea to nuscense datase and the #565

meliketanrikulu commented 2 months ago

Hello @sezan92 . Thanks for your efforts. Is there any update or open PR related this issue ?

sezan92 commented 2 months ago

@meliketanrikulu yes, I am continuously working on this. I am in the middle of setting up the nuscense dev kit in my local machine , then I plan to use Autoware perception stack . My plan is to feed the data from nuscenes to the perception node (sorry if my naming is wrong) and get the prediction. then i will evaluate using the prediction. but I am progressing slowly as I am also busy in my full time job. is that fine?

meliketanrikulu commented 2 months ago

@meliketanrikulu yes, I am continuously working on this. I am in the middle of setting up the nuscense dev kit in my local machine , then I plan to use Autoware perception stack . My plan is to feed the data from nuscenes to the perception node (sorry if my naming is wrong) and get the prediction. then i will evaluate using the prediction. but I am progressing slowly as I am also busy in my full time job. is that fine?

@sezan92 Thanks for your effort. No problem . I just asked to check the assignee part. I wish you good work :)

sezan92 commented 1 month ago

Update 2024/06/06

[x] sorry for late update. I have successfully run the prediction notebook provided by nuscense. I am going through a similar PR https://github.com/autowarefoundation/perception_benchmark_tool/pull/1 to get idea how to create a benchmarking tool for this

TODO

[ ] start working on a basic benchmarking tool using autoware.

mitsudome-r commented 3 weeks ago

@sezan92 Are you working on this at the moment? If you cannot work on this, I would like to assign someone else.

sezan92 commented 3 weeks ago

@mitsudome-r i am working on this but the progress is slow, if it is urgent for autoware, then you are free to re-assign.

autowarefoundation / autoware.universe