Closed lexavtanke closed 1 month ago
@lexavtanke could you create a Discussion here https://github.com/orgs/autowarefoundation/discussions/categories/design outlining your proposed approach and how this will integrate with the Autoware architecture?
@BonoloAWF I've started discussion in design section link
This pull request has been automatically marked as stale because it has not had recent activity.
Model which will be used as a baseline for the integration in to Autoware is BEVDet (it has occupancy head and it is realtime). In the future it is possible to move to a multimodal model (lidar + cams) like BEVfusion. Current actual code for the model and its conversion into TensorRT is in the repo Current status is to make a code to convert BEVDet occ model to TensorRT. The autors of the original repo provide tensorrt conversion only for base BEVDet model but alll occ versions based on BEVDet4D with mods (stereo, depth).
Latest progress report can be acceded in Perception and Sensing WG Meeting notes
Latest progress report can be accessed here.
And current brief status: Code for exporting BEVDet4D to TensorRT is done and in the repo.
Inference speed test results on RTX 3080 Mobile:
Now I'm working on conversion BEVDet4DStereoOCC to TensorRT.
Currently I it possible to export BEVDet4dStereoOcc model to ONNX with the use of the contrib operators because it supports inverse operator. But after it is impossible to convert this model to TensorRT because it doesn't support inverse operator.
Inverse operator is used in method gen_grid of the DepthNet part of the view_transformer net. So I start to investigate how I can work with this.
After some investigation I found that It is possible to export a pytorch model as several separate engines (as it was done with the center point model in Autoware) but I didn't find any documentation in Autoware about how it was exported after training. There is a mention that center_point was trained with mmdetection3d. But I didn't find any export code for ONNX and TensorRT export there.
@miursh Can you please provide the information about export center_point to ONNX and TensorRT or tell who can help with this question?
After it will be possible to separate the model in 2 parts and write code for gen_grid in the node which will call the engines and process the data.
Also possible way which need a bit more investigation is TPAT. They tell that it is possible to autogenerate plugins for unsupported operators but I'm not sure that It will works with contrib operators.
And the last way is to write own custom plugin for TensorRT.
This pull request has been automatically marked as stale because it has not had recent activity.
I will close this one since it has been stale for a year.
This was sort of superseeded by:
Checklist
Description
Integrate occupancy prediction to the autoware perception stack to increase reliability of occupancy grid map. Because now only heuristic method used in this task in autoware as https://autowarefoundation.github.io/autoware.universe/main/perception/ground_segmentation/. This method is lack of semantic information and also It is tedious to adjust to environment where vehicle operates.
With ML based occupancy prediction it is possible to apply for sophisticated logic for obstacle detection and use additional information in planning to increase reliability and comfort. Also 3d occupancy information allow to cope with situations then shape of detected objects can't be incorporated to the simple form as bounding box (unusual vehicle or just simple open door)
Purpose
Enhance Autoware with ML based occupancy prediction method.
Possible approaches
Definition of done