search

tusen-ai / SST

Code for a series of work in LiDAR perception, including SST (CVPR 22), FSD (NeurIPS 22), FSD++ (TPAMI 23), FSDv2, and CTRL (ICCV 23, oral).
Apache License 2.0
765 stars 96 forks source link

Deploy FSDv2 on TensorRT #162

Closed ArseniuML closed 6 months ago

ArseniuML commented 9 months ago

I want to deploy FSDv2 on TensorRT, but it seems that I need an MMDeploy config to do this.

Here is an example config from MMDeploy docs:

backend_config = dict(
    type='tensorrt',
    common_config=dict(
        fp16_mode=False, max_workspace_size=1 << 30),
    model_inputs=[
        dict(
            input_shapes=dict(
                input=dict(
                    min_shape=[1, 3, 512, 1024],
                    opt_shape=[1, 3, 1024, 2048],
                    max_shape=[1, 3, 2048, 2048])))
    ])

What min_shape, opt_shape and max_shape do I have to paste into this config? It seems, that FSDv2 receives an unbounded list of points as input...

Abyssaledge commented 9 months ago

I don't think you can directly deploy FSDv2 with TensorRT, which doesn't support Sparse Convolution ...

ArseniuML commented 9 months ago

OK, but what about min_shape, opt_shape and max_shape? Let's assume we figured sparse convolution out.

Abyssaledge commented 9 months ago

It depends on you. You should set a fixed threshold to the number of points.

ArseniuML commented 9 months ago

Is it possible to deploy FSDv2 on TensorRT with the help of SparseConv3dLayerPlugin from here:

https://github.com/jingyue202205/SE-SSD-AI-TRT/blob/master/sparseConv3dlayer.h

How difficult will it be to do this? Will it be possible to do this (semi) automatically, or it will mean completely rewrite FSDv2 on C++ by hands?

Abyssaledge commented 6 months ago

The difficulty of FSDv2 deployment lies in it using spconv. So if you can find a solution for the spconv-based deployment, it should also work for FSDv2.