Question about Future States

[kaiyama12345679]

Hello, I'm very intersted in this work, and I would like to ask you two thing.

• First, how long does it take for this StateTransformer to predict future states? I'd like to know the wallclock time.
• Second, is it possible to get both motion prediction result(about other agents) and motion plannning(about the ego vehicle) at once? if possible, how?

I would appreciate your response.

[larksq]

Hi, thank you for your interest.

• The inference time is almost identical to other VLM models since we used language models as backbones. The wall clock time can vary a lot by inferencing with different sizes of the model. Here is the result of a quick test. Without further model quantization, the inference time for small GPT2 models with default 5 key points is about 0.052 seconds (code base origin/dev-sq). Note this does not include data pre-processions or post-processions.
• In short, it is not possible without changing the model structure and the training pipeline. But it is possible with some minor changes to the code. This is an interesting idea to explore. Shoot me an email for further discussions if interested.

[yukkysaito]

@larksq Thank you for your response. I am working together with @kaiyama12345679 .

I am very interested in StateTransformer and am considering how it could be integrated with Autoware. I asked the question to explore how this integration might be accomplished.

If I have any further questions, I look forward to your continued assistance.

[larksq]

No problem. Ask any time. Also feel free to ask through the Emails.