How to visualize the semantic segmentation results on Bev？

[exiawsh]

Hello, It seems that You sampled 64 points for 5 different categories during data set preprocessing, but in fact, the semantic segmentation in Bev is a dense prediction task. For example, in a range of 200 * 200 pixels, 40000 points need to be sampled, so how to process and visualize the results in the inference stage?

[kashyap7x]

To visualize the results while driving along an evaluation route, you need to use neat_agent.py as the TEAM_AGENT and set a SAVE_PATH in run_evaluation.sh. The code used for generating the dense visualizations is here: https://github.com/autonomousvision/neat/blob/686f4a0b5b5bf20c99f323e9542f5b68808df2de/leaderboard/team_code/neat_agent.py#L312 To handle the large number of points, we process the image row-by-row for visualizations.

Alternatively, to visualize the predictions for images in a dataset using a trained NEAT model, you can use the offline evaluation script: https://github.com/autonomousvision/neat/blob/main/neat/eval.py

[exiawsh]

Does it means that the semantic segmentation on the BEV mainly appears as an auxiliary task in the Neat pipeline, and the semantic segmentation point by point(row by row) can not achieve real-time application? I want to have a deeper understanding of the research on semantic segmentation on BEV by using the neural attention field.

[kashyap7x]

Yes, you are correct, the BEV prediction is mainly an auxiliary task in our work. For the CARLA driving task, we do not need to generate dense semantic segmentation, so we only sample a small number of points to generate good waypoint predictions in real-time. Slow inference speed is a key problem of implicit neural representations like NeRF, and there are several hybrid explicit/implicit representations now being used so that inference can be made faster (eg. https://nvlabs.github.io/instant-ngp/)

[exiawsh]

Thank you for your patient reply. I'll check more related work. Looking forward to more exchanges with you.

[UditSinghParihar]

Hi @kashyap7x , thanks for your wonderful work on NEAT. I would like to ask regarding the blobby nature of BEV output as compared to other BEV methods like LSS or Fiery, where BEV of vehicles is quite sharp. Can you tell how we improve this blobby vehicle bev coming in NEAT?

Thanks