How to reproduce Figure3 in the paper?

[odroidodroid]

Hi, I'm trying to reproduce the figure3 in the paper.

In the paper, did you mean the latent rendering is key to make geometric feature from ray-wise feature?

When I look into the code, only the third BEVFormer encoder layer includes latent rendering.

Why does not last layer include latent rendering?

Also, after latent rendering, the channels still look like ray-wise feature.

I attached the picture of channels after latent rendering layer and end of the encoder layers.

thank you for your great works.

[tomztyang]

Hi, thanks for your attention!

So what feature maps are you visualizing and what checkpoint is used?

Please try to visualize the occ_path_prob_grids at here or visualize occ_path_embed at here.

I just forget which variables are used for visualization. Please give them both a try. Then you should be able to reproduce Fig.3 by the provided pre-trained weights.

Best, Zetong

[odroidodroid]

Thank you for your quick response.

The feature maps I visualized are a bev query of the last layer of encoder and a final embedding of latent rendering layer with r101_dcn_fcos3d_pretrain.pth and finetune-ViDAR-RN101-nus-full-1future.pth.

However, I couldn't reproduce the geometric features from occ_path_prob_grids and occ_path_embed, and I still have a question about the structure.

Why didn't you use latent rendering in the last layer but use it in the third layer? Also, why should I reproduce the intermediate embedding, which is not the final embedding in the latent rendering layer?

These are the one of channel of occ_path_prob_grids and occ_path_embed each, which line you mentioned.

[tomztyang]

Ohhh,

please visualize the occ_path_embed at here or occ_path_prob at here.

Use the checkpoint finetune-ViDAR-RN101-nus-full-1future.pth.

Some mistakes on yesterday's update, since I haven't touched this for a long time. Sorry for that.

BTW, it is not to visualize some intermediate embeddings but to visualize the embedding from LatentRendering.

The reason for using it in the third layer is to align our submission in CVPR, some old reasons, not in purpose (At that time, we do ablation on BEVFormer-small, but forget to modify when it comes to BEVFormer-base :-( ). You can try whether it works well when in the 6-th layer, I haven't tried it yet.

Best, Zetong

[tomztyang]

I close it for now. Feel free to re-open it for further questions!

Best, Zetong

[Wuqiman]

Ohhh,

please visualize the occ_path_embed at here or occ_path_prob at here.

Use the checkpoint finetune-ViDAR-RN101-nus-full-1future.pth.

Some mistakes on yesterday's update, since I haven't touched this for a long time. Sorry for that.

BTW, it is not to visualize some intermediate embeddings but to visualize the embedding from LatentRendering.

The reason for using it in the third layer is to align our submission in CVPR, some old reasons, not in purpose (At that time, we do ablation on BEVFormer-small, but forget to modify when it comes to BEVFormer-base :-( ). You can try whether it works well when in the 6-th layer, I haven't tried it yet.

Best, Zetong

可以分享下这两个变量的可视化脚本吗？我们做了可视化，发现没有论文里面贴的那么明显。