search

wzzheng / TPVFormer

[CVPR 2023] An academic alternative to Tesla's occupancy network for autonomous driving.
https://wzzheng.net/TPVFormer/
Apache License 2.0
1.19k stars 107 forks source link

How to apply TPVFormer to a different set of camera configurations? #43

Open WBB-USTC opened 1 year ago

WBB-USTC commented 1 year ago

Hi,

I'm trying to run TPVFormer on a vehicle with only 5 cameras for occupancy prediction, but the code is throwing size mismatch error while loading model weights:

" copying a param with shape torch.Size([6, 256]) from checkpoint, the shape in current model is torch.Size([5, 256])."

Any suggestions on how to bypass this problem, other than creating a dummy image with rgb values being 0?

Many thanks!

huang-yh commented 1 year ago

We think the param causing this problem is the cams_embeds in TPVFormerHead. https://github.com/wzzheng/TPVFormer/blob/bbed1886bb740c7e1ee87f9a899f2d9d6009cce1/tpvformer04/tpv_head.py#L53 Cams_embeds may encode information related to camera extrinsics and intrinsics. You might want to discard this param in your new model by commenting out relevant lines. Or you can train a new model with _numcams=5. https://github.com/wzzheng/TPVFormer/blob/bbed1886bb740c7e1ee87f9a899f2d9d6009cce1/config/tpv04_occupancy.py#LL27C1-L27C12