Nightmare-n
commented
3 months ago Hi, thanks for your interest.
The official training pipeline of uvtr-m is: scratch uvtr-c, scratch uvtr-l -> finetune uvtr-m, where the uvtr-m uses trained weights of scratch uvtr-c and scratch uvtr-l for initialization.
For our pre-training setting, the pipeline is: pretrain uvtr-c, pretrain uvtr-l -> finetune uvtr-c, finetune uvtr-l -> finetune uvtr-m, where the uvtr-m also uses trained weights of finetune uvtr-c and finetune uvtr-l for initialization.
There indeed exist other pretraining pipelines: pretrain uvtr-m -> finetune uvtr-m or pretrain uvtr-m -> finetune uvtr-c, finetune uvtr-l -> finetune uvtr-m, but it would cost more training memory and computational costs, thus we do not carefully verify these pipelines (these pipelines can be easily implemented by directly merging the config of uvtr_lidar_vs0.075_pretrain.py and uvtr_cam_vs0.075_pretrain.py).
hottruong98
Hi, thank you for your amazing work.
As indicated in your project config files, the weights for finetuning fusion-based model are got by "merging the weights of uvtr_lidar and uvtr_cam". So, from that statement, uvtr_lidar and uvtr_cam (pretraining phase) are trained individually before, and you just combine state_dicts of those 2 pretrained models to use for finetuning fusion-based model.
But when I first read your paper, I thought the fusion weights are got by training both lidar and camera branches "simultaneously" (as indicated in your framework), which is quite different from the above implementation.
This makes me confused and I want to ask you if I am understanding your ideas correctly. Is "camera-based pretrained weights + lidar-based pretrained weights = fusion-based pretrained weights" what you mean in your paper too?
Thank you so much in advance.