Tsinghua-MARS-Lab / ViP3D

MIT License
133 stars 19 forks source link

Error when samples_per_gpu>1 #6

Closed yichen928 closed 1 year ago

yichen928 commented 1 year ago

Thank you for your nice work!

I find that the training cannot run when setting samples_per_gpu>1 (line 231 of your configure file).

The error happens in line 440 of vips3d/models/attention_dert3d.py. I print the intermediate data and find this error comes from different shapes of img_meta['lidar2img'] in the same batch.

Can you kindly check your implementation? I believe it is important for running your code on large memory GPUs.

GentleSmile commented 1 year ago

Thanks for your suggestion. The code does not support multiple samples per gpu now. The whole pipeline is quite long and queries in each sample differ a lot, so we need query padding in each module, which is quite complicated. Moreover, memory of most gpus cannot support larger samples. If your gpu memory is large enough, we strongly recommend using ResNet101 for stronger performance, which will comsume more gpu memory.

yichen928 commented 1 year ago

Thank you for your kind reply!