search

PJLab-ADG / 3DTrans

An open-source codebase for exploring autonomous driving pre-training
https://bobrown.github.io/Team_3DTrans.github.io/
Apache License 2.0
585 stars 72 forks source link

A problem about GPUs . #6

Closed NoName5990 closed 1 year ago

NoName5990 commented 1 year ago

Hello, I met a problem when I ran your program, I am using kitti and nuscenes-mini datasets, but when I run the source-only commandsh scripts/dist_train.sh 8 --cfg_file ./cfgs/DA/nusc_kitti/source_only/pvrcnn_old_anchor_sn_kitti.yamlmy 8 NVIDIA RTX A5000 GPUs have reached 100% utilization, but the memory is used less than half, no error messages and nothing happens，I adjusted BATCH_SIZE_PER_GPU=1, but still does not work.

BOBrown commented 1 year ago

Hello, I am not sure I understand your question. You mean that the used GPU memory is less than half of the total? The GPU memory utilization is related to the BATCH_SIZE you used. Try to set BATCH_SIZE_PER_GPU=8

NoName5990 commented 1 year ago

Sorry ,I find the main cause is Ceph, I want to ask whether I have to install Ceph?

BOBrown commented 1 year ago

Ceph is not a necessary thing for running the 3DTrans, you could dsiable the Ceph by changing the setting in yaml