search

sosppxo / MDIN

[MM2024 Oral] 3D-GRES: Generalized 3D Referring Expression Segmentation
26 stars 0 forks source link

About GPU Utility #9

Open HyperbolicCurve opened 1 month ago

HyperbolicCurve commented 1 month ago

Hi👋, when I was trying to train the MDIN on multi3dres whether on single GPU or multiple GPUs, I found the GPU utility was always very low, about 10%~20%. Even after I tried some tricks to accelerate the dataloader like moving the dataset to the memory and prefetching the batch with additional CUDA stream, the GPU utility was still 30%-40%. May I ask how to fix this problem?

sosppxo commented 3 weeks ago

Thank you for your attention! We have also observed the same issue of low GPU utilization. This may be due to the following reasons:

  1. Heavy data preprocessing (mainly data augmentation).
  2. Batching operations: To retain the full point cloud while enabling parallel processing, we batch point cloud data of varying lengths during training, which might take up some inference time.
  3. Sampling operations: When extracting sparse queries, farthest point sampling can also be time-consuming, which might reduce overall GPU utilization.

Of course, there may be additional factors as well. We would greatly appreciate it if you could contribute to optimizing this repository!

alala521 commented 2 weeks ago

Hi👋, when I was trying to train the MDIN on multi3dres whether on single GPU or multiple GPUs, I found the GPU utility was always very low, about 10%~20%. Even after I tried some tricks to accelerate the dataloader like moving the dataset to the memory and prefetching the batch with additional CUDA stream, the GPU utility was still 30%-40%. May I ask how to fix this problem?

Hi, I've noticed a significant drop in model performance when using multiple GPUs. Have you encountered this issue?