only one GPU was occupied

mit-han-lab / bevfusion

[ICRA'23] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

https://bevfusion.mit.edu

Apache License 2.0

2.26k stars 409 forks source link

only one GPU was occupied #472

Closed wulinaaa closed 4 months ago

wulinaaa commented 1 year ago

When training, I run: torchpack dist-run -np 2 python tools/train.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth --load_from pretrained/lidar-only-det.pth --run-dir output/bev_result/ I looked at the GPU usage and found that only one GPU was occupied, how do I make the other piece also occupied.

quantumdotsss commented 1 year ago

I'm facing the same issue now. Have you solved this?

vram, I think

wulinaaa commented 1 year ago

我现在面临同样的问题。你解决了这个问题吗？

弗拉姆，我想

I think the memory was not freed when running torchpack before. I first looked at the GPU usage and found that there was a process that kept taking up memory, so I killed it. But killing it remains. Then I'll reboot just fine.