chungyiweng / humannerf

HumanNeRF turns a monocular video of moving people into a 360 free-viewpoint video.
MIT License
786 stars 86 forks source link

How to limit the gpu usage of humannerf training?? #56

Closed garrisonz closed 7 months ago

garrisonz commented 1 year ago

I would like to assign only gpus 0 and 1, or a single gpu, to humannerf for training. How can I do it ?

Try 1: run single_gpu.yaml

python train.py --cfg configs/human_nerf/zju_mocap/387/single_gpu.yaml

but, it still use all gpus instead of single one.

Try 2: I run single_gpu.yaml, and hard code all_gpus = [0, 1] in configs/config.py, and the output of humannerf train.py is

$python train.py --cfg configs/human_nerf/zju_mocap/387/single_gpu.yaml
------------------ GPU Configurations ------------------     
Primary GPUs: [0]                                            
Secondary GPUs: [1]                                          

but, actually, humannerf program(zhangyupeng) still occupy more than 2 gpus.

[0] Tesla P100-PCIE-16GB | 32°C,   0 % |  1725 / 16280 MB | zhangyupeng(1723M)
[1] Tesla P100-PCIE-16GB | 39°C,   0 % |  8057 / 16280 MB | zhangyupeng(8055M)
[2] Tesla P100-PCIE-16GB | 59°C, 100 % | 15047 / 16280 MB | tiiktak(1695M) tiiktak(12415M) zhangyupeng(911M)
[3] Tesla P100-PCIE-16GB | 35°C,   0 % |   937 / 16280 MB | zhangyupeng(935M)
[4] Tesla P100-PCIE-16GB | 47°C,  58 % | 13858 / 16280 MB | wangzihan(12857M) zhangyupeng(999M)
[5] Tesla P100-PCIE-16GB | 49°C,  35 % | 13678 / 16280 MB | wangzihan(12667M) zhangyupeng(1007M)
[6] Tesla P100-PCIE-16GB | 45°C,   0 % | 12641 / 16280 MB | wangzihan(12639M)
[7] Tesla P100-PCIE-16GB | 47°C,   0 % | 12639 / 16280 MB | wangzihan(12637M)

How to limit the gpu usage of humannerf training?? Thanks

hi-zhangjie commented 1 year ago

set environment variable like this,export CUDA_VISIBLE_DEVICES=0,1,2,3 May i have your Email,I also try to reproduce the repo

garrisonz commented 7 months ago

I have solved it by setting visiblility of the gpu in python code.