volatile gpu-util=0% when training

HRNet / HRNet-Facial-Landmark-Detection

This is an official implementation of facial landmark detection for our TPAMI paper "Deep High-Resolution Representation Learning for Visual Recognition". https://arxiv.org/abs/1908.07919

MIT License

1.04k stars 263 forks source link

volatile gpu-util=0% when training #83

Open actor1996 opened 2 years ago

actor1996 commented 2 years ago

here are the settings in https://github.com/HRNet/HRNet-Facial-Landmark-Detection/blob/master/experiments/wflw/face_alignment_wflw_hrnet_w18.yaml

GPUS: (0,1,2,) WORKERS: 33 PRINT_FREQ: 3 ...

I didn't change other options

as showed in the picture above, the gpus 0,1,2 I used for train hrnet got 0% usage

How could that happen, is it because too much time is spent on processing data with the cpu

tianmaxingkong12 commented 2 years ago

Preprocessing takes too much time. It's just a snap by using nvidia-smi. when using watch , it 's not always zero.