thtrieu / darkflow

Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices
GNU General Public License v3.0
6.13k stars 2.08k forks source link

Unable to train in GPU #490

Open vaishnavm217 opened 6 years ago

vaishnavm217 commented 6 years ago

Running the following command is giving me allocation error in GPU

./flow --model cfg/face-yolo.cfg --load yolo.weights --train --annotation annotationfolder --gpu 0.75

Using:

  1. Tensorflow 1.4.1
  2. K80 GPU (12 GB)
  3. Ubuntu 16.04
  4. yolo v2 608x608
  5. One class : face training on WIDERFACE dataset if I put gpu 1.0, it directly tries to allocate 11.17 GB rather than the required space.
vaishnavm217 commented 6 years ago

Assigning batch size 4 fixed the issue, but dunno why can't it train with the calculated batch size.

Ridhwanluthra commented 6 years ago

That is because when you give the option to use 1.0 that is all of the gpu then it allocates it all even if it doesnt need to use it! if you allocate less and need more then you face the problem of having less space than needed i hope this answered you question