Open sadimoodi opened 4 years ago
- What cuDNN version do you use?
7.6.3
- What command do you use for training?
darknet.exe detector train data/obj.data CFG/yolo-obj.cfg darknet53.conv.74 -map
- What GPU usage do you see in GPU-Z?
GPU:2% CPU:100%
- Try to use subdivisions=32
i tried that but it didnt work.
I reinstalled everything (CUDA, CUNN), CMAKE), i recomplied darknet, i do NOT get any error, just VERY slow training and -map dialog box is freezing
the training command window is stuck here:
Windows task-manager shows wrong GPU usage. Use GPU-Z. It seems that you compiled Darknet without CUDA (GPU).
this is how i see CMake whne i open it, CUDA isnt enabled by default:
but when i compile i DO tick that box, but it seems like its not compiling with CUDA, see the log:
Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.17763. PThreads_windows_DLL_DIR: C:/darknet-master/darknet-master/3rdparty/pthreads/include/../bin OpenCV ARCH: x64 OpenCV RUNTIME: vc15 OpenCV STATIC: OFF Found OpenCV 4.1.1 in C:/opencv/build/x64/vc15/lib You might need to add C:\opencv\build\x64\vc15\bin to your PATH to be able to run your applications. ZED SDK not enabled, since it requires CUDA Configuring done
what am i missing? CUDA is already installed!
Yes, CUDA (GPU) isn't enabled. You must install CUDA after MSVS installation. Set CUDA checkbox, press Configure, Generate and compile.
i found the problem, CMAKE-> delete cashe then redo everything!
I am having extreme slowness in training while on windows server 2019, with 5 GPUs (NVidia RTX 2070):
C:\Users\Administrator>nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:12:52_Pacific_Daylight_Time_2019 Cuda compilation tools, release 10.1, V10.1.243
my config file: [net]
Testing
batch=64 subdivisions=64
Training
batch=64
subdivisions=16
width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1
learning_rate=0.001 burn_in=1000 max_batches = 4000 policy=sgdr sgdr_cycle=1000 sgdr_mult=2 steps=4000,6000,8000,9000 scales=1, 1, 0.1, 0.1
the same EXACT configuration works perfectly well on another windows machine running windows 10, though all of my drivers are running well on both