traininig extremely slow on windows server 2019

[sadimoodi]

I am having extreme slowness in training while on windows server 2019, with 5 GPUs (NVidia RTX 2070):

C:\Users\Administrator>nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:12:52_Pacific_Daylight_Time_2019 Cuda compilation tools, release 10.1, V10.1.243

my config file: [net]

Testing

batch=64 subdivisions=64

Training

batch=64

subdivisions=16

width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1

learning_rate=0.001 burn_in=1000 max_batches = 4000 policy=sgdr sgdr_cycle=1000 sgdr_mult=2 steps=4000,6000,8000,9000 scales=1, 1, 0.1, 0.1

the same EXACT configuration works perfectly well on another windows machine running windows 10, though all of my drivers are running well on both

[AlexeyAB]

• What cuDNN version do you use?
• What command do you use for training?
• What GPU usage do you see in GPU-Z?
• Try to use subdivisions=32

[sadimoodi]

• What cuDNN version do you use?

7.6.3

• What command do you use for training?

darknet.exe detector train data/obj.data CFG/yolo-obj.cfg darknet53.conv.74 -map

• What GPU usage do you see in GPU-Z?

GPU:2% CPU:100%

• Try to use subdivisions=32

i tried that but it didnt work.

I reinstalled everything (CUDA, CUNN), CMAKE), i recomplied darknet, i do NOT get any error, just VERY slow training and -map dialog box is freezing

the training command window is stuck here:

[AlexeyAB]

Windows task-manager shows wrong GPU usage. Use GPU-Z. It seems that you compiled Darknet without CUDA (GPU).

[sadimoodi]

this is how i see CMake whne i open it, CUDA isnt enabled by default: but when i compile i DO tick that box, but it seems like its not compiling with CUDA, see the log:

Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.17763. PThreads_windows_DLL_DIR: C:/darknet-master/darknet-master/3rdparty/pthreads/include/../bin OpenCV ARCH: x64 OpenCV RUNTIME: vc15 OpenCV STATIC: OFF Found OpenCV 4.1.1 in C:/opencv/build/x64/vc15/lib You might need to add C:\opencv\build\x64\vc15\bin to your PATH to be able to run your applications. ZED SDK not enabled, since it requires CUDA Configuring done

what am i missing? CUDA is already installed!

[AlexeyAB]

Yes, CUDA (GPU) isn't enabled. You must install CUDA after MSVS installation. Set CUDA checkbox, press Configure, Generate and compile.

[sadimoodi]

i found the problem, CMAKE-> delete cashe then redo everything!