Ubuntu (PopOS) Training Faster

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )

http://pjreddie.com/darknet/

Other

21.74k stars 7.96k forks source link

Ubuntu (PopOS) Training Faster #4574

Open BernoGreyling opened 4 years ago

BernoGreyling commented 4 years ago

Hi,

I'm running the same cfg file on the same setup on Windows 10 and PopOs 19.10 and finding that linux runs much faster than Windows 10. Is there a reason for this or a way that I can fix my Windows version? A training iteration on PopOS takes 16secs while on Windows it takes 21secs

The setup :

Intel 8750H 6 core laptop with a RTX 2070 Super connected over Thunderbolt 3. CUDNN_HALF enabled on both and both show "Tensor cores are used" while running

Opencv 4 on both not compiled with CUDA cfg : yolov3-spp width=736 height=1280 letter_box=1 mosaic=1

Thanks!

AlexeyAB commented 4 years ago

Try to download the latest Darknet and compile on Windows. There was fixed bug recently.

BernoGreyling commented 4 years ago

Hi @AlexeyAB ,

Tanks for the reply. Unfortunately, the new version made no difference. Been running on a repo that was just 4 days old though.

AlexeyAB commented 4 years ago

A training iteration on PopOS takes 16secs while on Windows it takes 21secs

Can you show screeshots?

BernoGreyling commented 4 years ago

This is from Windows

Annotation 2019-12-23 181608

And then From Linux

Screenshot from 2019-12-23 18-18-41

AlexeyAB commented 4 years ago

@BernoGreyling Do you use these 2 OS on the same 1 PC?

BernoGreyling commented 4 years ago

Yes.

The only difference is that Windows is installed on a nvme m.2 drive and PopOS is on a SATA m.2 drive. However, both the compiled directories are on the same SATA m.2 drive.

sealedtx commented 4 years ago

@BernoGreyling, is you dataset location same?

BernoGreyling commented 4 years ago

Hi @sealedtx, The data is on the same sata drive.

I'm seeing the same performance difference when performing detection on a video. Ubuntu being much faster.

BernoGreyling commented 4 years ago

Hi @AlexeyAB ,

It seems that compiling without cudnn on Windows does not make a difference so I'm assuming that it is not using cudnn properly?

I'm running CUDA 10.2 with cudnn 7.6.5. It is compiling with cudnn fine and all the library paths in Visual studio seems good. Is there a way to confirm that it is actually using the cudnn library or where the issue might lie?

AlexeyAB commented 4 years ago

@BernoGreyling Hi,

Do you see these 2 lines and CUDNN_HALF=1?

BernoGreyling commented 4 years ago

Hi, @AlexeyAB ,

Yup those are there.

SS1 SS2

AlexeyAB commented 4 years ago

@BernoGreyling

Is there a way to confirm that it is actually using the cudnn library or where the issue might lie?

CUDA + cuDNN are used in your case.

It seems that compiling without cudnn on Windows does not make a difference so I'm assuming that it is not using cudnn properly?

Darknet uses cuDNN on Windows and cuDNN accelerates Yolo. Why it doesn't work in your case, I don't know. Try to test on default yolov3-spp.cfg/weights files. Also try to download the latest Darknet version.

BernoGreyling commented 4 years ago

@AlexeyAB ,

Thanks. Appreciate the effort. Will fiddle around a bit more. Maybe redo all installations etc and report back in case someone else struggles with the same issue