search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.57k stars 7.95k forks source link

CUDA Graphs - +20% Detection faster on GPU (CUDA >= 10.1) #7444

Open AlexeyAB opened 3 years ago

AlexeyAB commented 3 years ago

I added CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/ So now you can add to any cfg-file:

[net]
use_cuda_graph = 1

and Detection will be +20% faster on GPU (starting from the 2nd frame).

It requires CUDA >= 10.1

GPU RTX 3090, Windows 10 x64, CUDA 11.2, cuDNN 8.1.0, CUDNN_HALF=1, OpenCV 4.5.2, CPU Intel Core i7 6700K (4 Cores / 8 HT)

Command: darknet.exe detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights test.mp4 -benchmark

YOLOv4:

YOLOv4-CSP (Scaled-YOLOv4):

GPU RTX 3090, Windows 7 x64, CUDA 11.1, cuDNN 8.0.5, CUDNN_HALF=1, OpenCV 4.2.0, CPU Intel Core i7 6700K (4 Cores / 8 HT) nVidia Driver 475.30

Command: darknet.exe detector demo cfg/coco.data cfg/yolov4.cfg yolov4.weights test.mp4 -benchmark

YOLOv4:

YOLOv4-CSP (Scaled-YOLOv4):

stephanecharette commented 3 years ago

Thanks, Alexey.

Does this apply to every configuration type? (v3, v4, tiny, etc?)

And should we always set use_cuda_graph=1 or is there a criteria we should follow to determine if/when it should be used?

AlexeyAB commented 3 years ago

It is suitable for all cfg-files. Just it isn't well tested, especially for training.

stephanecharette commented 3 years ago

Ran some tests tonight on my Ubuntu 18.04 server rig. Network is yolov4-tiny.cfg, 416x416, custom network with single class trained with max_batches=10K. This network was trained about 1 week ago. The only changes to the .cfg are the standard ones described in the readme related to max_batches, steps, filters, and classes.

When I run with use_cuda_graph=1 it takes my RTX2070 an average of 11.84 seconds to run inference across 2273 images pre-sized at exactly 416x416.

When I run the same test but with use_cuda_graph=0 it takes an average of 11.958 seconds.

So while there is a small increase in performance, this means I've gone from 190 FPS to 192 FPS, a modest increase of ~1%.

The command I ran was /usr/bin/time --verbose DarkHelp -j -l ~/nn/fire/fire_train.txt ~/nn/fire/fire{.cfg,.names,_best.weights}

To rule out any strange behaviour from DarkHelp, I then switched to calling darknet directly with this command: /usr/bin/time --verbose ./darknet detector test ~/nn/fire/fire{.data,.cfg,_best.weights} -ext_output -dont_show -out ~/results.json < ~/nn/fire/fire_train.txt.

With that command, the average when use_cuda_graph=1 was 26.048 seconds, while =0 was 26.282 seconds. Again, the numbers are so close it doesn't make much of a difference: 87.3 FPS vs 86.5 FPS.

The version of darknet I'm running is this one:

> git log -1
commit b8c9c9d457a47d27710082c6e16206fc50af21f3 (HEAD -> master, origin/master, origin/HEAD)
Author: AlexeyAB <...@mail.ru>
Date:   Mon Mar 1 05:44:04 2021 +0300

    minor fix

CUDA is 11.2.1-1, CUDNN is 8.1.0.77. Darknet's makefile has these modifications:

Is there something else I need to do, or is there something you can think of that would describe why I don't see much difference regardless of how this flag is set?

bernorev commented 3 years ago

Hi,

Thanks for the addition!

Also seeing quite similar results with a 3070

erikguo commented 3 years ago

Hi @AlexeyAB ,

When I updated the source code to latest version to use this feature, I found one problem:

  1. When calling from darknet.py with use_cuda_graph=1 cfg, it will consume almost double GPU memory. This so strange.

I compile the source code with CUDA 11.2

Thanx

stephanecharette commented 3 years ago

Tried to train my first network tonight with the new option use_cuda_graph=1 but ran into an error at iteration #1000 when it was time to do the first mAP calculations:

 (next mAP calculation at 1000 iterations) 
 1000: 1.620506, 1.410898 avg loss, 0.002610 rate, 0.854260 seconds, 64000 images, 2.028033 hours left
4cuDNN Error: CUDNN_STATUS_NOT_INITIALIZED: Success
darknet: ./src/utils.c:331: error: Assertion `0' failed.

Restarted the training with use_cuda_graph=0 and the error has gone away.

elias-work commented 3 years ago

On CUDA 11.2 I, too, see a very modest performance increase (<2%). I am grateful for any improvement but is there a preventable cause for the disparity in results?

Fetulhak commented 3 years ago

Hi @AlexeyAB , In your latest version of the repo I got into trouble and need help. In the previous version, it works fine training both YOLOV3 and YOLOV4 on my custom dataset with very small object detection. When I use your current repo it is showing -nan for average loss within the first 1-30 epochs. I do not know the detail of the source code how to fix it?

here is the sample output I get from both the previous and current versions of your repo. I have used the same dataset which is prepared carefully. When I use your current repo the average loss becomes -nan even at the beginning of the training.

Previous version output: CUDA-version: 8000 (11020), GPU count: 1
OpenCV version: 3.2.0 0 : compute_capability = 750, cudnn_half = 0, GPU: Tesla T4 layer filters size/strd(dil) input output 0 conv 32 3 x 3/ 1 608 x 608 x 3 -> 608 x 608 x 32 0.639 BF 1 conv 64 3 x 3/ 2 608 x 608 x 32 -> 304 x 304 x 64 3.407 BF ......

[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00 Total BFLOPS 173.535 avg_outputs = 1351435 Allocate additional workspace_size = 106.46 MB Loading weights from build/darknet/x64/darknet53.conv.74... Prepare additional network for mAP calculation... net.optimized_memory = 0 mini_batch = 1, batch = 32, time_steps = 1, train = 0 yolov3 net.optimized_memory = 0 mini_batch = 2, batch = 64, time_steps = 1, train = 1 Done! Loaded 75 layers from weights-file Create 6 permanent cpu-threads v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.449484, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 508.443695, iou_loss = 0.000000, total_loss = 508.443695 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485419, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2344.686035, iou_loss = 0.000000, total_loss = 2344.686035 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.055346, GIOU: 0.041472), Class: 0.484173, Obj: 0.629255, No Obj: 0.584863, .5R: 0.000000, .75R: 0.000000, count: 90, class_loss = 53192.523438, iou_loss = 784.425781, total_loss = 53976.949219 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.448548, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 507.361084, iou_loss = 0.000000, total_loss = 507.361084 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485286, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2341.652588, iou_loss = 0.000000, total_loss = 2341.652588 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.068896, GIOU: 0.065986), Class: 0.514756, Obj: 0.578799, No Obj: 0.585964, .5R: 0.010638, .75R: 0.000000, count: 94, class_loss = 53402.976562, iou_loss = 696.402344, total_loss = 54099.378906 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.448517, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 507.130219, iou_loss = 0.000000, total_loss = 507.130219 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.484696, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2345.052734, iou_loss = 0.000000, total_loss = 2345.052734 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.089818, GIOU: 0.088995), Class: 0.495767, Obj: 0.601108, No Obj: 0.586660, .5R: 0.000000, .75R: 0.000000, count: 94, class_loss = 53530.812500, iou_loss = 530.648438, total_loss = 54061.460938

......

seen 64, trained: 0 K-images (0 Kilo-batches_64) Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005 Resizing, random_coef = 1.40

896 x 896 try to allocate additional workspace_size = 231.21 MB CUDA allocate done! Loaded: 6.192834 seconds

(next mAP calculation at 1000 iterations) 1: 19033.039062, 19033.039062 avg loss, 0.000000 rate, 150.682134 seconds, 64 images, -1.000000 hours left v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.449121, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 509.070679, iou_loss = 0.000000, total_loss = 509.070679 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485577, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2350.911865, iou_loss = 0.000000, total_loss = 2350.911865 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.063431, GIOU: 0.061033), Class: 0.480117, Obj: 0.635891, No Obj: 0.586292, .5R: 0.015385, .75R: 0.000000, count: 65, class_loss = 53426.765625, iou_loss = 514.140625, total_loss = 53940.906250 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.449470, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 510.548035, iou_loss = 0.000000, total_loss = 510.548035 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.484945, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2338.005371, iou_loss = 0.000000, total_loss = 2338.005371 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.069151, GIOU: 0.032462), Class: 0.423765, Obj: 0.563249, No Obj: 0.584826, .5R: 0.000000, .75R: 0.000000, count: 20, class_loss = 53192.472656, iou_loss = 180.625000, total_loss = 53373.097656 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.449060, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 508.103943, iou_loss = 0.000000, total_loss = 508.103943 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.484962, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 2346.081055, iou_loss = 0.000000, total_loss = 2346.081055

.......

(next mAP calculation at 1000 iterations) 14: 6427.955566, 14699.434570 avg loss, 0.000000 rate, 24.729473 seconds, 896 images, 162.151322 hours left v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.448834, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 165.781021, iou_loss = 0.000000, total_loss = 165.781021 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.484807, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 764.631104, iou_loss = 0.000000, total_loss = 764.631104 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.026335, GIOU: 0.026335), Class: 0.510651, Obj: 0.611109, No Obj: 0.590604, .5R: 0.000000, .75R: 0.000000, count: 22, class_loss = 17737.009766, iou_loss = 247.255859, total_loss = 17984.265625 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.447847, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 164.905609, iou_loss = 0.000000, total_loss = 164.905609 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485121, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 768.601807, iou_loss = 0.000000, total_loss = 768.601807 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.053705, GIOU: 0.039216), Class: 0.514480, Obj: 0.590892, No Obj: 0.592018, .5R: 0.000000, .75R: 0.000000, count: 31, class_loss = 17825.955078, iou_loss = 248.896484, total_loss = 18074.851562 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.447277, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 164.479248, iou_loss = 0.000000, total_loss = 164.479248 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485849, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 767.477783, iou_loss = 0.000000, total_loss = 767.477783 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.025991, GIOU: 0.025482), Class: 0.490552, Obj: 0.574159, No Obj: 0.591296, .5R: 0.000000, .75R: 0.000000, count: 100, class_loss = 17785.001953, iou_loss = 1045.468750, total_loss = 18830.470703 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.446812, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 163.261032, iou_loss = 0.000000, total_loss = 163.261032 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485050, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 767.719910, iou_loss = 0.000000, total_loss = 767.719910 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.035148, GIOU: 0.031677), Class: 0.522515, Obj: 0.587202, No Obj: 0.590659, .5R: 0.000000, .75R: 0.000000, count: 96, class_loss = 17747.035156, iou_loss = 907.103516, total_loss = 18654.138672

..........

(next mAP calculation at 1000 iterations) 20: 6430.400391, 10817.592773 avg loss, 0.000000 rate, 25.639439 seconds, 1280 images, 154.295924 hours left v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.446967, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 231.357376, iou_loss = 0.000000, total_loss = 231.357376 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485143, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 1079.028076, iou_loss = 0.000000, total_loss = 1079.028076 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.042310, GIOU: 0.042310), Class: 0.447892, Obj: 0.592023, No Obj: 0.587610, .5R: 0.000000, .75R: 0.000000, count: 15, class_loss = 24730.488281, iou_loss = 131.875000, total_loss = 24862.363281 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.447886, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 232.699631, iou_loss = 0.000000, total_loss = 232.699631 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485382, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 1083.580444, iou_loss = 0.000000, total_loss = 1083.580444 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.034265, GIOU: 0.034265), Class: 0.494861, Obj: 0.525071, No Obj: 0.588013, .5R: 0.000000, .75R: 0.000000, count: 25, class_loss = 24778.859375, iou_loss = 240.267578, total_loss = 25019.126953 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.447387, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 232.355560, iou_loss = 0.000000, total_loss = 232.355560 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485086, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 1082.332397, iou_loss = 0.000000, total_loss = 1082.332397 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU: 0.045536, GIOU: 0.043708), Class: 0.508552, Obj: 0.559455, No Obj: 0.589944, .5R: 0.000000, .75R: 0.000000, count: 74, class_loss = 24959.287109, iou_loss = 607.191406, total_loss = 25566.478516 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.446765, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 231.257843, iou_loss = 0.000000, total_loss = 231.257843 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.485479, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 1083.417847, iou_loss = 0.000000, total_loss = 1083.417847 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 106 Avg (IOU:

Current repo output: CUDA-version: 11000 (11020), GPU count: 1
OpenCV version: 3.2.0 0 : compute_capability = 750, cudnn_half = 0, GPU: Tesla T4 layer filters size/strd(dil) input output 0 conv 32 3 x 3/ 1 608 x 608 x 3 -> 608 x 608 x 32 0.639 BF

Allocate additional workspace_size = 106.46 MB Loading weights from build/darknet/x64/yolov4.conv.137... Prepare additional network for mAP calculation... net.optimized_memory = 0 mini_batch = 1, batch = 32, time_steps = 1, train = 0 Create CUDA-stream - 0 nms_kind: greedynms (1), beta = 0.600000 nms_kind: greedynms (1), beta = 0.600000 nms_kind: greedynms (1), beta = 0.600000 yolo-obj net.optimized_memory = 0 mini_batch = 2, batch = 64, time_steps = 1, train = 1 nms_kind: greedynms (1), beta = 0.600000 nms_kind: greedynms (1), beta = 0.600000 nms_kind: greedynms (1), beta = 0.600000 Done! Loaded 137 layers from weights-file Create 6 permanent cpu-threads v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.452500), count: 270, class_loss = 38072.460938, iou_loss = 4615.625000, total_loss = 42688.085938 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 276, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 283, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 829, rewritten_bbox = 3.980700 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.520526), count: 156, class_loss = 38011.753906, iou_loss = 3132.019531, total_loss = 41143.773438 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 159, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 175, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 1319, rewritten_bbox = 3.866565 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 1, class_loss = 37913.054688, iou_loss = 0.000000, total_loss = 37913.054688 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 1319, rewritten_bbox = 3.866565 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.369757), count: 314, class_loss = 37723.484375, iou_loss = 3829.695312, total_loss = 41553.179688 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 338, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 394, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 2365, rewritten_bbox = 3.932347 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.579773), count: 30, class_loss = 37683.835938, iou_loss = 685.660156, total_loss = 38369.496094 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 30, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 37, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 2462, rewritten_bbox = 3.777417 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.252727), count: 6, class_loss = 37660.289062, iou_loss = 26.839844, total_loss = 37687.128906 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 7, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 29, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 2504, rewritten_bbox = 3.714057 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.289301), count: 196, class_loss = 37635.343750, iou_loss = 1382.359375, total_loss = 39017.703125 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 302, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 458, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 3460, rewritten_bbox = 3.526011 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.299082), count: 70, class_loss = 37565.480469, iou_loss = 524.734375, total_loss = 38090.214844 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 100, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 130, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 3760, rewritten_bbox = 3.244681 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.332979), count: 67, class_loss = 37654.410156, iou_loss = 660.062500, total_loss = 38314.472656 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 83, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 163, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 4073, rewritten_bbox = 2.995335 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.551592), count: 253, class_loss = 37557.203125, iou_loss = 5326.710938, total_loss = 42883.914062 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 256, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 275, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 4857, rewritten_bbox = 4.488367 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.273497), count: 51, class_loss = 37497.757812, iou_loss = 284.738281, total_loss = 37782.496094 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 75, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 169, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 5152, rewritten_bbox = 4.328416 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.410402), count: 24, class_loss = 37483.183594, iou_loss = 348.863281, total_loss = 37832.046875 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 24, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 24, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 5224, rewritten_bbox = 4.268760 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.300087), count: 9, class_loss = 37496.585938, iou_loss = 73.386719, total_loss = 37569.972656 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 12, class_loss = -nan, iou_loss = -nan, total_loss = -nan

AlexeyAB commented 3 years ago

@Fetulhak

Fetulhak commented 3 years ago

Hi @AlexeyAB, No I dont use [net] use_cuda_graph = 1

Should I have to put these on the first two lines?

incase here are my .cfg files I am using for YOLOV3 and YOLOV4:

for YOLOV3: yolov3-custom.txt

Fetulhak commented 3 years ago

Hi @AlexeyAB I have included the use_cuda_graph = 1 in my .cfg file as you suggested but the problem still persists.

here is the output of my training after including use_cuda_graph = 1. As you can see the avr_loss is nan.

......... total_bbox = 23228, rewritten_bbox = 7.189599 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 398, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 399, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.134002), count: 399, class_loss = 310.455994, iou_loss = 926.203918, total_loss = 1236.659912 total_bbox = 24424, rewritten_bbox = 7.169178 % Loaded: 0.000047 seconds

(next mAP calculation at 1000 iterations) 2: -nan, -nan avg loss, 0.000000 rate, 34.954638 seconds, 128 images, 32.050234 hours left v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 208, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 220, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.152179), count: 230, class_loss = 299.755157, iou_loss = 451.484528, total_loss = 751.239685 total_bbox = 25082, rewritten_bbox = 7.112670 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 275, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 306, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.150717), count: 315, class_loss = 305.569916, iou_loss = 500.240753, total_loss = 805.810669 total_bbox = 25978, rewritten_bbox = 7.075218 % ..........

Fetulhak commented 3 years ago

When I train YOLOV4 the training log goes from iteration number 2 with the following output where some of the numbers are not -nan

2: -nan, -nan avg loss, 0.000000 rate, 34.954638 seconds, 128 images, 32.050234 hours left v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 208, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 220, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.152179), count: 230, class_loss = 299.755157, iou_loss = 451.484528, total_loss = 751.239685

to iteration number 147 where all the numbers become -nan

147: -nan, -nan avg loss, 0.000000 rate, 23.773124 seconds, 9408 images, 20.681359 hours left v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 152, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 150, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 141, class_loss = -nan, iou_loss = -nan, total_loss = -nan total_bbox = 2658827, rewritten_bbox = 11.277079 %

Does anyone have any idea what the solution will be?

Fetulhak commented 3 years ago

HI @AlexeyAB why is the current version of your repo makes the average loss differently calculated for your current repo and previous repo?

Previous repo output:

(next mAP calculation at 1000 iterations) 559: 14.733596, 17.242596 avg loss, 0.000098 rate, 41.046122 seconds, 35776 images, 26.555945 hours left v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 82 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.000161, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 0.000087, iou_loss = 0.000000, total_loss = 0.000087 v3 (mse loss, Normalizer: (iou: 0.75, cls: 1.00) Region 94 Avg (IOU: 0.000000, GIOU: 0.000000), Class: 0.000000, Obj: 0.000000, No Obj: 0.000077, .5R: 0.000000, .75R: 0.000000, count: 1, class_loss = 0.000089, iou_loss = 0.000000, total_loss = 0.000089

current repo output:

2: -nan, -nan avg loss, 0.000000 rate, 34.954638 seconds, 128 images, 32.050234 hours left v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.000000), count: 208, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 220, class_loss = -nan, iou_loss = -nan, total_loss = -nan v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.152179), count: 230, class_loss = 299.755157, iou_loss = 451.484528, total_loss = 751.239685

folkien commented 3 years ago

My settings : 2xRTX2060, I7-6700K, ubuntu 18.04 LTS, CUDA 11.2, LIBCUDNN 8.2.0.53 I tried this option, 

[net]
use_cuda_graph = 1

but every program run ends in :

 Create stream 0 
 Create cudnn-handle 0 
 Create stream 1 
 Create cudnn-handle 1 
 Create stream 2 
 Create cudnn-handle 2 
 Create stream 3 
 Create cudnn-handle 3 
 Create stream 4 
 Create cudnn-handle 4 
 Create stream 5 
 Create cudnn-handle 5 
 Create stream 6 
 Create cudnn-handle 6 
 Create stream 7 
 Create cudnn-handle 7 
 Create stream 8 
 Create cudnn-handle 8 
 Create stream 9 
 Create cudnn-handle 9 
 Create stream 10 
 Create cudnn-handle 10 
 Create stream 11 
 Create cudnn-handle 11 
 Create stream 12 
 Create cudnn-handle 12 
 Create stream 13 

 cuDNN status Error in: file: ./src/dark_cuda.c : () : line: 295 : build time: Jun  4 2021 - 09:20:46 
 Create cudnn-handle 13 
 Create stream 14 

 cuDNN Error: CUDNN_STATUS_MAPPING_ERROR
python3: : Unknown error -725824958
 Create cudnn-handle 14 
 Create stream 15 
 Create cudnn-handle 15 
Try to capture graph... 

 cuDNN status Error in: file: ./src/convolutional_kernels.cu : () : line: 553 : build time: Jun  4 2021 - 09:20:57 

 cuDNN Error: CUDNN_STATUS_EXECUTION_FAILED
python3: : Unknown error -725824958
free(): corrupted unsorted chunks
arnaud-nt2i commented 2 years ago

@stephanecharette @AlexeyAB Hi ! Has the issue with cuda_graph being solved for training yet ?

stephanecharette commented 2 years ago

I've not tried. As far as I know it has not been fixed.

arnaud-nt2i commented 2 years ago

I've not tried. As far as I know it has not been fixed.

thanks