search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.66k stars 7.96k forks source link

CUDA Error Prev: an illegal memory access was encountered #5075

Open vamsiduranc opened 4 years ago

vamsiduranc commented 4 years ago

We have downloaded latest master branch code and compiled darknet using Cmake-GUI. We are encountering an error "CUDA Error Prev: an illegal memory access was encountered" at specific interval of time. Can you please let us know how can we fix this issue?

Below are the details:

 7218: 0.624855, 0.829652 avg loss, 0.002000 rate, 1.766000 seconds, 923904 images
Resizing to initial size: 416 x 416  try to allocate additional workspace_size = 1245.71 MB
 CUDA allocate done!
 try to allocate additional workspace_size = 1245.71 MB
 CUDA allocate done!

 calculation mAP (mean average precision)...
4
 CUDA Error Prev: an illegal memory access was encountered

CUDA Error Prev: an illegal memory access was encountered: No error

Current Server Details: Operating System: Windows Server 2016 Processor: Intel Xeon E5-2690 v3 2.6GHz RAM: 112GB GPU Card: Tesla K80 - 2 Nos.

 CUDA-version: 10010 (10010), cuDNN: 7.6.5, GPU count: 2
 OpenCV version: 3.4.0
 compute_capability = 370, cudnn_half = 0
net.optimized_memory = 0

Please let us know if you need more information. Thanks in advance!

AlexeyAB commented 4 years ago

Try to train with -map flag with the latest Darknet version: https://github.com/AlexeyAB/darknet/commit/0ef5052ee51e82b2862fab5e9135b7bae060354f