ryamldess
commented
4 years ago So I wrote a batch script to remove all of the images and associated labels from the dataset and attempted to re-train, being careful to edit the training document as well. Unfortunately, this now results in execution halting every time after the network is initialized and runs for a few seconds, while adding new files to the bad.list. It seems to be slowly corrupting the entire dataset. I'm going to start over re-downloading the entire dataset directly to the computer and unzipping it directly in the destination folder. I'm using a Java project called cocotoyolo.jar to generate the labels: [(https://bitbucket.org/yymoto/coco-to-yolo/src/master/)]
AlexeyAB
jackneil
Hello, I checked the forum for similar issues and the only one that seemed somewhat similar and related was this one:
https://github.com/AlexeyAB/darknet/issues/4119
...but ultimately it did not lead me to a solution. I am running darknet on a Windows 10 machine compiled without Python bindings but with MKL + TBB using Cmake-GUI (v3.16.0). I'm using OpenCV 4.1.2 and CUDA 10.2. Generation and compilation is with Visual Studio Community 2019.
Here are my specs dumped from Speccy during training, including environment vars:
Attempts to fix so far
I originally had 4 Titan Black GPUs in the machine and wanted to increase training speed on a budget, so I purchased 4 secondhand 980 Tis. After turning on the BIOS flag for 4G addressing and reinstalling graphics drivers, everything seemed to work great. I am training a custom dataset for a project but have not captured the data yet, although I have successfully trained a single test class in about 1300 iterations. I wanted to conduct a test to approximate training time for the full dataset, so I decided to train the MS Coco dataset. Unfortunately, after about 9,000 iterations, which took about 8 hours to train, it stopped with the error "CUDA Error: No Error". That's when I discovered issue #4119.
I realized I needed to recompile OpenCV with CUDA because the compute level for the Titan Black is only 3.5, while for the 980 Ti it is 5.2, so I did so. However, since then (I have run about 5 training sessions), it fails every time after about 10,500 iterations with the error: "CUDA Error: Out of memory".
I tried running it with cuda-memcheck, but it simply hangs after loading the network with the message "Loaded: 0.000000 seconds". I tried compiling gdb for Windows but ran into issues with that; I may try to come back to that later. I settled on attaching to the darknet process via the Visual Studio debugger after compiling the Debug versions of OpenCV and darknet and copying over the debug symbols. Unfortunately, as CUDA itself is not compiled with debug symbols, it didn't yield much better results; I was only able to verify a cudaStatus of 2, or cudaOutOfMemory. It also runs about 30% slower, so it takes the Debug version about 12 hours to get to this many iterations.
Here are what a couple of the exits looked like in the CLI:
Session 1:
Session 2:
Here is the top portion of the Makefile with which I compiled darknet:
I did not alter any values before that top section. CUDNN_HALF was 1, but I set it to 0 after reading #4119. My GPUs are definitely being utilized as verified in MSI Afterburner - and what is strange is that when the out of memory error occurs, no card is using much more than 2 GB of RAM, and they all have 6 GB of RAM... so memory really should not be an issue. When I rebuilt OpenCV, I did set CUDA_GENERATION to Auto so it would properly generate the correct ARCH, which it did.
Here's the output from Cmake-GUI for configuration and generation of OpenCV with CUDA, MKL & TBB:
...all projects built successfully in the VS solution.
Here's the Cmake-GUI output for darknet:
Again, all projects build successfully for both Debug and Release in the VS solution.
Here is the output of nvidia-smi:
...and here is the output of nvcc --version:
There are bad.list and bad_label.list files present. bad.list has 625 entries, bad_label.list is much shorter:
I notice these files are mentioned at the end of #4119 so I will look at that issue again and try to determine next steps - I guess removing those files and restarting?
I'm not sure what to try after that if that doesn't solve the issue. I'm crossing my fingers at this point that I will be able to train against my own dataset under 10,500 iterations, but it's quite likely I'll need more and I have a deadline of January 11th of having the network trained and inference working. The VS object dumps don't reveal anything special and the call stacks for the lines of code mentioned in the errors don't lead to any insights either:
cuda_free():
check_error():
Links including a few screenshots:
nvidia-smi.txt nvcc --version.txt application-exit_1.txt application-exit_2.txt check_error-stack_3.txt cuda_free_stack.txt