search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.68k stars 7.96k forks source link

Yolov4 darknet training issue when chaning the cfg file #8700

Open richard3333p opened 1 year ago

richard3333p commented 1 year ago

Hi,

I'd like to increase object detection performance using my custom dataset. So, I tried to increase the bach size and training images resolution in the cfg file.

In some cases, I got the "CUDA Error : out of Memory" when trying to train with the Tesla V100(32G) GPU x 4 and it has a lot of free space on GPU. Look at these cases. When I implemented trining Case2, I got the error.(Case1 is good) So, I changed the sub-divisions as 8, It worked.

I want to know the relationship between training and GPU capacity. Are there any rules for allocate the GPU's memory for image resolution, batchsize and sub-batchsize ? How to check the limit? Please give me some imfomation to utilize them.

Case1 : batchsize=32, sub-divisions : 4, resolution : 512x512 -> Allocate additional workspace_size = 555 MB Case2 : batchsize=32, sub-divisions : 4, resolution : 608x608 -> "CUDA Error" occur (Attached to screen shot) Case3 : batchsize=32, sub-divisions : 8, resolution : 608x608 -> Allocate additional workspace_size = 412.28 MB

Thanks for your reply in advance.

Selection_336

brucefay1115 commented 1 year ago

In the deep learning field, you must understand the training principle. For example if you have 1000 images for your dataset. We will not train one by one, but will random to take a batch of images from the entire dataset for training, which is called mini-batch. The mini-batch of calculate method as batchsize / sub-divisions, that represent how many images need to train in a batch. Each of batch will decide how many GPU memory will consume.

richard3333p commented 1 year ago

Thanks for your response. Of cource I know that... But, how can I expect how much memory is needed for my custom training configuration setting.