AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.66k stars 7.96k forks source link

multi train out of menory #5411

Open ax4688 opened 4 years ago

ax4688 commented 4 years ago

when i change detector.c like this:

else if (0 == strcmp(argv[2], "train")) {
    int _trycount = 0;
    printf("train\n");
    while (_trycount < 100) {
        _trycount++;
        train_detector(datacfg, cfg, weights, gpus, ngpus, clear, dont_show, calc_map, mjpeg_port, show_imgs, benchmark_layers, chart_path);
    }
}

and disable train_detector function[ while (get_current_iteration(net) < net.max_batches) ] at line(178),The next time my train, it will be prompted out of memory,how can I solve it?

ax4688 commented 4 years ago

it is the same mistake when use detect,change "darknet.c" at function main (line 489) like this int _trycount = 0; while (_trycount < 100) { _trycount++; float thresh = find_float_arg(argc, argv, "-thresh", .24); int ext_output = find_arg(argc, argv, "-ext_output"); char *filename = (argc > 4) ? argv[4] : 0; test_detector("cfg/coco.data", argv[2], argv[3], filename, thresh, 0.5, 0, ext_output, 0, NULL, 0, 0); } CUDA:10.2 compute_capability=610 OPENCV:4.3.0 CPU:I7-7700HQ GPU:GTX 1050TI(4G) CFG:yolov3_max.cfg batch=1 subdivisions=1 width=1856 height=1856 after 10 time it will be error: CUDA status Error: file: F:\project\cmake\darknet\source\src\dark_cuda.c : cuda_make_array() : line: 373 : build time: Apr 29 2020 - 18:21:24

CUDA Error: out of memory

CUDA Error: out of memory: No error