AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.75k stars 7.96k forks source link

GPU Memory leak ? free_network #4742

Open kossolax opened 4 years ago

kossolax commented 4 years ago

After doing "free_network" there are still memory used on nvidia-smi. Adding a loop will full-fill gpu then crash "out of memory".

for(int p=0; p<1000; p++) {

        network subnet = parse_network_cfg(cfgfile);
        if (weightfile) {
            load_weights(&subnet, weightfile);
        }

        *subnet.seen = 0;

        while ( *subnet.seen < train_images_num ) {

            pthread_join(load_thread, 0);
            train = buffer;
            load_thread = load_data(args);

            float loss = train_network_waitkey(subnet, train, 0);
            free_data(train);
        }

        int tmp = subnet.batch;

        set_batch_network(&subnet, 1);
        float map = validate_detector_map(datacfg, cfgfile, weightfile, 0.25, 0.5, 0, subnet.letter_box, &subnet);
    printf("%f", map);

        set_batch_network(&subnet, tmp);

        free_network(subnet);
}
AlexeyAB commented 4 years ago

It seems that this is fixed: https://github.com/AlexeyAB/darknet/issues/4841#issuecomment-631177987