Open kossolax opened 4 years ago
After doing "free_network" there are still memory used on nvidia-smi. Adding a loop will full-fill gpu then crash "out of memory".
for(int p=0; p<1000; p++) { network subnet = parse_network_cfg(cfgfile); if (weightfile) { load_weights(&subnet, weightfile); } *subnet.seen = 0; while ( *subnet.seen < train_images_num ) { pthread_join(load_thread, 0); train = buffer; load_thread = load_data(args); float loss = train_network_waitkey(subnet, train, 0); free_data(train); } int tmp = subnet.batch; set_batch_network(&subnet, 1); float map = validate_detector_map(datacfg, cfgfile, weightfile, 0.25, 0.5, 0, subnet.letter_box, &subnet); printf("%f", map); set_batch_network(&subnet, tmp); free_network(subnet); }
It seems that this is fixed: https://github.com/AlexeyAB/darknet/issues/4841#issuecomment-631177987
After doing "free_network" there are still memory used on nvidia-smi. Adding a loop will full-fill gpu then crash "out of memory".