AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.69k stars 7.96k forks source link

Weight file and GPU, are they relative each other? I think not. #2368

Open mrkieumy opened 5 years ago

mrkieumy commented 5 years ago

Dear all, First of all, thanks @AlexeyAB with the great work on this fork. I have a very strange question is: Does the weight file include GPU information? I think NO because I understand the structure of the weight file. But my situation like this: I used your darknet test on YoloV3 weight file, it runs normally, with this command: ./darknet detector map data/obj.data cfg/yolov3-obj.cfg yolov3-darknet.weights in 43 seconds, I have mAP. I saw that it used all of my GPU 99%. But I used the same command with another weight file: ./darknet detector map data/obj.data cfg/yolov3-obj.cfg yolov3-pytorch.weights It takes 1 hour to have mAP, I saw that it didn't use GPU, only around 17% of my GPU. yolov3-pytorch.weights is the weight file I trained on pytorch from this github: https://github.com/eriklindernoren/PyTorch-YOLOv3 I trained with the same everything (configuration and epochs, hyperparameters,...) The problem is not on yolov3-pytorch.weights file because I used pytorch test 2 those weight files, it can run normally and have result with the same time. This problem look likes in the weight file, it has a GPU information (like weight file trained from pytorch only uses CPU, weight file from darknet uses GPU), but I don't think it is. This is 2 my weight files, you can check that to see.(https://drive.google.com/drive/u/0/folders/1GTwjnep3FcHhmRIJSj3VDyrlVGgFNt3v) I trained with person detection, only 1 class. Do you know the reason why? I very appreciate some help. Best regards.

AlexeyAB commented 5 years ago

@mrkieumy Hi,

Can you rename your cfg file to the txt file and attach it to your message?

./darknet detector map data/obj.data cfg/yolov3-obj.cfg yolov3-pytorch.weights It takes 1 hour to have mAP, I saw that it didn't use GPU, only around 17% of my GPU.

This can be due too much false-positives, so do_nms_sort() function will work very slow, and CPU will be a bottleneck. I will check it.

mrkieumy commented 5 years ago

yolov3-obj.txt Hi @AlexeyAB , Thank you very much for your very fast reply, this is my cfg file.

@mrkieumy Hi,

Can you rename your cfg file to the txt file and attach it to your message?

./darknet detector map data/obj.data cfg/yolov3-obj.cfg yolov3-pytorch.weights It takes 1 hour to have mAP, I saw that it didn't use GPU, only around 17% of my GPU.

This can be due too much false-positives, so do_nms_sort() function will work very slow, and CPU will be a bottleneck. I will check it.

You're right, the possible problem is PyTorch source code training has some errors, so too much false-positives, because the map result is too low (14%) compare with darknet-weights file (45%). I also test mAP on that PyTorch source code, but the map was a lot lower comparing to your darknet.

mrkieumy commented 5 years ago

@AlexeyAB, You're really amazing, And another question is that: I would like to regularize the total loss by my custom loss function. For example, I want to implement MMD function (Maximum Mean Discrepancy) and add this loss to the total loss like (loss += MMD_loss). And to calculate MMD function, I want to get the feature map of the layer 80th. How can I save those feature map to a file on disk? And how can I add this MMD loss to the total loss? For PyTorch, I implemented that like, instead the module() function return only the sum(loss), I added return output_layer[80], and in training function I got that output layer, save to file to calculate MMD, and add to the loss before backward step. Is that right in your darknet, I have to modify the forward_network_gpu() function to return featuremap at layer 80th, and I have to modify train_network() function to get the activation to save the file? Can you give me some guide to easy to implement that. I really appreciate to your fast answer.