search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.68k stars 7.96k forks source link

FEATURE REQUEST - GRAD CAM FOR YOLO #7306

Open holger-prause opened 3 years ago

holger-prause commented 3 years ago

Hello,

For debugging a model its a very userfull to show a heatmap of the classactivations. This is described in paper https://arxiv.org/pdf/1610.02391v1.pdf

This would be very usefull imho.

AlexeyAB commented 3 years ago

https://github.com/AlexeyAB/darknet/issues/5117 Set in your cfg-file and run training:

[net] 
adversarial_lr=0.05 
attention=1
holger-prause commented 3 years ago

Thank you very much :-) Do i need to specify -show_imgs and will this produce a heatmap? Feel free to decline this feature request when its already there :-) Probaply think about putting this into the documentation.

holger-prause commented 3 years ago

Oh i am stupid - i just have to read the linked issue https://github.com/AlexeyAB/darknet/issues/5117 - its all there! Thank you very much again - yolo is awesome!

holger-prause commented 3 years ago

@AlexeyAB My goal is to visualize the classactivation for a certain class - even the classactivationmap for all classes would be fine. Before testing my own model - i want to try out on sample image dog.jpg and the coco weights (yolov4.weights) and try to get the attention visualization.

I created a small dataset containing the dog.jpg sample image. I dont created any annotation for this sample.

After this i set the mentioned config parameter into the net section

    [net] 
    adversarial_lr=0.05 
    attention=1

After this is started training with - darknet_no_gpu.exe detector train C:\development\datasets\gradcamtest\train.data C:\development\datasets\gradcamtest\yolov4.cfg C:\development\datasets\gradcamtest\yolov4.weights -show_imgs

After this - i see a bunch auf augemented images - but none of them contains any kind of heatmap or attention visualization. What am i doing wrong (should i pick yolov4.conv.137?, do i need to picked annotated images?) ?

Thank you very much for your patience - i did some trainings before - for me this is not really straighforward and any help is highly welcome.

AlexeyAB commented 3 years ago
holger-prause commented 3 years ago

@AlexeyAB Ok i did the suggest changes and removed -show_imgs and added the burn in - the console output is Saving weights to C:\development\datasets\gradcamtest\backup/yolov4_final.weights

But i still dont see any attention image generated(i checked dataset backup and darknet folder). Note that i use the no gpu version. I also compiled yolo with opencv support. Am i missing something very obvious - do i need to annotate my sample image(i want to get the attention visualization for the sample image dog.jpg contained in the yolo folder)?

AlexeyAB commented 3 years ago

It supports on GPU. It isn't implemented for CPU.

holger-prause commented 3 years ago

@AlexeyAB Ah ok - i was assuming something like this - fair enough, i got a new pc on work and it has no cuda gpu... I train in cloud.

Can i please summary(my understanding) what i need to do to get the attention image: First of all: Make sure you use gpu (cuda) variant of yolo framework

AND also 

    burn_in=0

Now i can only guess:

Is this correct? I am not sure about step4 - does not make much sense to me as grad cam used guided backward propagation (i guess this why we need to train) Thank you again very much for your time and patience.

AlexeyAB commented 3 years ago
  1. Train the model or use already trained cfg/weights files
  2. Change in cfg [net] adversarial_lr=0.05 attention=1 burn_in=0
  3. Run training on GPU with these cfg/weights files (you can tune adversarial_lr=0.05) and with flag -clear at the end of training command
    • just an additional note: it will generate new weights in the /backup/ directory by using SAT (self-adversarial trainin), it can be better if model(yolov4) has higher capacity than dataset(ua-detrac), and can be worse for model(yolov4) has lower capacity than dataset(mscoco)
holger-prause commented 3 years ago

@AlexeyAB OMG its finally working - i tried out on another computer(which has gpu version of yolo), followed all the steps and i had to use -clear for some reason. I made sure it is training for at least some batches. I also made sure my object is annotated so the weights do not change too much.

Next thing would be to get attention for only one specific class.

bjajoh commented 3 years ago

@holger-prause I'm not sure if I'm understanding it correctly, but is there a working Grad Cam implementation to give me the heat map?

jojo0513 commented 2 years ago
  1. Train the model or use already trained cfg/weights files
  2. Change in cfg [net] adversarial_lr=0.05 attention=1 burn_in=0

  3. Run training on GPU with these cfg/weights files (you can tune adversarial_lr=0.05) and with flag -clear at the end of training command

    • just an additional note: it will generate new weights in the /backup/ directory by using SAT (self-adversarial trainin), it can be better if model(yolov4) has higher capacity than dataset(ua-detrac), and can be worse for model(yolov4) has lower capacity than dataset(mscoco)

Hi Alexey, I do what you said and without -flag, the following error raised.

"[yolo] params: iou loss: ciou (4), iou_norm: 0.07, obj_norm: 1.00, cls_norm: 1.00, delta_norm: 1.00, scale_x_y: 1.05 nms_kind: greedynms (1), beta = 0.600000 Total BFLOPS 59.614 avg_outputs = 490698 Allocate additional workspace_size = 111.05 MB Loading weights from /mydrive/yolov4/backup/2021_12_21_weight_with_8_species.weights... seen 64, trained: 819 K-images (12 Kilo-batches_64) Done! Loaded 162 layers from weights-file Learning Rate: 0.001, Momentum: 0.949, Decay: 0.0005 Detection layer: 139 - type = 28 Detection layer: 150 - type = 28 Detection layer: 161 - type = 28 If error occurs - run training with flag: -dont_show Unable to init server: Could not connect: Connection refused

(chart_yolov4-obj.png:2416): Gtk-WARNING **: 09:45:31.916: cannot open display: "

Do you have any idea what happen?

stephanecharette commented 2 years ago

Did you try to search for that error?

https://github.com/AlexeyAB/darknet/issues?q=cannot+open+display

jojo0513 commented 2 years ago

Did you try to search for that error?

https://github.com/AlexeyAB/darknet/issues?q=cannot+open+display

I did, then someone suggested that I conclude the code with "-dont show." However, after adding it, the code continues to produce the same error and no CAM plot appears...

stephanecharette commented 2 years ago

You have to spell it correctly: -dont_show.

stephanecharette commented 2 years ago

Or you have to use X-forwarding when you SSH into the device. Or you have to run it on a device that has X installed.

jojo0513 commented 2 years ago

Yes, in my code, it is -dont show. I made a mistake in my last response. Additionally, I use Google colab; do I still need to instal X?

stephanecharette commented 2 years ago

You still typed it wrong in your last comment, so I'm not convinced you're using it correctly. It should be -dont_show. No, you don't have to install X if you use -dont_show.