search

imatge-upc / detection-2016-nipsws

Hierarchical Object Detection with Deep Reinforcement Learning
http://imatge-upc.github.io/detection-2016-nipsws/
MIT License
423 stars 129 forks source link

something about precision-recall curve and loss #12

Closed Anida-qin closed 7 years ago

Anida-qin commented 7 years ago

I came with problems about precision-recall function. I don't know how to caculate them and your code don't show that. Could you please give me the formulas? And the second question is about loss, i only use voc2007 to train your code, but it didn't decrease so much. There's still big flunctuate, 0.3 round but sometimes more than 1.0, but i don't change anything.. Thank you

miriambellver commented 7 years ago

Hi!

For the precision recall curves I needed detections depending on some threshold, so I used the q-value of the trigger as the confidence of each detection. Once I had different detections for different thresholds, I simply calculated the precision recall (precision = tp/(tp+fp), recall = tp(tp+fn)). A detection is considered correct if the intersection over union with the ground truth is superior to 0.5. For the losses, in my experience in RL, it is better observing directly the reward at each step than the losses.

Best! Miriam

Anida-qin commented 7 years ago

Hi ! Thanks for your reply. It really helps a lot. You readme.txt says that his model is not the one in your paper. The origin model trained only for one plane detection, right? I run the image_zoom_model_testing for test multistep of finding a plane, but it shows a bad result. I want to know if i should change some codes to get a good result as your paper showed. I think maybe i did something wrong, but i didn't change your codes. That confuses me a lot.

Thanks a lot

miriambellver commented 7 years ago

Hi!!

The model is just trained for planes, as the one for the paper, but this model is trained to search for more than one plane, so that when there is a trigger action, it covers the detected object and searches again for planes. The results may not be equal but there is no reason for them to be bad. For simulating the paper, the most similar approach would be to indicate that you only want to look for one object with the flag of only_first_object set to 1.

Miriam

Anida-qin commented 7 years ago

Hi Thanks a lot. I tried only_first_object before. But still don't ideal. I have already sent you my pic results for the model by email. I don't know if these results the right results since your paper said this method will miss some planes. Maybe I'm misunderstanding. Do you mind send me your test results for this model on muliti step searching or give some comments on the results i sent ? It will be a great help for me. Thanks again for your patient.

Anida

klvn930815 commented 7 years ago

@miriambellver ,hi!

I'm also puzzled about the precision-recall curve. Did you use the q-value of the trigger in all the steps( even when the current actions were not trigger),or just use the ones in the steps when it triggered?

thanks!

miriambellver commented 7 years ago

Hi! Even when the action was not triggered.

Regards, Miriam