search

MIC-DKFZ / medicaldetectiontoolkit

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.
Apache License 2.0
1.3k stars 297 forks source link

How to find AP score of the best trained model? #97

Closed catmangoeswild closed 4 years ago

catmangoeswild commented 4 years ago

Hi I want to first thank you for posting this toolkit on Github. The whole thing is well-coded!

I learned from the config file that the model selection criteria is AP score, which is awesome! I have used my own dataset and got multiple models (best check points) in exp_dir. Your code generated a file: epoch_ranking.npy, which contains ranked epoch numbers. I noticed there are multiple precision-recall graphs in plots folder, however, precision-recall graphs don't have AP score on them.

If you don't mind me asking, how can I find the AP score of each of these epoch numbers ranked in epoch_ranking.npy?

Thank you very much! -Bo

pfjaeger commented 4 years ago

Hi, thanks for your feedback.

The scores are not written out in epoch_ranking.py, but you could change that around here:

https://github.com/MIC-DKFZ/medicaldetectiontoolkit/blob/8a2218f89f8bee05af4db95ca8ee991a932fd855/utils/exp_utils.py#L150

Otherwise, did you see the monitoring plot, that displays AP values for all classes over epochs?

catmangoeswild commented 4 years ago

Hi, thanks for your feedback.

The scores are not written out in epoch_ranking.py, but you could change that around here:

https://github.com/MIC-DKFZ/medicaldetectiontoolkit/blob/8a2218f89f8bee05af4db95ca8ee991a932fd855/utils/exp_utils.py#L150

Otherwise, did you see the monitoring plot, that displays AP values for all classes over epochs?

catmangoeswild commented 4 years ago

Hi, thanks for your feedback.

The scores are not written out in epoch_ranking.py, but you could change that around here:

https://github.com/MIC-DKFZ/medicaldetectiontoolkit/blob/8a2218f89f8bee05af4db95ca8ee991a932fd855/utils/exp_utils.py#L150

Otherwise, did you see the monitoring plot, that displays AP values for all classes over epochs?

catmangoeswild commented 4 years ago

Hi pfjaeger, I want to thank you for instant reply! I now know how to extract AP score of the best models. Due to WiFi issue, I accidentally clicked 'close'.

I can see 'monitor_0_0.png' of which the x axis is 'epochs' and y axis is 'loss / metrices'. Is 'loss / metrices' in this graph AP?

Thank you. -Bo

pfjaeger commented 4 years ago

There should be a legend to the plot as in https://github.com/MIC-DKFZ/medicaldetectiontoolkit/blob/master/assets/loss_monitoring.png ?

catmangoeswild commented 4 years ago

Yes that's right. Thank you very much for getting back to me and help me solve this problem. I appreciate it! -Bo