Get the mAP and Accuracy parameters during training ...

[MyVanitar]

Hello,

is there anyway to know the mAP and Accuracy values during or after each training epoch? That's essential to know.

other parameters is also good to know but these two are the most important ones.

[Jumabek]

Hi, yes there is. As you might already experienced during training network saves weight files. Using those weight files you can test the network where it will output the detections for all the images that you provided in data/test.data file

This is my example for calling the validation function darknet.exe detector valid data/caltech-test.data cfg/caltech-custom-anchor-test-exc.cfg C:\darknet\build\darknet\x64\backup\caltech_exc_people_10x\caltech-custom-anchor-exc_42000.weights -i 1

By default it will save the file that contains detections for all the test images in 'results' folder.

from that output you can calculate the mAP using Pascal VOC script.

[MyVanitar]

@Jumabek

I did what you said, but it crashes with this output.

[ghost]

@VanitarNordic

Create a "results" directory in the darknet root. I got the same segmentation fault because the validation function tries to write result files to a directory that doesnt exist.

[MyVanitar]

@hth013

Yess, Thank you. now it works and generated a txt file. Then how can I calculate mAP from this?

[ghost]

I personally use py-faster-rcnn/tools/reval.py which i have modified to work with my dataset. Specifically the "_do_python_eval" function in py-faster-rcnn/lib/datasets/pascal_voc.py can load the generated txt files for each class and calculate mAP, but it has to be modified if you have custom classes.

[RunZH]

@hth013 hi. Could you please show more details on how to find reval.py and modify it?

[meedddhhhhaaaaa]

@RunZH @hth013

Details on plotting mAP for custom datasets? i.e., modifying reval.py?

[ghost]

Instead of calling "imdb.evaluate_detections", you call "imdb._do_python_eval(output_dir)" directly, where output_dir contains the PASCAL VOC result files for each class generated by the "darknet detector valid ..." command. This will give you mAP.

[meedddhhhhaaaaa]

@hth013 yay! Thanks! Will try right now! fingers crossed

[meedddhhhhaaaaa]

@hth013 @RunZH

All bugs fixed; the reval.py now needs a "detections.pkl" file but the results files generated by "darknet detector valid ..." is in .txt

Anything I have missed?

I ran reval.py as: python reval.py \output\dir\path

[katsanz13]

@medhasn @hth013 @RunZH @Jumabek Can anyone explain in more detail how to modify reval.py and pascal_voc.py to get the mAP of custom objects?

[yyy03]

@medhasn have you ever solve the "detections.pkl" problem? i have the "/result/" ,but do not know how to get the mAP

[Cartucho]

I translated the official PASCAL VOC code into Python. It allows you to use your custom classes with some extra plots and visualizations!

Give it a try, if it is useful to you: https://github.com/Cartucho/mAP

[AlexeyAB]

Or just use darknet.exe detector map data/obj.data yolo-obj.cfg backup\yolo-obj_7000.weights

to get mAP, APs, IoU, F1, TP/FP/FN, precision-recall: https://github.com/AlexeyAB/darknet#when-should-i-stop-training