TF's COCO evaluation wrapper doesn't actually support include_metrics_per_category and all_metrics_per_category

[netanel-s]

System information

• What is the top-level directory of the model you are using: models/research/object_detection/
• Have I written custom code: stock script of COCO's detection evaluation
• OS Platform and Distribution: Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.8.0
• Bazel version (if compiling from source): -
• CUDA/cuDNN version: CUDA 9
• GPU model and memory: Titan XP, 12GB
• Exact command to reproduce: object_detection/eval.py with checkpoint and config file of ssdlite_mobilenet_v2_coco_2018_05_09 with added metrics_set: "coco_detection_metrics" and include_metrics_per_category: true in eval_config.

Describe the problem

Flags include_metrics_per_category and all_metrics_per_category in eval_config of the config file of the model are not used, and therefore are always False. Hence, the output metrics are always the usual 12 metrics of COCO's AP and AR for all categories together.

Source code / logs

In get_evaluators() of object_detection/evaluator.py: EVAL_METRICS_CLASS_DICT[eval_metric_fn_key](categories=categories) should also be using include_metrics_per_category and all_metrics_per_category in case they're existing attributes of eval_config. But then I get the 'Category stats do not exist' in ComputeMetrics() of object_detection/metrics/coco_tools.py since COCOEvalWrapper instance doesn't have category_stats attribute. I tried figuring out how these are supposed to be calculated and fed, but I'm not sure the code supports per category stats (while ComputeMetrics() suggests that it is).

A fix would be highly appreciated. Thank you very much in advance.

[netanel-s]

Just to make sure we're on the same page, the OD API has changed, but this part is essentially the same. When include_metrics_per_category is set to be true, I get the same 'Category stats do not exist' in ComputeMetrics(). And why all_metrics_per_category is no longer a flag for eval_config?

Thanks in advance!

[bsaendig]

+1 I am experiencing the same error. Changing the include_metrics_per_category parameter to true leads to ValueError: Category stats do not exist. A fix would be very helpful.

I looked at the files. So as i understand it, in object_detection/metrics/coco_tools.py there is a wrapper for COCOeval (from pycocotools) which somehow expects that class to hold a dictionary 'category_stats' that should get filled after running the functions evaluate() ,accumulate() and summarize() but 'category_stats' does not actually appear in cocoeval.py. Is there a different version of pycocotools with this capability implemented? Or did someone plan to implement it but didn't do it yet?

[zhangmengya]

I meet the same error when i run model_main.py. The reason is self.category_stats do not exist.

Anyone could solve the problem or how to add the self.category_stats?

Thanks every much

[liuchang8am]

any updates on this one?

[ldalzovo]

Will it be fixed in the next release? Thank you

[liuchang8am]

UPDATE: I implemented per_category_ap by extracting values from matrix s in pycocotools/cocoeval.py. It seems correct, as the mean of each category equals to the mAP calculated by the original code, so are the values from IoU 0.5:0.95:0.05. However, small, medium and large ap seems incorrect.

You may want to look at the following functions if you want to implement you own version.

1) _summerize() in pycocotools/cocoeval.py. Basically, you want to create a list to store the aps of each category, which are extracted from the 5-dimension matrix s. 2) ComputeMetrics() in metrics/coco_tools.py. Here you want to get the values stored in the list in 1), and print them with a loop over categories.

[netanel-s]

Hey @liuchang8am , thanks for the idea. Can you please share why do you think S/M/L AP seems incorrect by this computation? How is your implementation of per-category different than of all categories? If it is not, then it sounds like it's either correct or that the implementation for all categories is incorrect. Thanks again.

[pialin]

Inspired by @liuchang8am, I implemented "include_metrics_per_category"option in "eval_config", which has almost been completed by the author actually. Three code additions are listed below: 1.addtion in "tensorflow/models/research/pycocotools/cocoeval.py":from line 499 to line 648 2.addtion in "tensorflow/models/research/object_detetcion/metrics/coco_tools.py":from line 240 to line 244 3.add

{
    metrics_set: "coco_detection_metrics" 
    include_metrics_per_category: true 
}

to "eval_config" part in pipeline config file.

The code changes are only tested in my code to train a "fastrcnn_resnet_101" model. You should see per category metrics stats plotted in tensorboard if they work. You can try it if you need this feature and don't want to wait for an official implementation. Any feedback would be appreciated.

[tsoi2]

@pialin Thanks for sharing. But your 2nd link to the coco_tools.py directs to the cocoeval.py script listed in the 1st bullet point. Could you confirm

[pialin]

@tsoi2 You are right. Sorry for that mistake and it has been corrected. Thanks for your feedback.

[netanel-s]

@pialin , thanks for sharing. A question though. The original summarize code has a warning which says that the function only works with default parameter setting. Any idea what's that about?

[pialin]

@pialin , thanks for sharing. A question though. The original summarize code has a warning which says that the function only works with default parameter setting. Any idea what's that about?

I don't know either. :)

[rsandler00]

@pialin your solution worked. Thank you!

[austinmw]

@pialin Thanks, worked for me too! Any chance you could submit pull requests to cocoapi and object detection api?

[ubill88]

@pialin Thanks for your solution! I can see all the classes and their mAP displayed on my TensorBoard, and stuff likes below on the terminal (I use legacy/eval.py to evaluate).

INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/bc: 0.152886 INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/br: -1.000000 INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/fchm: -1.000000 INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/fon|: 0.000000 INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/fow-: 0.166513 INFO:tensorflow:DetectionBoxes_PerformanceByCategory/mAP/fs: 0.288510

I'm just wondering if this mAP refer to below? Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ]

Also, is there any way I can see other evaluation results like small mAP, medium mAP, large mAP, and AR?

last, I think coco is a little strict, is there any way i can change the IoU threshold (e.g. 0.50:0.95 to 0.30:0.75)?

Thank you very much! All the best!

[ChowderII]

Hi sorry to necropost,

The solution mentioned above works but only for the precision, I've tried to look at @pialin 's code but I couldn't figure out how to add the recall per category. I see on your edit at line 546 the length of category_stats matches the number of categories my model is classifying and it also matches the amount of new graphs that appear in the category for detection precision. I also in the function _summarize_single_category there is the parameter ap which dictates whether to calculate the average precision or average recall. How should I proceed to add the same number of graphs but for average recall on each of my classes in addition the ones you implemented? Thank you very much!

[ChowderII]

I figured it out, if anyone has that issue as well, let me try to walk you through: @pialin 's code works very nicely, you don't have to change much.

First things first, you have to make coco compute what you want. @pialin 's code already runs coco's metrics by category so you don't have to worry about that. So go inside inside Tensorflow/research/models/cocoapi/PythonAPI/pycocotools/cocoeval.py from line 460 to line 471 is where the stats are stored, so you would have to place a new spot for yours. (Of course, increase the size of stats as appropriate on line 459.

In the same file, we made a spot for a new metric, now we have to tell coco how to calculate it. To do that, go to the function at line 545, which is _summarizeDets_per_category(). As one can see here, there are 12 metrics being added to category_stats. You have two choices here: create your own function and assign your new category_stats cell to its return value or have this function _summarize_single_category(ap=1, iouThr=None, categoryId=None, areaRng='all', maxDets=100) at line 504 do it. I chose the second option and will therefore continue with that, but it should be relatively the same after that. But either way make sure to increase the size of category_stats accordingly. (Note that in my case, I use coco detection boxes. Using the masks should be no different except the location of a few lines. They should be very close nonetheless.)

We can see that each list item of category_stats is being assigned to the return value of _summarize_single_category(...) so as mentioned earlier, we are going to go to line 504 and edit this function : _summarize_single_category(ap=1, iouThr=None, categoryId=None, areaRng='all', maxDets=100). We are now under the hood of the coco metric calculation. For my purpose, I did not have to add anything there because the function already calculates the Average Recall. If you want to calculate more things, you would add it there or inside your own function.

Great, now coco calculates our new metric per category and it saves it at the right spot. Next step is make sure it gets rendered to tensorboard.

At line 269 inside of TensorFlow/models/research/models/object_detection/metrics/coco_tools.py: add the metric group you want. The metric PerformanceByCategory/mAP/... is already there, just copy and paste it right below and change the name of the metric, but remember this name or names because they will be very important later on. In my case, I changed the name to PerformanceByCategory/AR/{} so metrics/coco_tools.py at line 269 should look like this

per_category_ap['PerformanceByCategory/mAP/{}'.format(
          category)] = self.category_stats[0][category_index]
per_category_ap['PerformanceByCategory/AR/{}'.format(
          category)] = self.category_stats[6][category_index]

Notice on the last line of this block of code: self.category_stats[6][category_index]. The number inside the first set of square brackets dictates what metric you are going to fetch from category_stats in the cocoeval.py file from earlier. In my case, the Average Recall was calculated per category at index 6 but not displayed. So I just went ahead and fetched it.

If you run this code, it should technically work with no errors, but your graphs won't show up. That's because tensorboard doesn't know they exist. That is going to be the next step.

Inside the file TensorFlow/models/research/models/object_detection/metrics/coco_evaluation.py, we need to add the name of the metric from earlier (remember how I said it was important?). if we go to line 738, we will see we are already appending metrics_names with a string value. We want to follow exactly this format. That is keeping the 'DetectionBoxes_ ... prefix, otherwise you will get a KeyError everytime. So again, in my case, my code ended up looking like this:

metric_names.append('DetectionBoxes_PerformanceByCategory/mAP/' +
                            category_dict['name'])
metric_names.append('DetectionBoxes_PerformanceByCategory/AR/' +
                            category_dict['name'])

And there you have it! You should have successful added a new metric to the object_detection pipeline from TensorFlow. I used tensorflow-gpu and tensorboard version 1.12.0 for this.

If you get KeyError during execution, it is most likely due to a typo when you placed the names of your new metric either inside of TensorFlow/models/research/models/object_detection/metrics/coco_evaluation.py or TensorFlow/models/research/models/object_detection/metrics/coco_tools.py. Remember that you need the prefix inside of coco_evaluation.py!

Sorry for this stupidly long post but hopefully I can help someone! Let me know if there's typos or section that are unclear and I'll do my best to correct them. Cheers,

[yashwantram97]

Inspired by @liuchang8am, I implemented "include_metrics_per_category"option in "eval_config", which has almost been completed by the author actually. Three code additions are listed below: 1.addtion in "tensorflow/models/research/pycocotools/cocoeval.py":from line 499 to line 648 2.addtion in "tensorflow/models/research/object_detetcion/metrics/coco_tools.py":from line 240 to line 244 3.add

{
    metrics_set: "coco_detection_metrics" 
    include_metrics_per_category: true 
}

to "eval_config" part in pipeline config file.

The code changes are only tested in my code to train a "fastrcnn_resnet_101" model. You should see per category metrics stats plotted in tensorboard if they work. You can try it if you need this feature and don't want to wait for an official implementation. Any feedback would be appreciated.

I dont know why i m getting - AttributeError: 'COCOEvalWrapper' object has no attribute 'summarize_per_category'. Still not working for me

[Marwen-Bhj]

I am still having the same issue while using the model_main.py, even tho the lines @pialin haev mentioned were added by default, after one epoch, my training process runs through the same issue.

[yashwantram97]

I am still having the same issue while using the model_main.py, even tho the lines @pialin haev mentioned were added by default, after one epoch, my training process runs through the same issue.

Try using the train.py in the legacy folder. Thats how i was able to train. Just copy and paste the train.py from object detection/legacy/train.py to object detection/train.py

[hanifjunos]

I have followed @pialin code but nothing changed on the evaluation result. There are no cocoeval.py file in the original tensorflow/models. Where should i put this file in tensorflow model?

[Jaideepm08]

metric_names.append('DetectionBoxes_PerformanceByCategory/AR/' +
                            category_dict['name'])

Thanks. This worked. But I had to do one extra change in TensorFlow/models/research/models/object_detection/metrics/coco_evaluation.py The changes are the same as on line 738 but have to be done after line 381 as well.

[Kuz-man]

I am still having the same issue while using the model_main.py, even tho the lines @pialin haev mentioned were added by default, after one epoch, my training process runs through the same issue.

You can check the contents of the cocoeval file you're importing. It might be that you're using the original API and there's truly no attribute 'summarize_per_category'.

from pycocotools import cocoeval
print(os.path.abspath(cocoeval.__file__))

[zishanahmed08]

@Marwen-Bhj @hanifjunos Where you able to get it to work with the model_main.py or legacy train.py?

[zishanahmed08]

@Jaideepm08 @Kuz-man Where you able to get it to work with the model_main.py or legacy train.py?

[zishanahmed08]

@ChowderII Are your changes vailable in a gist or PR somewhere?

[birdman9391]

@zishanahmed08 I installed custom pycocotools and now able to get per category evaluation. You can check the instruction written by Philipp Schwarz https://stackoverflow.com/questions/51848274/show-per-category-metrics-in-tensorboard/55435466#55435466

By following this instruction, tensorboard shows the per category AP.

[ruzgarkanar]

Hi guys. HELPP!!!. I want to see the accuracy of the data in the test set and see which pictures know and which did not. I couldn't do that please help

[ruzgarkanar]

@pialin I don't understand where I'm doing wrong, can you help me? the solution didn't work for me still can't see.

[RohitKeshari]

per_category_ap['PerformanceByCategory/AR/{}'.format(
          category)] = self.category_stats[6][category_index]

Hi, It would be really helpful if you will provide your updated files. Since, line number is not consistent, therefore making workable changes are little hard. I have possibly find the places where you refereed to change, but still not getting class wise AP and AR in tensorboard (not even on terminal). I would appreciate, If anyone can help me showing class wise AP and AR on terminal. Thanks

[ChowderII]

@zishanahmed08

Hi sorry for the very long wait time. Unfortunately not because I hardcoded most of the changes. Meaning that my changes are most likely not going to work for you. Your best bet is to try and follow what I said and make it work for yourself!

[zishanahmed08]

Did someone try it out with the tf2 scripts?

[dinis-rodrigues]

@ChowderII Thanks for the explanation, I've struggle really hard to find a way to make custom metrics. I guess I was missing the changes in the coco_evaluation.py file...

[ChowderII]

@dinis-rodrigues I'm not sure about custom metrics. I know that the metrics I was interested in already had a function in coco_evalaluation.py I have no idea how one could go about to add his/her very own custom metrics but if you are looking for metrics that coco_eval provides, I suggest trying out my blob of text! Good luck!

[dinis-rodrigues]

@ChowderII You got me wrong! By custom metrics, I meant adding different IoU thresholds and intervals. Your solution worked perfectly for what I wanted.

[McBusinessTime]

How is this not an official fix by now?

pycocotools is not even included in the tensorflow models repository anymore.

[ChowderII]

How is this not an official fix by now?

pycocotools is not even included in the tensorflow models repository anymore.

My guess is it is relatively difficult to place in a PR because of the nature means the solution is custom to your needs. Someone at TF needs to change a lot of things under the hood to make it work for everyone's use case. That is also why I did not submit a PR because I am not capable enough in python and TF to change this behaviour in a meaningful way beyond my own hardcoded use case.

[mohammedayub44]

Just ran into the same issues this week while training. I will try the changes proposed by @ChowderII
Surprising that this is not fixed officially as of yet.