search

open-mmlab / cocoapi

COCO API - Dataset @ http://cocodataset.org/
Other
46 stars 35 forks source link

Evaluation are two slow compared to pycocotools #26

Closed John-Yao closed 3 years ago

John-Yao commented 3 years ago

Hi, I am trying to evaluate result by mmpycocotools=12.0.3. However, the evaluation is two slow. (> 20mins ; about 6600 images). I killed the process by CTRL-C and got below messages:

` Evaluating bbox... Loading and preparing results... DONE (t=33.08s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox ^CTraceback (most recent call last): File "tools/test.py", line 244, in main() File "tools/test.py", line 238, in main print(dataset.evaluate(outputs, **eval_kwargs)) File "xxxxxxx/mmdet/datasets/coco.py", line 492, in evaluate cocoEval.evaluate() File "xxxxxxx/lib/python3.7/site-packages/mmpycocotools-12.0.3-py3.7-linux-x86_64.egg/pycocotools/cocoeval.py", line 163, in evaluate evaluateImg(imgId, catId, areaRng, maxDet) for catId in catIds File "xxxxxxx/lib/python3.7/site-packages/mmpycocotools-12.0.3-py3.7-linux-x86_64.egg/pycocotools/cocoeval.py", line 164, in for areaRng in p.areaRng for imgId in p.imgIds File "xxxxxxx/lib/python3.7/site-packages/mmpycocotools-12.0.3-py3.7-linux-x86_64.egg/pycocotools/cocoeval.py", line 296, in evaluateImg if gtm[tind, gind] > 0 and not iscrowd[gind]: KeyboardInterrupt Traceback (most recent call last): File "xxxxxxx/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "xxxxxxx/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "xxxxxxx/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in main() File "xxxxxxx/lib/python3.7/site-packages/torch/distributed/launch.py", line 254, in main process.wait() File "xxxxxxx/lib/python3.7/subprocess.py", line 990, in wait return self._wait(timeout=timeout) File "xxxxxxx/lib/python3.7/subprocess.py", line 1624, in _wait (pid, sts) = self._try_wait(0) File "xxxxxxx/lib/python3.7/subprocess.py", line 1582, in _try_wait (pid, sts) = os.waitpid(self.pid, wait_flags) KeyboardInterrupt

`

Compared to pycocotools:

Loading and preparing results... DONE (t=2.94s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=172.79s). Accumulating evaluation results... DONE (t=6.67s).

John-Yao commented 3 years ago

I figure out the problem. Thanks! Evaluation are too slow due to larger number of output boxes.