thodan / bop_toolkit

A Python toolkit of the BOP benchmark for 6D object pose estimation.
http://bop.felk.cvut.cz
MIT License
376 stars 135 forks source link

Add multiprocessing for eval_bop19_pose.py #110

Closed nv-nguyen closed 2 months ago

nv-nguyen commented 7 months ago

Adding multiprocessing with 10 workers by default for evaluation script eval_bop19_pose.py. The original script can still be run with argument --num_worker 1.

I made some unit tests with MegaPose's results.

Here are the run-time comparisons with and without multiprocessing. Note that the scores remains unchanged. The run-time improvement is more significant for large datasets such as T-LESS.

Screenshot 2023-12-01 at 21 26 39

I can do some refactoring to make the code cleaner after this pull-request is accepted.

@thodan @MartinSmeyer @ylabbe

nv-nguyen commented 5 months ago

To test the multiprocessing:

bash bop_toolkit_lib/tests/unit_test.sh
nv-nguyen commented 5 months ago

I resolved all reviews and add a comparison with expected outputs. Here is the output in my terminal when running the unit tests (I removed unit tests of YCB-V and T-LESS to make it fast since these two datasets are quite large):

Screenshot 2024-01-17 at 14 12 30
nv-nguyen commented 4 months ago
  1. I fixed a bug and finalised here the scoring function for 6D detection task. The main difference with the scoring function in 6D localization task is this part where we count the number of false positives to calculate precision.
  2. The final mAP score for 6D detection task is calculated across all object IDs here (similar to 2D detection task).

To test eval_bop24_pose.py script: python bop_toolkit_lib/tests/eval_bop24_pose_test.py

To test eval_bop24_pose.py script with "fake" 10.000 false positives: python bop_toolkit_lib/tests/eval_bop24_pose_test.py --num_false_positives 10000