search

openvinotoolkit / training_extensions

Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™
https://openvinotoolkit.github.io/training_extensions/
Apache License 2.0
1.14k stars 442 forks source link

Accuracy drop on Post-Training Quantization (PTQ) optimized model for maskrcnn-swimt #3593

Closed yunchu closed 1 week ago

yunchu commented 2 months ago

Post-Training Quantization (PTQ) optimization applied to maskrcnn_swimt in the instance segmentation task may result in significantly reduced accuracy.

How to run the ins_seg benchmark tests for OTX 2.0.0

pytest -vs tests/perf/test_instance_segmentation.py --output-root benchmark_otx_2.0.0_result --model-category all --data-root /path/to/val-data/v2 --eval-upto optimize --num-repeat 5

image-2024-06-05-16-29-29-929

chuneuny-emily commented 2 months ago

cc'ed @eugene123tw

eugene123tw commented 2 months ago

Fixed in https://github.com/openvinotoolkit/training_extensions/pull/3685

chuneuny-emily commented 1 month ago

This issue is expected to be addressed with an upgrade to OpenVINO and NNCF in a future release.

sovrasov commented 1 week ago

Fixed in https://github.com/openvinotoolkit/training_extensions/pull/3900