Accuracy Aware Quantization- Object Segmentation fails for OpenVINO 2024.3 and ultralytics_8.2.74 with yolov8-seg model and coco128-seg dataset

fmishra747 commented 2 months ago

🐛 Describe the bug

Accuracy Aware segmentation script fails for openvino 2024.3 at

validator = model.ValidatorClass(args) File "/data/venv/openvino_2024.1.0/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1729, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'YOLO' object has no attribute 'ValidatorClass'

looks like need to find the right version of ultralytics, nncf which works with ov2024.3

Attached log with error for openvino 2024.3 accuracy_aware_segmentation_openvino2024.3.log

The same script works with openvino 2024.1,
ultralytics 8.0.43 nncf 2.11.0.dev0+24f0ee70

Attached working log for openvino 2024.1 accuracy_aware_object_segmenttaion_ov2024.1_ultralytics8.0.43-working.log

Environment

The accuracy aware segmentation fails in the environment with openvino 2024.3 and the following packages:

OpenVINO environment openvino 2024.3.0 openvino-dev 2024.3.0 openvino-genai 2024.3.0.0 openvino-telemetry 2024.1.0 openvino-tokenizers 2024.3.0.0 ultralytics 8.2.74 ultralytics-thop 2.0.0 nncf 2.13.0.dev0+eb91af2d torch 2.4.0+cpu torchvision 0.19.0+cpu numpy 1.23.5

Minimal Reproducible Example

quantize_accuracy_aware_object_segmentation .py.md quantize_accuracy_aware_object_segmentation .sh.md

The attached accuracy aware segmentation python and shell script were run with the following command : ./quantize_accuracy_aware_object_segmentation.sh /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/yolov8n-seg.xml /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/yolov8n-seg.pt /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/coco128-seg.yaml /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/datasets/coco128-seg FP32 CPU PERFORMANCE None None 640 640 /home/u01/dlworkbench/66a011d3b17d8e407a74799a yolov8m INT8 0.01 128 300

yolov8n-seg IR and coc128-seg dataset coco128-seg.zip

Are you going to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

daniil-lyakhov commented 2 months ago

@fmishra747, hello,

I still could not help you because I could not run the script you have filed to me. I don't see following required parameters to reproduce:

MODEL_XML="$1"
MODEL_PT="$2"
YAML_FILE_PATH="$3"
ACCURACY_AWARE_DATASET_PATH="$4"
PRECISION="$5"
TARGET_DEVICE="$6"
PRESET="$7"
IGNORED_SCOPE_NAMES="$8"
IGNORED_SCOPE_TYPE="$9"
TARGET_W="${10}"
TARGET_H="${11}"
OUTPUT_PATH="${12}"
QUANTIZED_MODEL_NAME="${13}"
QUANTIZED_SCHEME="${14}"
MAX_ACCURACY_DROP="${15}"
SUBSET_SIZE="${16}"
RANKING_SUBSET_SIZE="${17}"

Could you please make an isolated reproducer without any parameters so I can run it locally? Please confirm that it is a NNCF issue as it looks like it is a bug in your internal script

alexsu52 commented 2 months ago

@andrey-churkin, I tried to update ultralytics to 8.2.74 for NNCF yolo examples and encountered the same problem. The branch is https://github.com/alexsu52/nncf/tree/as/ultralytics. Please, take a look at it.

andrey-churkin commented 1 week ago

@fmishra747 Please use changes from https://github.com/openvinotoolkit/nncf/pull/3038 to modify your script accordingly. Please reopen if you have any other questions.

openvinotoolkit / nncf