NNCF v2.5.0 accuracy and performance degradation of EfficientDet

lisosia commented 1 year ago

I tried NNCF 2.5.0 for my effdet model and observed accuracy and performance degradation. Even if I use pot-backend (use_pot=True), the accuracy and performance is degraded compared to nncf 2.4.0.

The issue is reproduced with the efficientdet-d0 which was exported by the official procedure.
The throughput was measured by benchmark-app of openvino-dev=2023.0.0.

model	coco_precision	Throughput
v2.4.0	30.68%	67.14 FPS
commit ca54404 (pot backend)	30.50%	65.40 FPS
commit ca54404	24.53%	60.92 FPS

I attaches the reproduction code and environments.
I hope the issue will be fixed so that we can migrate to the newer NNCF.

repro_effdet_issue.zip

MaximProshin commented 1 year ago

@l-bat , please help with this issue cc @alexsu52

l-bat commented 1 year ago

@lisosia have you measured accuracy of initial (FP32/FP16) OpenVINO IR? Have you tried to obtain OpenVINO IR with the latest instruction? Could you please share efficientdet-d0_frozen.xml and efficientdet-d0_frozen.bin files?

lisosia commented 1 year ago

@l-bat @MaximProshin

have you measured accuracy of initial (FP32/FP16) OpenVINO IR?

model	coco_precision	Throughput
FP32	31.93%	33.79 FPS

Could you please share efficientdet-d0_frozen.xml and efficientdet-d0_frozen.bin files?

I attaches the zip file repro_effdet_issue.0627.zip

Have you tried to obtain OpenVINO IR with the latest instruction?

No. I refered to doc of 2022.3 instead of the doc of 2023.0. See 01_PREPARE_MODEL.sh for the detail.

l-bat commented 1 year ago

@lisosia could you please try to use latest NNCF version? I tried to reproduce problem on 240cc24

python ~/nncf/tests/openvino/tools/calibrate.py -c quantization_config.json --impl native
accuracy_check -c accuracy_check.yaml -m efficientdet-d0_quantization/efficientdet-d0_frozen.xml

coco_precision: 31.33%

python ~/nncf/tests/openvino/tools/calibrate.py -c quantization_config.json --impl pot
accuracy_check -c accuracy_check.yaml -m efficientdet-d0_quantization/efficientdet-d0_frozen.xml

coco_precision: 30.87%

quantization_config.json

{
    "compression": {
        "algorithms": [
            {
                "name": "DefaultQuantization",
                "params": {
                    "preset": "performance",
                    "stat_subset_size": 300
                }
            }
        ],
        "dump_intermediate_model": true
    },
    "engine": {
        "config": "<path to accuracy_check.yaml>"
    },
    "model": {
        "model": "efficientdet-d0_frozen.xml",
        "model_name": "efficientdet-d0",
        "weights": "efficientdet-d0_frozen.bin"
    }
}

I couldn't reproduce accuracy drop with provided script too.

lisosia commented 1 year ago

@l-bat Performance degradation is also observed in 240cc24. Throughput degradation is minor when I retried.

Model	coco_precision	Throughput
v2.4.0	30.68%	61.80 FPS (retry)
240cc24 (pot backend)	30.55%	61.98 FPS
240cc24	24.53%	61.10 FPS

I found benchmark-app -api sync yields more reliable (less variance) results. The throughput gap between pot/nncf(pot-backend) and latest(native-backend) is small but still visible.

Model	`benchmark-app -api sync`
v2.4.0	54.24 FPS
ca54404 (pot)	54.22 FPS
ca54404	53.90 FPS
240cc24 (pot)	54.28 FPS
240cc24	53.87 FPS

coco_precision: 31.33%

It's weird that FP32 accuracy is different from my result 31.93%

lisosia commented 1 year ago

Note that I hope nncf (native-backend) will yield the same accuracy/throughput with pot-backend because use_pot=True will be deleted in the future (https://github.com/openvinotoolkit/nncf/issues/1923#issuecomment-1598277970)

l-bat commented 1 year ago

It's weird that FP32 accuracy is different from my result 31.93% The accuracy of the FP32 is the same as your result.

INT8 native: coco_precision: 31.33%

INT8 use_pot: coco_precision: 30.87%

lisosia commented 1 year ago

Oh, I misread that. 31.33% was the int8(native) accuracy. How was the accuracy of FP32 model?

BTW, I'll try nncf/tests/openvino/tools/calibrate.py tomorrow.

l-bat commented 1 year ago

FP32 accuracy is 31.93%.

Why do you use has_background: False parameter in accuracy_check.yaml? I could reproduce the degradation in accuracy on colab, but it still works fine locally. On colab with PERFORMANCE preset I obtained default_coco_precision: 30.98% for use_pot=False.

l-bat commented 1 year ago

@lisosia Could you please try to use

advanced_parameters=AdvancedQuantizationParameters(overflow_fix="enable")

? I think the issue is only reproducible on machines without VNNI support.

lisosia commented 1 year ago

@l-bat

Why do you use has_background: False parameter in accuracy_check.yaml?

Regarding dataset settings, I copied the following sample

Could you please try to use advanced_parameters=AdvancedQuantizationParameters(overflow_fix="enable")? I think the issue is only reproducible on machines without VNNI support.

I use Intel(R) Core(TM) i7-8700 CPU, which does not support VNNI I think.

Model	coco_precision	`benchmark-app -api sync`
FP32	31.93%	36.65 FPS
v2.4.0	30.68%	54.24 FPS
240cc24 (pot)	30.55%	54.28 FPS
240cc24 (native)	24.53%	53.87 FPS
240cc24 (pot, overflow-fix=enable)	30.55%	-
240cc24 (native, overflow-fix=enable)	24.27%	53.92 FPS
240cc24 (native, PERFORMANCE)	30.98%	53.53 FPS

overflow-fix="enable" didn't make significant effect. native+PERFORMANCE yields 30.98%, which is the same with you. (the FPS is a little more slower, that is strange).

l-bat commented 1 year ago

But you provided config with dataset: name: ms_coco_detection_91_classes https://github.com/openvinotoolkit/open_model_zoo/blob/c7a13a842d41333397ea6c3f9bc5a7053da00eec/data/dataset_definitions.yml#L72-L76

lisosia commented 1 year ago

@l-bat Sorry for the confusion. It should have been named ms_coco_detection_90_class_without_background. While looking for a working config, the name seems to have been inappropriate. I didn't use the --definitions option of accuracy-checker, so the name wouldn't affect the behavior I believe.

lisosia commented 1 year ago

@l-bat From the above, the following problems were identified at the latest NNCF

native+preset=MIXED: the accuracy and throughput is worse than pot/nncf240
native+preset=PERFORMANCE: accuracy is good, but throughput is even worse than native+preset=MIXED (this is not expected)

We hope these problems will be corrected.

l-bat commented 1 year ago

@lisosia We currently working on ChannelAlignment algorithm which should improve accuracy on your machine. Could you please verify changes at https://github.com/openvinotoolkit/nncf/pull/1928? Please enable channel_alignment in your code:

advanced_parameters=AdvancedQuantizationParameters(disable_channel_alignment=False)

lisosia commented 1 year ago

@l-bat sorry for the late reply. I tested commit 76d6b3a.

Model	coco_precision	`benchmark-app -api sync`
FP32	31.93%	36.65 FPS
v2.4.0	30.68%	54.24 FPS
240cc24 (pot)	30.55%	54.28 FPS
240cc24 (native)	24.53%	53.87 FPS
240cc24 (native, PERFORMANCE)	30.98%	53.53 FPS
76d6b3a (pot, disable_channel_alignment=False)	30.56%	54.17 FPS
76d6b3a (native, disable_channel_alignment=False)	30.73%	54.15 FPS
76d6b3a (native, PERFORMANCE, disable_channel_alignment=False)	30.95%	53.33 FPS

76d6b3a (native, disable_channel_alignment=False) yields almost same accuracy and throughput compared to v2.4.0. Thank you!

76d6b3a (native, PERFORMANCE, disable_channel_alignment=False) result is still strange (better accuracy but slower than preset=MIXED).

MaximProshin commented 1 month ago

Closed as done

openvinotoolkit / nncf

NNCF v2.5.0 accuracy and performance degradation of EfficientDet #1935