[Bug] Postraining quantization with DefaultQuantization and use_layerwise_tuning=True fails

pskoko commented 1 year ago

System information (version)

OpenVINO=> 2022.2.0
Operating System / Platform => Ubuntu20.04
Compiler => PyPi
Problem classification => Model quantization
Model name: ResNet50

Detailed description

Performing posttraining quantization with DefaultQuantization algorithm and use_layerwise_tuning parameter set to True, quantization fails with the following error:

Traceback (most recent call last):
  File "example.py", line 78, in <module>
    compressed_model = pipeline.run(model=model)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/pipeline/pipeline.py", line 52, in run
    result = self.collect_statistics_and_run(model, current_algo_seq)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/pipeline/pipeline.py", line 62, in collect_statistics_and_run
    model = algo.run(model)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/algorithms/quantization/default/algorithm.py", line 95, in run
    model = algo.run(model)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/algorithms/quantization/layerwise_finetuning/algorithm.py", line 85, in run
    retval = self._fine_tuning_loop(
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/algorithms/finetuning/algorithm.py", line 149, in _fine_tuning_loop
    fp_activations = self._update_batch_from_model(self._original_model,
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/algorithms/finetuning/algorithm.py", line 189, in _update_batch_from_model
    _, output_activations = self._engine.predict(model_callbacks, batch_indices_sample)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/engines/ie_engine.py", line 119, in predict
    self._predict(stats_layout=stats_layout,
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/engines/ie_engine.py", line 174, in _predict
    self._process_dataset_async(stats_layout=stats_layout,
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/engines/ie_engine.py", line 396, in _process_dataset_async
    batch_annotations, image_batch, batch_meta = self._process_batch(data_batch)
  File "/home/pskoko/anaconda3/envs/openvino_env_2022.2.0/lib/python3.8/site-packages/openvino/tools/pot/engines/ie_engine.py", line 451, in _process_batch
    if not all([isinstance(item, tuple) for item in batch]):
TypeError: 'int' object is not iterable

OpenVINO distriubution is from pip for python 3.7: openvino_dev-2022.2.0-7713-py3-none-any.whl . The same issue is with python3.8 or openvino 2022.1.0.

Steps to reproduce

Download resnet-50-pytorch model using _omzdownloader --name resnet-50-pytorch and convert it to OpenVINO IR using _omzconverter --name resnet-50-pytorch, and download some images for qunatization (for example https://github.com/fastai/imagenette)
Using DataLoader for Images from https://docs.openvino.ai/latest/pot_default_quantization_usage.html#prepare-data-and-dataset-interface and qunatization code from https://docs.openvino.ai/latest/pot_default_quantization_usage.html#run-quantization create quantization script

Set use_layerwise_tuning to true in algorithm parameters:

algorithms = [
{
    "name": "DefaultQuantization",
    "params": {
        "target_device": "ANY",
        "stat_subset_size": 300,
        "use_layerwise_tuning": True,
        "stat_batch_size": 1
    },
}
]

Run quantization script

In there is example script for reproducing errors, without model and sample images since github allows files up to 25MB. example.txt

Issue submission checklist

[x] I report the issue, it's not a question
[x] I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
[x] There is reproducer code and related data files: images, videos, models, etc.

rkazants commented 1 year ago

@AlexKoff88, @KodiaqQ, please help.

AlexKoff88 commented 1 year ago

Hi @pskoko! Thanks for noticing that. This is an experimental feature that we would like to implement but still cannot make it work correctly due to resource constraints. If you are experienced in that feel free to fix it and contribute. Since it is experimental we don't have strong acceptance criterium for contributions. For now, I cannot give any timeline for when we get back to it, unfortunately.

akladiev commented 1 year ago

This issue will be closed in 2 weeks in case of no activity.

avitial commented 1 year ago

Ref. 111486

avitial commented 1 year ago

@pskoko we don't think the _use_layerwisetuning was ever a mature feature, an initial version was actually implemented for experiments. May I ask how important is this feature for you and what are you trying to achieve with it? Perhaps we could help you to achieve the required accuracy without using this feature/parameter.

Please also note that POT will be deprecated since v2023.0 and we highly recommend to use NNCF instead (docs here). Though _use_layerwisetuning is not available in NNCF, there is a chance to get the required accuracy without this parameter.

pskoko commented 1 year ago

@avitial Hi, thanks for response. I managed to use accuracy aware quantization to achieve results I want.

openvinotoolkit / openvino