The model in edgeai-benchmark/tutorials/tutorial_classification.ipynb is not compiled

[wintersurvival]

Hi,

I try to run edgeai-benchmark/tutorials/tutorial_classification.ipynb, but it seems that the FP32 model is not compiled and the fold "artifacts" is empty. Any ideas? Thank you!

`runtime_options: {'tensor_bits': 8, 'accuracy_level': 1, 'debug_level': 0, 'inference_mode': 0, 'advanced_options:high_resolution_optimization': 0, 'advanced_options:pre_batchnorm_fold': 1, 'advanced_options:calibration_frames': 10, 'advanced_options:calibration_iterations': 10, 'advanced_options:quantization_scale_type': None, 'advanced_options:activation_clipping': 1, 'advanced_options:weight_clipping': 1, 'advanced_options:bias_calibration': 1, 'advanced_options:output_feature_16bit_names_list': '', 'advanced_options:params_16bit_names_list': '', 'advanced_options:add_data_convert_ops': 3, 'ti_internal_nc_flag': 83886080, 'info': {'prequantized_model_type': 0}} {'cl-mnv2': {'task_type': 'classification', 'calibration_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ffde7fa0>, 'input_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ea9da380>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fe4c6edde10>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fe4c6eddde0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fe4c6edde70>, 'model_info': {'metric_reference': {'accuracy_top1%': 71.88}}}} configs to run: ['cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx'] number of configs: 1

INFO:20240624-154818: parallel_run - parallel_processes:1 parallel_devices=range(0, 1) TASKS | | 0% 0/1| [< ] INFO:20240624-154818: starting process on parallel_device - 0 0%| || 0/1 [00:00<?, ?it/s]

INFO:20240624-154818: starting - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx INFO:20240624-154818: model_path - /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx INFO:20240624-154818: model_file - /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/model/mobilenet_v2_tv.onnx INFO:20240624-154818: quant_file - /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/model/mobilenet_v2_tv_qparams.prototxt Downloading 1/1: /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx Download done for /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx Downloading 1/1: /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx Download done for /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx Converted model is valid!

INFO:20240624-154818: running - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx INFO:20240624-154818: pipeline_config - {'task_type': 'classification', 'calibration_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ffde7fa0>, 'input_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ea9da380>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fe4c6edde10>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fe4c6eddde0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fe4c6edde70>, 'model_info': {'metric_reference': {'accuracy_top1%': 71.88}}} TASKS | 100%|██████████|| 1/1 [00:00<00:00, 1.98it/s]ome time...

configs to run: ['cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx'] number of configs: 1

INFO:20240624-154819: parallel_run - parallel_processes:1 parallel_devices=range(0, 1) TASKS | | 0% 0/1| [< ] INFO:20240624-154819: starting process on parallel_device - 0 0%| || 0/1 [00:00<?, ?it/s]

INFO:20240624-154819: starting - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx INFO:20240624-154819: model_path - /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx INFO:20240624-154819: model_file - /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/model/mobilenet_v2_tv.onnx INFO:20240624-154819: quant_file - /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/model/mobilenet_v2_tv_qparams.prototxt

INFO:20240624-154819: running - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx INFO:20240624-154819: pipeline_config - {'task_type': 'classification', 'calibration_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ffde7fa0>, 'input_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fe4ea9da380>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fe4c6edde10>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fe4c6eddde0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fe4c6edde70>, 'model_info': {'metric_reference': {'accuracy_top1%': 71.88}}} INFO:20240624-154819: infer - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx - this may take some time...libtidl_onnxrt_EP loaded 0x55b02937ac20

**** WARNING *** : Could not open /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/artifacts/allowedNode.txt for reading... Entire model will run on ARM without any delegation to TIDL ! Final number of subgraphs created are : 1, - Offloaded Nodes - 0, Total Nodes - 0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 14%|#3 || 6792/50000 [00:50<05:08, 139.97it/s] INFO:20240624-154919: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 31%|### || 15494/50000 [01:50<04:00, 143.38it/s] INFO:20240624-155019: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 48%|####7 || 23826/50000 [02:54<03:25, 127.63it/s] INFO:20240624-155119: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 65%|######4 || 32427/50000 [03:54<02:04, 140.81it/s] INFO:20240624-155219: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 82%|########2 || 41243/50000 [04:55<01:00, 144.74it/s] INFO:20240624-155319: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_t| 100%|##########|| 50000/50000 [05:58<00:00, 139.58it/s]

INFO:20240624-155417: infer completed - cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx - 358 sec

SUCCESS:20240624-155417: benchmark results - {'infer_path': 'cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx', 'accuracy_top1%': 72.054, 'num_subgraphs': 0, 'infer_time_core_ms': 2.694567, 'infer_time_subgraph_ms': 0.0, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0} TASKS | 100%|██████████|| 1/1 [05:58<00:00, 358.62s/it] TASKS | 100%|██████████|| 1/1 [05:58<00:00, 358.52s/it]

packaging artifacts to ./modelartifacts/8bits_package please wait... SUCCESS:20240624-155418: finished packaging - /opt/code/edgeai-benchmark/modelartifacts/8bits/cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx download the atricats files from the folder: ./modelartifacts/8bits_package ['extract.sh', 'artifacts.csv', 'cl-mnv2_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx.tar.gz', 'artifacts.yaml']`

[mathmanu]

Your issue seems to be resolved as you have closed the thread. What was the issue?

[wintersurvival]

Hi @mathmanu ,

When running edgeai-tensorlab/blob/main/edgeai-benchmark/run_custom_pc.sh for mobilenet_v2_tv_qat-p2.onnx model, there is an calibration iteration before compiling the QAT model, is that expected? runtime_options=settings.runtime_options_onnx_qat_v1() is used for it now.

Look forward to your reply, thank you!

[mathmanu]

For QAT models with runtime_options=settings.runtime_options_onnx_qat_v1(), there will be only a single calibration iteration. That is expected - it is not doing any modification to the model.

[wintersurvival]

Thank you @mathmanu ! I tried to run QAT-v2 model mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx with runtime_options=settings.runtime_options_onnx_qat_v2(), and there are two errors during running:

1. Context: Bad node spec for node. Name: /0/DequantizeLinear_1 OpType: DequantizeLinear
2. INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(uint8)) , expected: (tensor(float))

How to deal with them? Look forward to your reply, thank you!

TARGET_SOC:     TDA4VM
TARGET_MACHINE: pc
DEBUG MODE:     false @ rb05477:5678
TIDL_TOOLS_PATH=/opt/code/edgeai-benchmark/tools/TDA4VM/tidl_tools
LD_LIBRARY_PATH=/opt/code/edgeai-benchmark/tools/TDA4VM/tidl_tools
PYTHONPATH=::
===================================================================
work_dir = ./work_dirs/modelartifacts/TDA4VM/8bits
packaged_dir = ./work_dirs/modelpackage/TDA4VM/8bits
loading annotations into memory...
Done (t=0.41s)
creating index...
index created!
loading annotations into memory...
Done (t=0.62s)
creating index...
index created!
configs to run: ['imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx']
number of configs: 1

INFO:20240626-061323: parallel_run - parallel_processes:1 parallel_devices=range(0, 1)
TASKS                                                       |          |     0% 0/1| [< ]
INFO:20240626-061323: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]

INFO:20240626-061323: starting - imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx
INFO:20240626-061323: model_path - /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
INFO:20240626-061323: model_file - /opt/code/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM/8bits/imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx/model/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
INFO:20240626-061323: quant_file - /opt/code/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM/8bits/imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx/model/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_qparams.prototxt
Downloading 1/1: /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
Downloading http://software-dl.ti.com/jacinto7/esd/modelzoo/09_02_00/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx to /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
3825664it [00:03, 1235585.50it/s]                                                                                                                                              
Download done for /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
Downloading 1/1: /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
Download done for /opt/code/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv2/mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model.onnx
Converted model is invalid: Unrecognized attribute: axis for operator DequantizeLinear

==> Context: Bad node spec for node. Name: /0/DequantizeLinear_1 OpType: DequantizeLinear

INFO:20240626-061327: running - imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx
INFO:20240626-061327: pipeline_config - {'task_type': 'classification', 'calibration_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fdd863929b0>, 'input_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fdd4d822920>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fdd4d485fc0>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fdd4d4860e0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fdd4d4862f0>, 'model_info': {'metric_reference': {'accuracy_top1%': 72.476}}}
INFO:20240626-061327: import  - imagecls-qat-v2_onnxrt_imagenet1k_edgeai-tv2_mobilenet_v2_lite_wt-v2_qat-v2-wc8-at8_20231120_model_onnx - this may take some time...
Preliminary subgraphs created = 1 
Final number of subgraphs created are : 1, - Offloaded Nodes - 313, Total Nodes - 313 
Traceback (most recent call last):
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 203, in _run_pipeline
    result = cls._run_pipeline_impl(basic_settings, pipeline_config, description)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 176, in _run_pipeline_impl
    accuracy_result = accuracy_pipeline(description)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 87, in __call__
    param_result = self._run(description=description)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 114, in _run
    self._import_model(description)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 178, in _import_model
    self._run_with_log(session.import_model, calib_data)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 282, in _run_with_log
    return func(*args, **kwargs)
  File "/opt/code/edgeai-benchmark/edgeai_benchmark/sessions/onnxrt_session.py", line 69, in import_model
    outputs = self.interpreter.run(output_keys, calib_dict)
  File "/opt/.pyenv/versions/py310/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(uint8)) , expected: (tensor(float))