openvinotoolkit / training_extensions

Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™
https://openvinotoolkit.github.io/training_extensions/
Apache License 2.0
1.14k stars 442 forks source link

otx pot optimize convnext_maskrcnn error #2417

Closed tonyeatsm closed 1 year ago

tonyeatsm commented 1 year ago

I use the version of otx1.4.1, running pot optimize, an error is reported (the error log is below), how to solve it, I see that the template is marked in trainingextensions-1.4.1/sources/training_extensions-1.4.1/src/otx/algorithms/detection/configs/instance_segmentation/convnext_maskrcnn The template_experiment mark is set, is the convnext_maskrcnn function not tested yet?

Error log:

root@22ff2bf8d6a7:~/trainingextensions/models/otx-workspace/INSTANCE_SEGMENTATION_MaskRCNN-ConvNeXt_1024x1024_guabao/outputs# otx optimize \

--train-data-roots /root/trainingextensions/datasets/${DATASET_NAME}/coco/${DATASET_VERSION} \ --val-data-roots /root/trainingextensions/datasets/${DATASET_NAME}/coco/${DATASET_VERSION} \ --workspace ${OTX_WORKSPACE} \ --load-weights ${OTX_WORKSPACE}/outputs/${MODEL_VERSION}/openvino/openvino.xml

/opt/conda/lib/python3.8/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. warnings.warn( INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino [] Update data configuration file to: /root/trainingextensions/models/otx-workspace/INSTANCE_SEGMENTATION_MaskRCNN-ConvNeXt_1024x1024_guabao/data.yaml 2023-08-03 16:26:36,123 | INFO : Loading OpenVINO OTXDetectionTask WARNING:root:The parameter "result_based_confidence_threshold" not found in OTX_MaskRCNN wrapper, will be omitted WARNING:root:The parameter "use_ellipse_shapes" not found in OTX_MaskRCNN wrapper, will be omitted 2023-08-03 16:26:37,570 | INFO : OpenVINO task initialization completed 2023-08-03 16:26:37,571 | INFO : Start PTQ optimization Statistics collection: 100%|██████████████████████████████████████████████████████████████████████| 300/300 [10:06<00:00, 2.02s/it] Biases correction: 100%|████████████████████████████████████████████████████████████████████████████| 72/72 [00:16<00:00, 4.43it/s] WARNING:root:The parameter "result_based_confidence_threshold" not found in OTX_MaskRCNN wrapper, will be omitted WARNING:root:The parameter "use_ellipse_shapes" not found in OTX_MaskRCNN wrapper, will be omitted Traceback (most recent call last): File "/opt/conda/bin/otx", line 33, in sys.exit(load_entry_point('otx', 'console_scripts', 'otx')()) File "/root/trainingextensions/sources/trainingextensions-1.4.1/src/otx/cli/tools/cli.py", line 77, in main results = globals()[f"otx{name}"]() File "/root/trainingextensions/sources/training_extensions-1.4.1/src/otx/cli/tools/optimize.py", line 126, in main task.optimize( File "/root/trainingextensions/sources/training_extensions-1.4.1/src/otx/algorithms/detection/adapters/openvino/task.py", line 758, in optimize self.inferencer = self.load_inferencer() File "/root/trainingextensions/sources/training_extensions-1.4.1/src/otx/algorithms/detection/adapters/openvino/task.py", line 461, in load_inferencer inferencer = OpenVINOMaskInferencer(args) File "/root/trainingextensions/sources/training_extensions-1.4.1/src/otx/algorithms/detection/adapters/openvino/task.py", line 272, in init model = Model.create_model(model_adapter, "OTX_MaskRCNN", configuration, preload=True) File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/models/model.py", line 186, in create_model return Model(inference_adapter, configuration, preload) File "/root/trainingextensions/sources/training_extensions-1.4.1/src/otx/algorithms/detection/adapters/openvino/model_wrappers/openvino_models.py", line 32, in init super().init(model_adapter, configuration, preload) File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/models/instance_segmentation.py", line 29, in init super().init(inference_adapter, configuration, preload) File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/models/image_model.py", line 60, in init super().init(inference_adapter, configuration, preload) File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/models/model.py", line 86, in init self.load() File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/models/model.py", line 395, in load self.inference_adapter.load_model() File "/opt/conda/lib/python3.8/site-packages/openvino/model_api/adapters/openvino_adapter.py", line 192, in load_model self.compiled_model = self.core.compile_model( File "/opt/conda/lib/python3.8/site-packages/openvino/runtime/ie_api.py", line 398, in compile_model super().compile_model(model, device_name, {} if config is None else config), RuntimeError: Check 'false' failed at src/inference/src/core.cpp:114: [ GENERAL_ERROR ] AssertionFailed: data.size() == OC

goodsong81 commented 1 year ago

@eugene123tw Could you take a look? Maybe you can get support from @sovrasov if needed.

eugene123tw commented 1 year ago

Issue fixed in CVS-118373. Fixed will be released in OpenVINO 2023.2