search

hailo-ai / hailo_model_zoo

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
MIT License
266 stars 40 forks source link

Xfeat onnx translate failed #95

Open dongxuanlb opened 4 months ago

dongxuanlb commented 4 months ago

the onnx use by the script: https://github.com/acai66/accelerated_features/blob/main/export.py and the batch size is modified to 1.

pytorch version is 2.1.0+cpu

the export command is : python3 export.py --xfeat_only_model_dualscale --export_path xfeat_ds.onnx --opset 17

>>> hn, npz = runner.translate_onnx_model('xfeat_ds.onnx', 'xfeat', start_node_names=['images'], end_node_names=['mkpts', 'feats', 'sc'])
[info] Translation started on ONNX model xfeat
[info] Restored ONNX model xfeat (completion time: 00:00:00.03)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.11)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:00.43)
Traceback (most recent call last):
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 179, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(onnx_model, valid_net_name, start_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 237, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(onnx_model, None, net_name, start_node_names, end_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 255, in parse_model_to_hn
    converter = ONNXConverter(model=model,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 77, in __init__
    super(ONNXConverter, self).__init__(
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 48, in __init__
    self._calculate_valid_subgraph_scope()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 422, in _calculate_valid_subgraph_scope
    current_vertex.in_valid_subgraph = True
AttributeError: 'NoneType' object has no attribute 'in_valid_subgraph'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 876, in translate_onnx_model
    parser.translate_onnx_model(model=model, net_name=net_name, start_node_names=start_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 211, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(simplified_model, valid_net_name,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 237, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(onnx_model, None, net_name, start_node_names, end_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 255, in parse_model_to_hn
    converter = ONNXConverter(model=model,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 77, in __init__
    super(ONNXConverter, self).__init__(
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 48, in __init__
    self._calculate_valid_subgraph_scope()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 422, in _calculate_valid_subgraph_scope
    current_vertex.in_valid_subgraph = True
AttributeError: 'NoneType' object has no attribute 'in_valid_subgraph'

any help?

dongxuanlb commented 4 months ago

when I use the basis model of xfeat:

python3 export.py --xfeat_only_model --export_path xfeat_ds.onnx --opset 17

I got :

>>> hn, npz = runner.translate_onnx_model('xfeat_ds.onnx', 'xfeat', start_node_names=['images'], end_node_names=['feats', 'keypoints', 'heatmaps'])
[info] Translation started on ONNX model xfeat
[info] Restored ONNX model xfeat (completion time: 00:00:00.02)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.07)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:00.21)
Traceback (most recent call last):
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 179, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(onnx_model, valid_net_name, start_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 237, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(onnx_model, None, net_name, start_node_names, end_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 263, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 76, in convert_model
    self._create_layers()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 28, in _create_layers
    self._add_direct_layers()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 110, in _add_direct_layers
    self._layer_callback_from_vertex(vertex)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 192, in _layer_callback_from_vertex
    if vertex.op in OPTIONAL_NULL_OPS and vertex.is_null_operation() and not is_flattened_global_maxpool:
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 3852, in is_null_operation
    if self.op == 'Slice' and self.is_null_slice():
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 3868, in is_null_slice
    if slices_args['steps'] != 1:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 876, in translate_onnx_model
    parser.translate_onnx_model(model=model, net_name=net_name, start_node_names=start_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 211, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(simplified_model, valid_net_name,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 237, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(onnx_model, None, net_name, start_node_names, end_node_names,
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 263, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 76, in convert_model
    self._create_layers()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 28, in _create_layers
    self._add_direct_layers()
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 110, in _add_direct_layers
    self._layer_callback_from_vertex(vertex)
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 192, in _layer_callback_from_vertex
    if vertex.op in OPTIONAL_NULL_OPS and vertex.is_null_operation() and not is_flattened_global_maxpool:
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 3852, in is_null_operation
    if self.op == 'Slice' and self.is_null_slice():
  File "/home/dx/Downloads/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 3868, in is_null_slice
    if slices_args['steps'] != 1:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
omerwer commented 4 months ago

Hi @dongxuanlb, This error can be caused by unsupported operations in the SW by Hailo or a bad ONNX. Can you try exporting this model with opset 13 and see if it parses successfully? If not, can you please open a ticket in the Hailo website about this issue with the relevant info (the error, the github from which you exported the ONNX and the ONNX itself)?

Regards,

dongxuanlb commented 4 months ago

I tried with opset to 13, the same error appeared. 

(rknn) dx@PN50:~/workspace/libs/accelerated_features$ python3 export.py --xfeat_only_model --export_path xfeat_ds.onnx --opset 13
loading weights from: /home/dx/workspace/libs/accelerated_features/modules/../weights/xfeat.pt
/home/dx/workspace/pyenvs/rknn/lib/python3.10/site-packages/torch/onnx/symbolic_helper.py:1513: UserWarning: ONNX export mode is set to TrainingMode.EVAL, but operator 'instance_norm' is set to train=True. Exporting with train=True.
  warnings.warn(
Model exported to xfeat_ds.onnx
omerwer commented 4 months ago

@dongxuanlb I see. It seems to be a bug in our Parser. I'll take this to the R&D and let you know if there's a WA and when this would be supported.

omerwer commented 4 months ago

Hi @dongxuanlb, One of our engineers created a fork for this repo and added support for the ONNX export that can be compiled into a Hailo HEF file + an example HEF files. You are welcome to try it yourself: https://github.com/guyp98/accelerated_features