export to ONNX tutorial doesn't work

[eagle705]

Hi,

I tried to succeed this tutorial https://github.com/huggingface/notebooks/blob/master/examples/onnx-export.ipynb

but I just get some error like below.. what should I do to solve this problem..?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-23-06a1d0b2a7b7> in <module>()
      7 # opt_options.enable_embed_layer_norm = False
      8 
----> 9 optimized_model = optimizer.optimize_model("onnx/bert-base-cased.onnx", model_type='bert', num_heads=12, hidden_size=768)
     10 optimized_model.save_model_to_file('bert.opt.onnx')

8 frames
/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/optimizer.py in optimize_model(input, model_type, num_heads, hidden_size, optimization_options, opt_level, use_gpu, only_onnxruntime)
    310 
    311     if not only_onnxruntime:
--> 312         optimizer.optimize(optimization_options)
    313 
    314     # Remove the temporary model.

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/onnx_model_bert.py in optimize(self, options, add_dynamic_axes)
    277 
    278         if (options is None) or options.enable_skip_layer_norm:
--> 279             self.fuse_skip_layer_norm()
    280 
    281         if (options is None) or options.enable_attention:

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/onnx_model_bert.py in fuse_skip_layer_norm(self)
    103 
    104     def fuse_skip_layer_norm(self):
--> 105         fusion = FusionSkipLayerNormalization(self)
    106         fusion.apply()
    107 

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/fusion_skiplayernorm.py in __init__(self, model)
     19     def __init__(self, model: OnnxModel):
     20         super().__init__(model, "SkipLayerNormalization", "LayerNormalization")
---> 21         self.shape_infer_helper = self.model.infer_runtime_shape({"batch_size": 4, "seq_len": 7})
     22 
     23     def fuse(self, node, input_name_to_nodes, output_name_to_node):

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/onnx_model.py in infer_runtime_shape(self, dynamic_axis_mapping, update)
     34             shape_infer_helper = self.shape_infer_helper
     35 
---> 36         if shape_infer_helper.infer(dynamic_axis_mapping):
     37             return shape_infer_helper
     38         return None

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/shape_infer_helper.py in infer(self, dynamic_axis_mapping)
     33         self._preprocess(self.model_)
     34         while self.run_:
---> 35             self.all_shapes_inferred_ = self._infer_impl()
     36 
     37         self.inferred_ = True

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/../symbolic_shape_infer.py in _infer_impl(self, start_sympy_data)
   1301                     in_dims = [s[len(s) - out_rank + d] for s in in_shapes if len(s) + d >= out_rank]
   1302                     if len(in_dims) > 1:
-> 1303                         self._check_merged_dims(in_dims, allow_broadcast=True)
   1304 
   1305             for i_o in range(len(node.output)):

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/../symbolic_shape_infer.py in _check_merged_dims(self, dims, allow_broadcast)
    527             dims = [d for d in dims if not (is_literal(d) and int(d) <= 1)]
    528         if not all([d == dims[0] for d in dims]):
--> 529             self._add_suggested_merge(dims, apply=True)
    530 
    531     def _compute_matmul_shape(self, node, output_dtype=None):

/usr/local/lib/python3.7/dist-packages/onnxruntime_tools/transformers/../symbolic_shape_infer.py in _add_suggested_merge(self, symbols, apply)
    156 
    157     def _add_suggested_merge(self, symbols, apply=False):
--> 158         assert all([(type(s) == str and s in self.symbolic_dims_) or is_literal(s) for s in symbols])
    159         symbols = set(symbols)
    160         for k, v in self.suggested_merge_.items():

AssertionError: 

[alenma04]

@eagle705 I am also getting the same error.Are you able to solve this?

[JonathanSum]

I am also getting the same error.

https://colab.research.google.com/drive/1F_PYyugN7G_M_Odv0mMRitxY1aalb2yB?usp=sharing

[osanseviero]

cc @lewtun

[lewtun]

Thanks for reporting this issue @eagle705. This notebook is sorely outdated and should be revamped to showcase:

• How to export models with different topologies (e.g. causal LM heads, masked LM heads etc)
• How to run basic inference with ONNX Runtime, i.e. no optimisation techniques which are now the domain of our optimum library

For point (2) I'd like to show how one can wrap ONNX Runtime InferenceSessions as pipelines (with pre- and post-processing), similar to what was done in Chapter 8 of the NLP with Transformers book or as @ChainYo did in his blog post

I'll get around to this at some point, but if anyone wants to tackle it beforehand they're very welcome!