[Phi3] List index out of range error on converting microsoft Phi3 onnx models to TfLite

[suyash-narain]

Issue Type

Others

OS

Linux

onnx2tf version number

1.22.4

onnx version number

1.15.0

onnxruntime version number

1.17.1

onnxsim (onnx_simplifier) version number

0.4.35

tensorflow version number

2.16.1

Download URL for ONNX

https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/blob/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/phi3-mini-4k-instruct-cpu-int4-rtn-block-32-acc-level-4.onnx

Parameter Replacement JSON

NA

Description

purpose: personal development, trying to use LLMs with TFLite framework

On trying to convert the phi3 model phi3-mini-4k-instruct-cpu-int4-rtn-block-32-acc-level-4.onnx model using onnx2tf -i <model_name.onnx>, i get the error:

Model conversion started ============================================
Traceback (most recent call last):
            File "/home/linuxbrew/.linuxbrew/bin/onnx2tf", line 8, in <module> sys.exit(main())
                                                                                                                                      ^^^^^
            File "/home/linuxbrew/.linuxbrew/opt/python@3.11/site-packages/onnx2tf/onnx2tf.py", line 2380, in main
                                               model = convert(
                                                              ^^^^^^
            File "/home/linuxbrew/.linuxbrew/opt/python@3.11/site-packages/onnx2tf/onnx2tf.py", line 924, in convert
                            batch_size = input[0].shape[0]
                                                 ~~~^^
IndexError: list index out of range

I am not able to understand why list index out of range issue is coming in at conversion and what can i do to move forward here? thanks

[PINTO0309]

This error may be corrected by adjusting onnx2tf.

IndexError: list index out of range

But before that, there is the most critical problem: the large amount of custom operations in the com.microsoft v1 namespace (Microsoft's own implementation of custom OPs) are used, making it impossible to convert this model using onnx2tf.

The following groups of operations are all outside the ONNX standard specifications.

e.g.

SimplifiedLayerNormalization
MatMulNBits
GroupQueryAttention

Unfortunately, it is also virtually impossible to implement the conversion operation because there is no corresponding operation in TensorFlow.

[github-actions[bot]]

If there is no activity within the next two days, this issue will be closed automatically.