Mt5 model loading fails

[OriAlpha]

Hallo, I have MT5 pretrained model, i am using fastt5 approch to convert the model to onnx. The convestion of the model works fine. But when creating the decoder_sess at decoder_sess = InferenceSession(str(path_to_decoder)) more specfic it fails at

# initialize the C++ InferenceSession
sess.initialize_session(providers, provider_options, disabled_optimizers)

it fails without any error, as Process finished with exit code 135 (interrupted by signal 7: SIGEMT) Loading the encoder model works, but not decoder model

I am using latest version of fastt5==0.1.4 Any ideas to create session.

[Ki6an]

are you using it for GPU?

[OriAlpha]

yes, but i tried with CPU. Still it fails

[Ki6an]

sorry, the library does not support GPU yet, but the issue is similar to https://github.com/microsoft/onnxruntime/issues/3113

for CPU are you facing the same issue?

[OriAlpha]

Yes on cpu too same issue

[Ki6an]

it looks like the issue is in onnxruntime itself, I suggest you to create an issue there.

[OriAlpha]

i found some more logs as, onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: SystemError : 2

[Ki6an]

could you please provide the reproducible code and full log of error

[OriAlpha]

I am not sure about code but you can use following model, i am using example mentioned in main readme page https://huggingface.co/google/mt5-large

[OriAlpha]

Also i was able to get soultion for this by onnx github, there is new tool for converting model from mt5 models https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/models/t5. just to give more idea, fastt5 exporter creates file size of 8.5 GB for https://huggingface.co/google/mt5-large model. But onnx tool creates around 9.2 GB so i think some additional files may be missing while exporting the model hence it was failing to create seesion

[OriAlpha]

I am facing input name issue as [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:pkv_95, is there any way to pass original name of inputs. I could see in encoder module the inputs are passed as pkv_0 to pkv_95, which doesnt match with original input names

[OriAlpha]

i tried different approch, now it gives out onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Deserialize tensor onnx::MatMul_4622 failed.tensorprotoutils.cc:637 TensorProtoToTensor External initializer: onnx::MatMul_4622 offset: 0 size to read: 11534336 given file_length: 4194304 are out of bounds or can not be read in full.