Closed remusao closed 8 months ago
Thank you @remusao . I believe this a PyTorch bug introduced in PyTorch 2.1, for reference https://github.com/pytorch/pytorch/issues/110597 https://github.com/pytorch/pytorch/pull/111694
We temporarily disabled the support due to this issue: https://github.com/huggingface/optimum/pull/1457
Could you try again with PyTorch 2.0.1?
Thanks for the quick response @fxmarty; pinning PyTorch to 2.0.1 fixes the conversion.
@fxmarty Should I close this issue or would you like to keep it open while the regression in PyTorch is addressed?
It's better to let it open I think. Instead of disabling the export like I did we should put a requirement on torch<2.1,>=2.2
Just stumbled upon this 😢 @fxmarty, are there any plans for a patch release that would include the merged fix?
Thank you @stekiri, we will do a release this week that includes https://github.com/huggingface/optimum/pull/1666 that fixes this issue for torch>=2.1.2. In the meantime you can use the dev version of optimum: pip uninstall optimum && pip install git+https://github.com/huggingface/optimum.git
.
optimum-cli export onnx --model facebook/bart-large-mnli bart_onnx --task sequence-classification
works again.
@remusao can you send inference script of converted model?
@akashAD98 You should be able to use: https://huggingface.co/docs/optimum/main/en/onnxruntime/package_reference/modeling_ort#optimum.onnxruntime.ORTModelForSequenceClassification
https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/models is a good reference.
im getting the error i tried below code
@fxmarty
!optimum-cli export onnx --task zero-shot-classification --model facebook/bart-large-mnli bart-large-mnli_onnx_zs_model
log: 2024-03-01 07:33:03.428142: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-03-01 07:33:03.428217: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-03-01 07:33:03.434604: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-03-01 07:33:05.711962: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Framework not specified. Using pt to export the model. Using the export variant default. Available variants are:
from transformers import AutoTokenizer,pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained("bart-large-mnli_onnx_zs_model")
model = ORTModelForQuestionAnswering.from_pretrained("bart-large-mnli_onnx_zs_model")
inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
outputs = model(**inputs)
error:
KeyError Traceback (most recent call last)
7 frames /usr/local/lib/python3.10/dist-packages/optimum/onnxruntime/modeling_ort.py in forward(self, input_ids, attention_mask, token_type_ids, **kwargs) 1258 outputs = self.model.run(None, onnx_inputs) 1259 -> 1260 start_logits = outputs[self.output_names["start_logits"]] 1261 end_logits = outputs[self.output_names["end_logits"]] 1262 if use_torch:
KeyError: 'start_logits'
@akashAD98 Answered on https://github.com/huggingface/optimum/issues/1739
System Info
Who can help?
@michaelbenayoun (Tagging you since the issue template suggests it for exports to ONNX)
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction (minimal, reproducible, runnable)
Install optimum:
Then try to convert
facebook/bart-large-mnli
to ONNX format:I also tried the following commands which all fail in a similar way:
Or,
I also tried to use opset 12, and remove the safetensor model to force conversion from torch tensors (not sure if that should make any difference).
The output is then:
Expected behavior
The conversion should succeed and result in a valid ONNX model store in
onnx_output
folder. This used to work in older versions and as far as I can see the model hosted on the hub has not been updated for a long time, apart from the addition of safetensors (I have not tracked which version of torch/transformers/onnxruntime broke the conversion).