Closed oborchers closed 1 year ago
stas00
commented
3 years ago Thank you very much, @oborchers for opening a new ticket and re-testing with other models and verifying that this problem is project-wide.
I hope @mfuntowicz gets a chance to have a look at it, or tag someone else who understands this sub-domain.
mriganktiwari
commented
3 years ago Hi @mfuntowicz @stas00 , is this a known issue with GPT2 as well? Please let me know if there is a workaround.
I was considering to convert gpt2 or gpt2-medium to ONNX using the notebook provided here.
On executing the line of code below:
convert(framework="pt", model="gpt2-medium", output=Path("onnx/gpt2-medium.onnx"), opset=11)
I get this error:
1115 for i, x in enumerate(value):
1116 if not isinstance(x, int):
-> 1117 raise ValueError("The type of axis index is expected to be an integer")
1118 if x in value_dict:
1119 warnings.warn('Duplicate dynamic axis index {} was provided for input {}.'
ValueError: The type of axis index is expected to be an integer```
d-lowl
commented
3 years ago I recently stumbled upon this issue myself. Specifically case 2. The same error appears for facebook/bart-large, facebook/bart-large-cnn, IlyaGusev/mbart_ru_sum_gazeta. The main issue here is that for some outputs the tokenizer/model gives not a tensor, but rather a tuple of tensors, which is then converted into a list of shape dicts.
torch.onnx._validate_dynamic_axes (line 1193 in the latest release) expects a dict (and does nothing) or a list of ints for dynamic_axes (and mocks up some axes names), however (for the reason above) it gets a list of dicts ([map int -> string])
for key, value in dynamic_axes.items():
if key not in valid_names:
warnings.warn("Provided key {} for dynamic axes is not a valid input/output name".format(key))
if isinstance(value, list):
warnings.warn('No names were found for specified dynamic axes of provided input.'
'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
value_dict = {}
for i, x in enumerate(value):
if not isinstance(x, int):
raise ValueError("The type of axis index is expected to be an integer")
if x in value_dict:
warnings.warn('Duplicate dynamic axis index {} was provided for input {}.'
.format(x, key))
else:
value_dict[x] = str(key) + '_dynamic_axes_' + str(i + 1)
dynamic_axes[key] = value_dictI will keep digging into that, but the core question here is why Bart and related models return tuple of tensors (for outputs 1 to 12; outputs 0 and 13 are fine)? Although, I'm not an expert in either transformers, pytorch or onnx, so I might be missing something.
On a slight tangent here, is there a specific reason why summarization pipeline is not in the supported pipeline types for this script?
bestpredicts
commented
3 years ago any update?
katerinafrid
commented
3 years ago any update?
VibhuJawa
commented
3 years ago any update?
LysandreJik
commented
3 years ago We're currently working on a rework of the ONNX implementation within Transformers, which is available here: https://github.com/huggingface/transformers/pull/11786
Instead of offering a script to enable conversions for all models (which was not kept up to date with recent model releases), we're opting for a case-by-case approach, while offering the tools to convert models manually in a straightforward and simple manner; by creating OnnxConfig configuration objects to specify the input and output types of each model.
Please take a look at the PR and give us your feedback.
oborchers
commented
3 years ago @LysandreJik: Thank you very much! I think this is an excellent way to go. Having converted a dozen models myself, we internally went for something similar, albeit not nearly as streamlined / sophisticated.
@attr.s(auto_attribs=True)
class TransformersONNXConfig(BaseConfig):
"""Provides the basic configuration for all models."""
base_model: str
trans_cfg: PretrainedConfig
input_names: List[str]
output_names: List[str]
dynamic_axes: Dict
model_args: Set[torch.tensor]
tokenizer: PreTrainedTokenizerFast
extra_args: Dictand
def create_and_export_onnx_model(self):
"""Creates a new model if the current model does not exist and exports it."""
torch.onnx.export(
self.create_torch_model(),
self.cfg.model_args,
f=self.onnx_posix_pth,
input_names=self.cfg.input_names,
output_names=self.cfg.output_names,
dynamic_axes=self.cfg.dynamic_axes,
do_constant_folding=True,
use_external_data_format=False,
enable_onnx_checker=True,
opset_version=12,
)Where the most important part is self.create_torch_model, as we regularly modify the basic torch model with custom layers down the line. Is support for such a feature planned? If not, is it considerable? As it would substantially easy conversion of custom models, such as the sbert ones.
Furthermore, would it make sense to make OnnxConfig a part of the PreTrainedModel config to enable support from the get-go?
And finally, I assume this leaves us with the export, so that for seq2seq models we need still need to re-write the .generate function? Or is it possible to add support for an ONNX model from your side (probably difficult, as it's a part of the pre-trained model already, which would require double loading the model)?
mfuntowicz
commented
3 years ago Thanks @oborchers for your comments and use-cases.
I will let @LysandreJik speak about a potential integration of the OnnxConfig within the PreTrainedModel config, my initial plan was to have 100% backward compatibility, this explain why I put this somewhere else (currently).
Regarding generate, this is something that might require some investigations but I'm seeing good opportunities to have something within the ONNX graph with the recent knobs released by Microsoft folks on the ONNXRuntime project (cc @tianleiwu for visibility on this).
Still, for this initial rework of the ONNX exporting capabilities we focused on "model only", with the ability to extend to full pipelines in the future. Generation is definitively one of the hardest task to get within the graph, but also one where I can see the biggest benefits.
oborchers
commented
3 years ago @mfuntowicz: Thank you for your feedback! Yes, I understand the point for the compatibility to the fullest. After all, it's not that difficult to get to the config if done once or twice.
Regarding the .generate function. Thanks for the link! Will look into this more! Yes, absolutely!!
AayushSameerShah
commented
1 year ago Hi @mfuntowicz @stas00 , is this a known issue with GPT2 as well? Please let me know if there is a workaround.
I was considering to convert
gpt2orgpt2-mediumto ONNX using the notebook provided here.On executing the line of code below:
convert(framework="pt", model="gpt2-medium", output=Path("onnx/gpt2-medium.onnx"), opset=11)I get this error:
1115 for i, x in enumerate(value): 1116 if not isinstance(x, int): -> 1117 raise ValueError("The type of axis index is expected to be an integer") 1118 if x in value_dict: 1119 warnings.warn('Duplicate dynamic axis index {} was provided for input {}.' ValueError: The type of axis index is expected to be an integer```
Hello @mriganktiwari Any update on this? I am still facing the issue with GPT-2. I have used the same code as yours. Please guide, thanks!
mahita2104
commented
1 year ago can i work on this? please assign this to me
ArthurZucker
commented
1 year ago Hey! This is no longer something handled at the transformers level, will close it! Sorry for the inconvenience
The way to handle it now is through optimum! See this documentation page for more information: ONNX exporter
@stas00's edit on top:
I currently don't have the know-how in this domain, so if there are members of the community with ONNX experience and this issue resonates with you, please don't hesitate to comment if you'd like to work on resolving this. Thank you very much!
Environment info
transformersversion: 4.3.0.dev0Who can help
@stas00 (based on his suggestion to open a new issue in #9722 and run this with bart) @patrickvonplaten (based on link of @stas00 in #9722) @mfuntowicz (based on link of @stas00 in #9722) @LysandreJik (based on link of @stas00 in #9722)
Information
Model I am using (Bert, XLNet ...): facebook/bart-large & facebook/wmt19-en-de
The problem arises when using:
The tasks I am working on is:
Description
Initially, I was about to use the export for facebook/wmt19-en-de via ONNX for our deployment. Yet, it turns out that the exported models do not work properly. It seems, that there are several things broken for the export of this model type.
To reproduce
1. Testing facebook/wmt19-en-de
Will raise the following exception:
As stated in #9722, I'd assume that some dynamic shape of was not inferred properly/not passed to the dynamic_shapes of torch.onnx.export. But thats just a quick guess, which I find when I build my own ONNX models. Important: The first string passes the assertions, the second one doesn't.
2. Testing facebook/bart-large (feature extraction)
@stas00 suggested to re-test the behavior with the underlying BART model. Now, say we run the same script with the following parameters:
Raises
3. Testing facebook/bart-large (text-generation)
4. Testing facebook/bart-large (fill-mask)
Expected behavior
1 & 2 & 4 point into the direction, that something is wrong with inferring the dynamic shapes, if I am right. 3 just popped up while I was testing the other pipelines.
In all cases, the export & usage should work properly.