swoook
commented
2 years ago - SKT-AI/KoBART requires
transformers==4.3.3 - In
transformers==4.3.3,transformers.convert_graph_to_onnxsupports exporting to ONNX --helpfortransformers.convert_graph_to_onnx:
$ python -m transformers.convert_graph_to_onnx --help usage: ONNX Converter [-h]
[--pipeline {feature-extraction,ner,sentiment-analysis,fill-mask,question-answering,text-generation,translation_en_to_fr,translation_en_to_de,translation_en_to_ro}]
--model MODEL [--tokenizer TOKENIZER] [--framework {pt,tf}] [--opset OPSET] [--check-loading] [--use-external-format]
[--quantize]
output
positional arguments:
output
optional arguments:
-h, --help show this help message and exit
--pipeline {feature-extraction,ner,sentiment-analysis,fill-mask,question-answering,text-generation,translation_en_to_fr,translation_en_to_de,translation_en_to_ro}
--model MODEL Model's id or path (ex: bert-base-cased)
--tokenizer TOKENIZER
Tokenizer's id or path (ex: bert-base-cased)
--framework {pt,tf} Framework for loading the model
--opset OPSET ONNX opset to use
--check-loading Check ONNX is able to load the model
--use-external-format
Allow exporting model >= than 2Gb
--quantize Quantize the neural network to be run with int8-
--model:- Hugging Face saves model into two files:
- config.json which saves the configuration of your model
- pytorch_model.bin which is the PyTorch checkpoint
- We can pass the directory in which they exist
- Or it also accepts model's id
- For example, the model's id of this model is skt/kobert-base-v1
-
--framework--framework pFor PyTorch--framework tFor TensorFlow
- Recall the examples of $REPO_ROOT/examples in SKT-AI/KoBART
- They use
pytorch-lightning - And
pytorch-lightningsaves the model into .ckpt and .yaml file - However,
transformers.convert_graph_to_onnxdoes NOT support such format - We can reverse them to the format of Hugging Face:
from dmp_kobart import KoBARTClassification
paths = dict()
paths['ckpt'] = $CKPT_PATH
paths['yaml'] = $YAML_PATH
paths['huggingface'] = $OUTPUT_DIR
pytorch_lightning_model_wrapper = KoBARTClassification.load_from_checkpoint(
checkpoint_path=paths['ckpt'],
hparams_file=paths['yaml'],
map_location=None,
)
pytorch_lightning_model_wrapper.model.save_pretrained(paths['huggingface'])
🚀 Feature request
transformers==4.3.3transformers>=4.9.0supports exporting BART to ONNXMotivation
ORT are sometimes on par with TRT
TRT only supports NVIDIA GPU
ORT supports NVIDIA GPU and Intel CPU
I cannot find any document about AMD for ORT 🙈
TRT performs device-specific optimizations [1, 2]
For example, an execution engine built for a Nvidia A100 GPU will not work on a Nvidia T4 GPU 🙃
Your contribution