Closed C00reNUT closed 1 year ago
Do you have any idea what could be the problem?
Hello thanks for the feedback!
I already read that kind of thing, it could come from onnx runtime library which doesn't optimize well for gpt models.
Check this issue which is related to T5 but i think it applies to gpt models: https://github.com/microsoft/onnxruntime/issues/6835
Take also a look to this: https://github.com/microsoft/onnxruntime/issues/11293
Hello, thank you for this tutorial, I have tried to modify the code in order to use the text generation pipeline with gpt2 model. The problem is that the performance of vanilla Pytorch is better than ONNX optimized models. This is true for my home setup and also on colab pro with T4 and P100 GPUs.
I have also tried text generation pipeline in https://github.com/AlekseyKorshuk/optimum-transformers library, but the results are similar - the ONNX performance is still slower.
Do you have any idea what could be the problem?