Does LightSeq support ONNX export and Triton Inference Server?

bytedance / lightseq

LightSeq: A High Performance Library for Sequence Processing and Generation

Other

3.17k stars 328 forks source link

Open stevezheng23 opened 1 year ago

stevezheng23 commented 1 year ago

Hi team, QQ: does lightseq support the followings,

Convert HuggingFace BERT/RoBERTa models to int8 precision directly
If yes, can the converted model be exported to ONNX format directly?
If so, can the exported ONNX model loaded correctly using Triton Inference Server?

neopro12 commented 1 year ago

Converting without calibration or finetune will cause loss of accuracy. So currently not supported.