search

microsoft / onnxruntime-extensions

onnxruntime-extensions: A specialized pre- and post- processing library for ONNX Runtime
MIT License
295 stars 80 forks source link

[Experimental] Add GPTNeoXTokenizer support to HFTokenizerConverter #678

Open kazssym opened 3 months ago

kazssym commented 3 months ago

This pull request adds experimental GPTNeoXTokenizer support.