Add a new tokenizer C/C++ API

microsoft / onnxruntime-extensions

onnxruntime-extensions: A specialized pre- and post- processing library for ONNX Runtime

MIT License

323 stars 84 forks source link

Closed wenbingl closed 8 months ago

wenbingl commented 8 months ago

discrepancy on phi-2 model tokenizer unit tests haven't been ported yet.