search

microsoft / onnxruntime-extensions

onnxruntime-extensions: A specialized pre- and post- processing library for ONNX Runtime
MIT License
323 stars 84 forks source link

add a gen_processing_model option to cast token-id for int64 #632

Closed wenbingl closed 8 months ago

wenbingl commented 8 months ago

The sentencepiece based tokenizer token-id is int32. To be consistent, add an option to cast the output token-id.