microsoft / BlingFire

A lightning fast Finite State machine and REgular expression manipulation library.
MIT License
1.82k stars 127 forks source link

Could java call the tokenizer of bin file #157

Open MichaleDong opened 1 year ago

MichaleDong commented 1 year ago

Hi, we use java as online service, could java call the bin file of tokenizer roberta ? I mean call the function directly. Like in python:

from blingfire import *

text = 'After reading this post, you will know: What "natural language" is and how it is different from other types of data. What makes working with natural language so challenging. [1]'

print(text_to_sentences(text)) print(text_to_words(text))