Are there any resources or documentation on how to integrate the tokenization part into Android?

huggingface / tflite-android-transformers

DistilBERT / GPT-2 for on-device inference thanks to TensorFlow Lite with Android demo apps

Apache License 2.0

391 stars 81 forks source link

Are there any resources or documentation on how to integrate the tokenization part into Android? #18

Open BENSAFOUAN-Abdelhalim opened 2 years ago

BENSAFOUAN-Abdelhalim commented 2 years ago

I want to integrate the CamembertQuestionAnswering model to Android, do I have to change the tokenization part because it's a different language?

twiceyuan commented 1 year ago

I also have the same question, because I saw that there are tools such as AutoTokenizer in huggingface/transformers in Python to automatically complete the tokenization work. On Android, each type of model has a corresponding work to be implemented, so I also look forward to similar tools.

Colin-Antony commented 6 months ago

I would also like to see if there is a solution to this