HuggingFaceTokenizer: add support for Android - Githubissues

deepjavalibrary / djl

An Engine-Agnostic Deep Learning Framework in Java

https://djl.ai

Apache License 2.0

4.16k stars 661 forks source link

HuggingFaceTokenizer: add support for Android #3531

Closed naveen521kk closed 6 days ago

naveen521kk commented 6 days ago

Description

This PR fixes https://github.com/deepjavalibrary/djl/issues/2170.

This PR does the following:

Add scripts to build the Rust library in extensions/tokenizers/rust using cargo-ndk
Disable the "http" feature for Rust library when building with Android
Create a wrapper android package to use the built libraries to publish, like pytorch-native.

CI support for this is yet to be figured out.