Closed baponkar closed 1 year ago
Huggingface transformers is a Python package that uses native components, sort of libraries, written in Rust programming language, that speed up some data (pre)processing (mainly tokenization) utilizing the fact that Rust is natively compiled a pretty fast.
The transformers' setup script searches for a (the?) Rust compiler which you don't have on your system.
In order to use this package on Chaquopy, you are going to have to build a so called Python Wheel (a .whl
file), which is a sort of zip archive that gets extracted on the Python's (or virtual environment's) Path and let's you use the package by simply importing it. You can get the all the info how to build your own packages in the server/pypi directory of this repository.
Keep in mind that you'll still have to actually setup a Rust build environment. I was able to compile and build a wheel for tiktoken (sadly haven't got time to test it at all) using rustup (rustup.rs) and installing additional compile targets for Arm/Arm64 architecture.
You can check the build files for tiktoken on my repo (https://github.com/vmitro/chaquopy-experimental/tree/master/server/pypi/packages/tiktoken). Please not it's a clumsy solution, ideally one would edit the setup.py
script and do the necessary changes and preparations in Python but meh.
Thankyou brother! for pointing me the right direction. I have been solved this issue by installing rust in my windows machine.
I have been solved this issue by installing rust in my windows machine.
I doubt that simply installing Rust will be enough on its own. Even if the build completes, it'll probably be a build for Windows which won't run correctly on Android.
I suggest you first try using one of the versions of transformers for which we already have a working tokenizers version (https://github.com/chaquo/chaquopy/issues/608#issuecomment-1107441743). If those versions are too old for you, please post a comment on #607 explaining why.
Sorry! It did not work, I mistakenly removed install "transformers"
then try build and I thought it worksbut now it is not working i.e. gradle building failed.
I have been solved this issue by installing rust in my windows machine.
I doubt that simply installing Rust will be enough on its own. Even if the build completes, it'll probably be a build for Windows which won't run correctly on Android.
I suggest you first try using one of the versions of transformers for which we already have a working tokenizers version (#608 (comment)). If those versions are too old for you, please post a comment on #607 explaining why.
install "transformers==4.15.0"
I think this worked for me as it is showing no error in gradle build.
Thankyou so much @mhsmith
.
Chaquopy 14.0.2
Python 3.8.10
Gradle 7.5
Android Studio 2022.1.1
Relevant parts of your build.gradle file
Inside of build.gradle
I also have tried with include
Describe your issue
When I try to install transformers then build failed with this error:
It will be very helpful for me for any help. Thankyou