search

mlc-ai / binary-mlc-llm-libs

200 stars 45 forks source link

add android libs for llama 7b and redpajama 3b #60

Closed Kartik14 closed 9 months ago

Kartik14 commented 9 months ago

This PR adds the following binaries for android:

hb-min commented 9 months ago

@Kartik14
Hi. I got below error while running ./prepare_libs.sh with your RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar from my local linux machine.

[100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/home/git/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o
[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._lib0.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [tvm4j_runtime_packed] Error 2

Have you ever experienced a similar problem?

CharlieFRuan commented 9 months ago

@hb-min Not sure if https://github.com/mlc-ai/mlc-llm/pull/1494 will fix this, but worth to try again after that PR is merged.

Kartik14 commented 9 months ago

@Kartik14 Hi. I got below error while running ./prepare_libs.sh with your RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar from my local linux machine.

[100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/home/git/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o
[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._lib0.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [tvm4j_runtime_packed] Error 2

Have you ever experienced a similar problem?

I haven't encountered this error, but it looks like there might be some issue with tvm. Can you try updating your tvm? If you are using 3rdparty/tvm, you can do it using git submodule update --recursive.

hferoze commented 8 months ago

Hi @Kartik14

Seeing the same issue. Using the latest repo (tvm)

Llama-2-7b-chat-hf-q4f32_1-android.tar [100%] Built target mlc_llm_static [100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o [100%] Linking CXX shared library libtvm4j_runtime_packed.so ld.lld: error: model_lib/libmodel_android.a(._llama_q4f32_1_devc.o): not an ELF file clang++: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1 make[2]: [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2 make[1]: [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2 make: [Makefile:203: tvm4j_runtime_packed] Error 2

Llama-2-7b-chat-hf-q4f16_1-android.tar [100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o [100%] Linking CXX shared library libtvm4j_runtime_packed.so ld.lld: error: model_lib/libmodel_android.a(._llama_q4f16_1_devc.o): not an ELF file clang++: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1 make[2]: [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2 make[1]: [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2 make: [Makefile:203: tvm4j_runtime_packed] Error 2

RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar [100%] Built target mlc_llm_static [100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o [100%] Linking CXX shared library libtvm4j_runtime_packed.so ld.lld: error: model_lib/libmodel_android.a(._lib0.o): not an ELF file clang++: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1 make[2]: [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2 make[1]: *** [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_ru

Kartik14 commented 8 months ago

@hferoze @hb-min I am unable to reproduce the error. Can you try the libs from #82 ? If that does not work, could you please try to recompile the android library from the model weights. The instructions for it are here

Thore1954 commented 8 months ago

@Kartik14 I tried the latest libs but got the same error. I am using the following script:

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"

# Install Android SDK Command-Line Tools
curl -O https://dl.google.com/android/repository/commandlinetools-linux-9123335_latest.zip
unzip commandlinetools-linux-9123335_latest.zip
rm commandlinetools-linux-9123335_latest.zip
mkdir -p android_sdk/cmdline-tools
mv cmdline-tools android_sdk/cmdline-tools/latest
export ANDROID_HOME="$PWD/android_sdk"
export PATH="$ANDROID_HOME/cmdline-tools/latest/bin:$PATH"
sdkmanager --licenses

# Install Android NDK
sdkmanager "ndk;26.1.10909125"

# Intall CMake
curl -sSf https://apt.kitware.com/kitware-archive.sh | sudo sh
sudo apt install -y cmake

# Populate submodules (in case you forgot to pass `--recurse-submodules` to `git clone`)
git submodule update --init --recursive

# Download prebuilt model libraries
mkdir -p dist/libs
curl -Lo dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-android.tar https://github.com/mlc-ai/binary-mlc-llm-libs/raw/main/Mistral-7B-Instruct-v0.2-q4f16_1-android.tar
curl -Lo dist/libs/phi-2-q4f16_1-android.tar https://github.com/mlc-ai/binary-mlc-llm-libs/raw/main/phi-2-q4f16_1-android.tar
curl -Lo dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar https://github.com/mlc-ai/binary-mlc-llm-libs/raw/main/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar
curl -Lo dist/libs/Llama-2-7b-chat-hf-q4f16_1-android.tar https://github.com/mlc-ai/binary-mlc-llm-libs/raw/main/Llama-2-7b-chat-hf-q4f16_1-android.tar

# Set up env vars
export ANDROID_NDK="$ANDROID_HOME/ndk/26.1.10909125"
export TVM_NDK_CC="$ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang"

# Install JDK 17
sudo apt install -y openjdk-17-jdk
export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-amd64"

# Set up env var
export TVM_HOME="$PWD/3rdparty/tvm"

# Install TVM Unity compiler
pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly

# Bundle model libraries
cd ./android/library
./prepare_libs.sh
Thore1954 commented 8 months ago

Would you please update the android apk to reflect mlc-ai/mlc-llm#1640?

Kartik14 commented 8 months ago

@Thore1954 I have just updated the model libs #85. It should fix the error. Can you try again please?

hferoze commented 8 months ago

I can verify the issue is resolved. Thanks @Kartik14