search

google / gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.
Apache License 2.0
5.9k stars 500 forks source link

Gemma.cpp on the Android arm64-v8a #62

Closed urim85 closed 6 months ago

urim85 commented 6 months ago

I have made Gemma executable to run on Android arm64-v8a with below option.

cmake -DCMAKE_TOOLCHAIN_FILE=/usr/lib/android-sdk/ndk/25.1.8937393/build/cmake/android.toolchain.cmake .

And it runs on my Android. However there is error for reading weights.

/# gemma --tokenizer tokenizer.spm --compressed_weights 2b-it-sfp.sbs --model 2b-it cache.path.c_str() : 2b-it-sfp.sbs 2b-it-sfp.sbs:3163184640 BlobReader::Open open BlobReader::Open Read BlobReader::Open Allocate 3163184640, 0 Cached compressed weights does not exist yet (code 155), compressing weights and creating file: 2b-it-sfp.sbs.

There is a failure in the "BlobReader::Open" function, on the line "blobstore->CheckValidity(IO::FileSize(filename))". The expected value of "IO::FileSize(filename)" is 3163184640, but it is currently returning 0.

However, before the line "hwy::CopySameSize(&bs, blobstore.get())" in the "BlobReader::Open" function, the "IO::FileSize(filename)" is functioning correctly.

Anyway, The comments in https://github.com/google/gemma.cpp/issues/21, it should work well.

Could you help me to figure out what I missed?

jan-wassenberg commented 6 months ago

Hi, the error codes are simply line numbers, but it looks like your code is quite different. On both main and dev, if (file_size_ != file_size) return __LINE__; should be line 218, not 155. Is the code modified?

I think the issue is that off_t on Android defaults to 32-bit. You can build with -D_FILE_OFFSET_BITS=64 which fixes it for 15c and above. We'll also change to lseek64, thanks for pointing this out :)

jan-wassenberg commented 6 months ago

I think this is fixed now, please reopen if not.

urim85 commented 6 months ago

It works well!!

But sentencepiece with add_definitions(-D_FILE_OFFSET_BITS=64) has an error.

/usr/lib/android-sdk/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/fstream:968:9: error: use of undeclared identifier 'fseeko' if (fseeko(_file, __sp, SEEK_SET))

I put the definition like this

FetchContent_Declare(sentencepiece GIT_REPOSITORY https://github.com/google/sentencepiece GIT_TAG 53de76561cfc149d3c01037f0595669ad32a5e7c) FetchContent_MakeAvailable(sentencepiece)

add_definitions(-D_FILE_OFFSET_BITS=64)

FetchContent_Declare(highway GIT_REPOSITORY https://github.com/google/highway.git GIT_TAG da250571a45826b21eebbddc1e50d0c1137dee5f) FetchContent_MakeAvailable(highway)

Thank you very much!

jan-wassenberg commented 6 months ago

Please help us out by searching for such error messages on the web :) This one is mentioned in https://developer.android.com/ndk/guides/common-problems, it seems we have to raise minSdkVersion to 24.