[Bug/Usage Issue]: Openvino 2024 on Android - x86_64 and arm64 fails to read/compile model resulting in existing public demo projects not working anymore

keyur-esper commented 3 months ago

OpenVINO Version

master

Operating System

Android System

Device used for inference

CPU

Framework

None

Model used

ssdlite_mobilenet_v2 / https://huggingface.co/dkurtaev/wav2vec2-base-ft-keyword-spotting/tree/main

Issue description

audio_recognition_android no longer works neither does the coco_detection_android_demo ever since the replacement of old arm plugin

Both of these demo projects have stopped working since these two commits that removed old arm plugin and replaced it by adding arm support in existing intel_cpu module https://github.com/openvinotoolkit/openvino_contrib/commit/4d35f32a797d0fbee38910bfedb127217ed9f3c9 https://github.com/openvinotoolkit/openvino/commit/c283d212151f38f00446db0fe43945e228193176

I am able to run them both without any issues when I go one commit before the above commits on both x86_64 and arm64

Step-by-step reproduction

Here are my build commands for arm64

export OPENVINO_ROOT=/root
export CMAKE_VERSION=3.22.1
export CMAKE_VERSION=3.22.1
export NDK_DIR=$OPENVINO_ROOT/sdk/ndk/25.1.8937393
export CMAKE_DIR=$OPENVINO_ROOT/sdk/cmake/3.22.1
cd oneTBB && git checkout CMakeLists.txt && cd ..
sed -i -E 's|sources/cxx-stl/llvm-libc\+\+/libs/\$\{ANDROID_ABI\}|toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android|' oneTBB/CMakeLists.txt
${CMAKE_DIR}/bin/cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS="-Wno-error=attributes" \
-DTBB_TEST=OFF \
-DANDROID_ABI=arm64-v8a \
-DANDROID_PLATFORM=26 \
-DANDROID_STL=c++_shared \
-DCMAKE_TOOLCHAIN_FILE=${NDK_DIR}/build/cmake/android.toolchain.cmake \
-DCMAKE_MAKE_PROGRAM=${CMAKE_DIR}/bin/ninja \
-DANDROID_USE_LEGACY_TOOLCHAIN_FILE=OFF \
-GNinja \
-S oneTBB \
-B oneTBB_build_arm64

${CMAKE_DIR}/bin/cmake --build oneTBB_build_arm64 -j$(nproc --all)

${CMAKE_DIR}/bin/cmake --install oneTBB_build_arm64 --prefix oneTBB_install_arm64
${CMAKE_DIR}/bin/cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=26 \
  -DANDROID_STL=c++_shared \
  -DCMAKE_TOOLCHAIN_FILE=${NDK_DIR}/build/cmake/android.toolchain.cmake \
  -DCMAKE_MAKE_PROGRAM=${CMAKE_DIR}/bin/ninja \
  -DANDROID_USE_LEGACY_TOOLCHAIN_FILE=OFF \
  -DENABLE_INTEL_MYRIAD=OFF \
  -DENABLE_CLDNN=OFF \
  -DENABLE_OPENCV=OFF \
  -DENABLE_PLUGINS_XML=ON \
  -DENABLE_TEMPLATE=OFF \
  -DENABLE_SAMPLES=OFF \
  -DTHREADING=TBB \
  -DTBB_DIR=$(realpath oneTBB_install_arm64/lib/cmake/TBB) \
  -DBUILD_nvidia_plugin=OFF \
  -DIE_EXTRA_MODULES=$(realpath openvino_contrib/modules) \
  -GNinja \
  -S openvino -B openvino_build_arm64
${CMAKE_DIR}/bin/cmake --build openvino_build_arm64 -j256
${CMAKE_DIR}/bin/cmake --install openvino_build_arm64 --prefix openvino_install_arm64
cp ${NDK_DIR}/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so openvino_install_arm64/runtime/lib/aarch64
cp oneTBB_install_arm64/lib/*.so openvino_install_arm64/runtime/lib/aarch64

${NDK_DIR}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip openvino_install_arm64/runtime/lib/aarch64/*.so

cd $OPENVINO_ROOT/openvino_contrib/modules/java_api
source ../../../openvino_install_arm64/setupvars.sh
unset ANDROID_HOME
export ANDROID_SDK_ROOT=$OPENVINO_ROOT/sdk/
mkdir -p src/main
cd $OPENVINO_ROOT/openvino_contrib/modules/java_api/src/main
wget https://raw.githubusercontent.com/dkurt/openvino_java/main/AndroidManifest.xml
cd ../..
rm -rf build.gradle
wget -O build.gradle https://raw.githubusercontent.com/dkurt/openvino_java/main/build_android.gradle
mv $INTEL_OPENVINO_DIR/runtime/lib/aarch64 $INTEL_OPENVINO_DIR/runtime/lib/arm64-v8a

../../../gradle-7.4/bin/gradle clean build -x test -x lint --info

I am using the aar generated at $OPENVINO_ROOT/openvino_contrib/modules/java_api/build/outputs/aar/openvino-release.aar inside the application code taken from the following two places audio_recognition_android - https://github.com/dkurt/audio_recognition_android coco_detection_android_demo - https://github.com/openvinotoolkit/openvino_contrib/tree/6aa7a163c81f36cc5cc6166cc7cb0ff64967e5d1/modules/arm_plugin/demos/coco_detection_android_demo

Relevant log output

Logs are as follows - 
audio_recognition_android -
          InferenceEngineException: 
            CompileModel: Exception from src/inference/src/cpp/core.cpp:109:
          Exception from src/inference/src/dev/plugin.cpp:54:
          Exception from src/plugins/intel_cpu/src/nodes/matmul.cpp:303:
          MatMul node with name '304' has invalid dims count

coco_detection_android_demo - 
03-31 10:17:12.839 14784 14784 E AndroidRuntime: InferenceEngineException: 
03-31 10:17:12.839 14784 14784 E AndroidRuntime:    Build: Check 'input_et == element::f32 || input_et == element::f16 || input_et == element::i8 || input_et == element::bf16 || input_et == element::u8 || input_et == element::i64 || input_et == element::i32 || input_et == element::dynamic' failed at src/core/src/op/util/interpolate_base.cpp:61:
03-31 10:17:12.839 14784 14784 E AndroidRuntime: While validating node 'opset11::Interpolate Interpolate_1470 (opset1::Parameter image_tensor[0]:u3[1,300,3,300], opset1::Constant Constant_1468[0]:i64[2], opset1::Constant Constant_1469[0]:i64[2]) -> (dynamic[...])' with friendly_name 'Interpolate_1470':
03-31 10:17:12.839 14784 14784 E AndroidRuntime: Input element type must be f32, f16, bf16, i8, u8, i64, i32

Full logs are available here - 
coco_detection_android_demo - https://gist.github.com/keyur-esper/23506756745d1efc5f80436e731333a1
audio_recognition_android - https://gist.github.com/keyur-esper/d92059dc56479b9a641d9fe33a99006e

Issue submission checklist

[X] I'm reporting an issue. It's not a question.
[X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
[X] There is reproducer code and related data files such as images, videos, models, etc.

myan-o commented 1 month ago

Please do something about it quickly.

alvoron commented 1 month ago

My colleague @allnes is recovering Android sample.

@allnes could you please share updated Android instruction as soon as it's ready?

allnes commented 1 month ago

My colleague @allnes is recovering Android sample.

@allnes could you please share updated Android instruction as soon as it's ready?

yes, we are in the process of updating instructions related to android build and coco sample, when all updates are done I will write here

allnes commented 3 weeks ago

@keyur-esper @myan-o hi, sample and new instrcutions is ready - please use this PR before merge

openvinotoolkit / openvino