Android build error - Cannot find flatbuffers.h

[v-prgmr]

Thank you for the amazing work. I cannot build it for android from source. I get build errors in android studio stating "flatbuffers/flatbuffers.h" is not found.

[nyadla-sys]

Would you be able to download the file located at https://github.com/usefulsensors/openai-whisper/blob/main/android_app/android_whisper_app.zip, extract its contents, and then transfer the "flatbuffer" and "absl" folders from the directory "/Whisper-TFLIte-Android-Example/app/src/main/cpp/tf-lite-api/include/ to android studio corresponding directory

[nyadla-sys]

other method would be to just git clone below two abseil and flatbuffers as part of ~/Whisper-TFLIte-Android-Example/app/src/main/cpp/tf-lite-api/include/ path https://github.com/abseil/abseil-cpp.git https://github.com/google/flatbuffers.git

[v-prgmr]

@nyadla-sys I have got this build error now. /home/usr/workspace/android-apps/openai-whisper/android_app/Whisper-TFLIte-Android-Example/app/src/main/cpp/tf-lite-api/tensorflow_src/tensorflow/lite/core/async/async_subgraph.h:165:57: error: copying member subobject of type 'std::atomic<TfLiteBufferHandle>' (aka 'atomic<int>') invokes deleted constructor

Any idea how to get past this?

[v-prgmr]

@nyadla-sys I used cloned the tensorflow repo into tensorflow_src under tflite-api and tried building which lead to the above error. Just as you suggested for flatbuffers and abseil, I copied your tensorflow_src DIR instead and now the build succeeded.

Thank you for the quick response :)

[nyadla-sys]

@v-prgmr if you have any specific steps for new Android app builders to use Whisper TFLite, could you please post a README?

[v-prgmr]

@v-prgmr if you have any specific steps for new Android app builders to use Whisper TFLite, could you please post a README?

Will do it, I am fairly new in the android space, so getting familiar with the TFlite CPP API and NNAPI. After a couple more experiments, I will sum them up in a useful way and post a README.

Meanwhile, did you try the CPU/ GPU delegation using the NNAPI to see if the model performed better on the GPU?

[v-prgmr]

@nyadla-sys I see that you have commented out the GPU delegation part, inorder for the GPU delegation to work, did you build the project by linking to the libtensorflow_gpu_delegate.so ? Now uncommenting your gpu delegation part of the code gives me error: undefined reference to 'TfLiteGpuDelegateV2Create'

[nyadla-sys]

@fquirin @StuartIanNaylor @j1nx Please provide if you have any inputs on this to Vishal

To enable the GPU delegate, it is necessary to construct a float tflite model. You can generate the float tflite model by running the following notebook: https://colab.research.google.com/drive/1Rj-C9SjetfS8EeCuSVwmZ1-ev1zyiKM7?usp=sharing. Once you have generated the model, you will need to build the necessary GPU delegate libraries and include them as part of the Android build system. To build the static library, run the following command using Bazel: bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:delegate

For the dynamic library, run the following command: bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so

For more information, please refer to the following link: https://www.tensorflow.org/lite/android/delegates/gpu#quantized_models.

[StuartIanNaylor]

Haven't tried anything with NNAPI on Android apols

[fquirin]

Me neither, sorry. Been working with the Python version mostly (which is C+ under the hood probably, but I usually don't touch that).

[v-prgmr]

@nyadla-sys I built the libtensorflowlite_gpu_delegate.so and added it to the cmake and built the app. And now the app crashes at auto* delegate = TfLiteGpuDelegateV2Create(nullptr);. Debugging this now.

UPDATE: I got around the crash at the delegation. And also your LOG statement "gpu delegate failed" does not appear, seems to be like the model is now running on the GPU or I still have note found out if its really running on the GPU cause while profiling the app, it seems like the CPU load is same for when I have uncommented the GPU delegation and commented the GPU delegation out.

UPDATE2: And here in the [documentation](https://www.tensorflow.org/lite/ios/delegates/gpu#c++:~:text=When%20using%20TensorFlow%20Lite%20with%20the%20GPU%20delegate%20with%20C%2B%2B%2C%20get%20the%20GPU%20delegate%20via%20the%20TFLGpuDelegateCreate()%20function%20and%20then%20pass%20it%20to%20Interpreter%3A%3AModifyGraphWithDelegate()%2C%20instead%20of%20calling%20Interpreter%3A%3AAllocateTensors().) it says to use ModifyGraphWithDelegate() instead of AllocateTensors(). So I even commented your line // Allocate tensor buffers. //TFLITE_MINIMAL_CHECK(g_whisper_tflite_params.interpreter->AllocateTensors() == kTfLiteOk); Now this should definitely mean that the model is running on the GPU or ??

[nyadla-sys]

I hope you are using the TFLite float model for the GPU delegate, and please share the ADB logcat so that I can check if it is delegated to the GPU or not. The simplest way is to test it with a minimal example and then try it on a mobile phone.

[v-prgmr]

Yes I did convert to float using the ntebook. The model size turned out to be 150mb. Which part of the logcat are you interested in cause it is a huge dump.

[nyadla-sys]

@v-prgmr you can capture the logcat messages while running the whisper app.

[v-prgmr]

Yes but the logcat is really huge...I though you might be interested in a specific tag. I will post the entire logcat then.

[v-prgmr]

Apparently, the GPU delegate does not work. In the logcat, I found gpu delegate failed. I built the dynamiclibrary using bazel and added the library to Cmakelist. But I built the GPU library from a different linux system than the one where I am running the app. I am wondering if I have mixed some configs between the two builds.

[nyadla-sys]

is it possible to build GPU delegate lib as part of the Linux system where you run the app.?

[v-prgmr]

Safe to say that the different Linux systems was not the problem. I rebuilt the library again using a VM with the exact same jdk, ndk and sdk versions. Somehow the delegate does not work. I tried the tflite object detection example and tried NNAPI and GPU delegation there, and it seems to work. But they use the java implementation and not the tflite cpp API. Were you able to run this on GPU?

[nyadla-sys]

I haven't tried gpu delegate yet.

[nyadla-sys]

@v-prgmr Some of the Ops in the model may not be supported by the GPU, which could be causing the GPU delegate to fail

[v-prgmr]

@nyadla-sys I have read a couple of blogs and tried different attempts at getting the f32 model to run on GPU, but the GPU delegate still fails.