Add Android Media NDK video I/O file capture back-end

[komakai]

relates #11952

This pullrequest adds a video I/O file capture back-end for Android

• add necessary CMake config to determine if the Android Media NDK is available (platform must be Android and the API level must be 21 or over)
• add code for decoding video file frames using the Android Media NDK
• add the back-end the videoio registry
• add a config file which specifies API level 21

android_pack_config=ndk-18-api-level-21.config.py

[komakai]

Tried to rebase onto 3.4 but looks like the Video I/O interfaces have changed quite a bit. Not sure about testing - to access the Media NDK APIs the tests would need to run on an Android emulator.

[komakai]

• switched to use std::shared_ptr with deleters where appropriate and to use std::vector instead of raw uint8_t* buffer.
• additional return value checks for API calls
• modified logic to take the first video track for which decoding could be successfully started

[yagnasrinath]

@alalek Really appreciate your work on enabling videoio APIs on android. We are very interested in using it. Do you happen to know the timeline for getting this merged into master?

[alalek]

@yagnasrinath Thanks to @komakai !

Feel free to try this patch and provide feedback.

[yagnasrinath]

@yagnasrinath Thanks to @komakai !

Feel free to try this patch and provide feedback.

Thank you @komakai. I tried out this patch and it seems to work fine for our use cases. Is there anything that we could help you with to get this patch merged into master ASAP?

[komakai]

@yagnasrinath You're welcome I think with the small spelling fix we should be really to go. (At least for an initial version - I will put some notes on #11952 about possible enhancements)

[PhilLab]

@alalek Will this be available in the Android pre-built provided by OpenCV ? Or does it have to be compiled ourselves?

[alalek]

You can try to download artifacts from "Android pack" builder of this PR. Feature should be available for arm64 / x86_64 architectures.

[yagnasrinath]

@komakai @alalek Is there any chance that open() API is supported for android as well?

[komakai]

@yagnasrinath not sure what you mean. VideoCapture.open should be working for files. What are you trying to open?

[komakai]

You can try to download artifacts from "Android pack" builder of this PR

Unless the API LEVEL is set to 21 for that build then probably that wouldn't work.

[yagnasrinath]

@komakai sorry for not being clear, I was talking about support for VideoCapture.open to read from camera device.

[alalek]

@yagnasrinath You may want to subscribe on this issue: #11952

[komakai]

@yagnasrinath please try out https://github.com/komakai/opencv/tree/android-ndk-camera

[yagnasrinath]

@komakai I tried using your PR. What would be the device path that I need to pass? I gave "1" as the device and I see this error NativeCodec: failed to stat file: 1 (No such file or directory)

[komakai]

@yagnasrinath you need to call the open(int index) overload i.e. pass an int instead of a String. Also make sure to build with native API level 24

[yagnasrinath]

@komakai I am not able to open the camera. Here are the errors:

03-25 00:23:51.895 23240 23248 W OpenCV/4.1.0-pre: [ WARN:0] VIDEOIO(ANDROID_NATIVE): trying capture cameraNum=0 ...
03-25 00:23:51.896   179   179 I Camera2ClientBase: Closed Camera 0. Client was: com.foghorn.edge (PID 23240, UID 10061)
03-25 00:23:51.897 23240 23248 W ACameraDevice: CameraDevice: bad count 0 for shading map size
03-25 00:23:51.897   179   179 I CameraService: CameraService::connect call (PID -1 "", camera ID 0) for HAL version default and Camera API version 2
03-25 00:23:51.898   179   179 I Camera2ClientBase: Camera 0: Opened. Client:  (PID 23240, UID 10061)
03-25 00:23:51.898   179   179 I CameraDeviceClient: CameraDeviceClient 0: Opened
03-25 00:23:51.900   179   179 I Camera  : openDevice:0: Opening camera device
03-25 00:23:51.903 23240 23248 E NdkImageReader: AImageReader_getWindow

03-25 00:23:51.906   179   179 D Camera3-Device: Set real time priority for request queue thread (tid 24263)
03-25 00:23:51.907 23240 24259 W NativeCamera: session 0xaf733848 active
03-25 00:23:51.907 23240 23248 W OpenCV/4.1.0-pre: [ WARN:0] VIDEOIO(ANDROID_NATIVE): created, isOpened=1
03-25 00:23:51.907 23240 23248 E NativeCamera: Acquire image failed with error code: -30001
03-25 00:23:51.907 23240 23241 E av-agent: [opencv_player.cc:398] Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.
03-25 00:23:51.907 23240 23241 E av-agent: [av_agent-main.cc:149] Error Handler Called: [Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.]

[komakai]

@yagnasrinath thanks for testing

NativeCamera: Acquire image failed with error code: -30001

Ah - I also got error -30001 (no buffer available) - seems this is not a fatal error just a timing thing. You need to wait until a buffer is available. As a work-around I put in some sleeps (~500ms) between the call to open and the first call to read and between each subsequent call to read. A more robust solution would be to try and find some callback from the NDK indicating when a buffer was available and wait for that before trying to acquire the image.

[yagnasrinath]

@komakai No matter how much ever sleeps I put, I still get read false, after open. I keep getting the same error. Do you think it is better to get the status of repeating request by passing a captureCallback object here https://github.com/komakai/opencv/blob/android-ndk-camera/modules/videoio/src/cap_android_camera.cpp#L367 instead of nullptr? May be that's when we will know that the capture is complete and buffer is available?

[komakai]

@yagnasrinath

Do you think it is better to get the status of repeating request by passing a captureCallback object here https://github.com/komakai/opencv/blob/android-ndk-camera/modules/videoio/src/cap_android_camera.cpp#L367 instead of nullptr? May be that's when we will know that the capture is complete and buffer is available?

Yeah - I think that's the way to go - will try it out when I get a minute

[komakai]

@yagnasrinath I've added some synchronization around the onCaptureCompleted callback and it seems to work without any sleeps - please try it out

[yagnasrinath]

@komakai Unfortunately, I still see the same error.

03-27 12:53:28.904 179 4087 I Camera2ClientBase: Camera 0: Opened. Client: (PID 13249, UID 10061) 03-27 12:53:28.904 179 4087 I CameraDeviceClient: CameraDeviceClient 0: Opened 03-27 12:53:28.906 179 4087 I Camera : openDevice:0: Opening camera device 03-27 12:53:28.908 13249 13257 E NdkImageReader: AImageReader_getWindow 03-27 12:53:28.911 179 4087 D Camera3-Device: Set real time priority for request queue thread (tid 13272) 03-27 12:53:28.912 13249 13268 W NativeCamera: session 0xa3db30d8 active 03-27 12:53:28.912 13249 13257 W NativeCamera: No Buffer Available error occured - waiting for callback 03-27 12:53:30.752 13249 13250 I av-agent: [opencv_player.cc:322] In the past 2.0s we've read 0 frames from "0" and emitted 0 of them to topic "/raw/av" (input 0.0 FPS, output 0.0 FPS) 03-27 12:53:30.913 13249 13257 E NativeCamera: Capture failed or callback timed out 03-27 12:53:30.913 13249 13250 E av-agent: [opencv_player.cc:397] Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.

[komakai]

@yagnasrinath Can you try out the Android NDK camera sample ("basic") at https://github.com/googlesamples/android-ndk/tree/master/camera (just to make sure your device works with Android NDK camera). Also what version of Android are you on?

[yagnasrinath]

I tried the google sample. It seems to works fine. But I could not get the binary built with opencv, working. We are busy with a release. I will try out the code on a different device after the release.

[komakai]

I tried the google sample. It seems to works fine

I tried as much as possible to copy the Google sample code; next step would be to try and work out what my implementation is doing differently. The only thing that comes to mind immediately is that I take the first output resolution available where as the Google sample looks for an output resolution matching the screen resolution

[HaimBendanan]

I have a working c++ opencv program running on windows, and trying to adapt it to run on android. I am using the android SDK, version 3.4.11 and it seems that reading from video files (cv::VideoCapture(videoPath)) doesn't work (the video is an mp4) Does this PR fixed that issue? I read on other threads that recompiling opencv with ffmpeg is needed in order to use cv::VideoCapture(videoPath) on Android - is that still the case?

[komakai]

Does this PR fix that issue?

Yes

recompiling opencv with ffmpeg is needed in order to use cv::VideoCapture(videoPath) on Android - is that still the case?

No - this PR uses the Android Media NDK to do the decoding so you don't need ffmpeg - note that Android Media NDK is only available on Android 5 or higher

[jogiji]

@komakai Sorry to bother you.. Can you let me know whether with this patch i can decode MJPG based network streams on android using opencv, or i still need to compile/build with ffmpeg.. Also if you can point me to the right direction for build steps with ffmpeg using NDK 21 for android ?

[komakai]

@jogiji this patch does not include support for MJPG based network streams

Also if you can point me to the right direction for build steps with ffmpeg using NDK 21 for android ?

Sorry I have no insight into how to build ffmpeg for Android

[MichaelBrasco]

Using the pre-built binaries, I am able to successfully read an mp4 file using the Android Media NDK backend in a 64-bit (arm64-v8a) Android app. However, I need a 32-bit (armeabi-v7a) compatible version and I cannot seem to use VideoCapture normally (i.e. with cv::CAP_ANY) without getting a SIGBUS crash. The only backend that I find works with the 32-bit version is cv::CAP_IMAGES with a sequence of images, but this suffers from a significant performance loss, so it's not feasible in my case. Could you please let me know if there is a reason for this problem or any solutions available?

I have tried this on numerous devices running Android 10 with identical results (build targets API level 28 and uses NDK 21.3).

[komakai]

@MichaelBrasco Can you share a stack trace for the crash? (Building with --debug_info should result in meaningful crash dumps). Also - are there any warnings output during the armeabi-v7a build of cap_android_mediandk.cpp (dodgy casts etc.)

[komakai]

@MichaelBrasco Can you try the following objdump -t android_build/OpenCV-android-sdk/sdk/native/libs/armeabi-v7a/libopencv_java4.so | grep AndroidMediaNdk and compare with: objdump -t android_build/OpenCV-android-sdk/sdk/native/libs/arm64-v8a/libopencv_java4.so | grep AndroidMediaNdk If the output is different then it is possible that armeabi-v7a build isn't being built with the correct ANDROID_NATIVE_API_LEVEL

[utibenkei]

@komakai I've encountered 32-bit app crashes under the same conditions as @MichaelBrasco. I downloaded the pre-built version 4.5.2 binary from OpenCV official, created a project in AndroidStudio, and created a test android app that loads MJPEG video files in the VideoCapture class. It all works fine in the 64-bit (arm64-v8a) build, but in the 32-bit (armeabi-v7a) build, the app crashes. I have attached the stack trace logs for both 64- bit and 32-bit.

opencv452_android_VideoCapture_arm64-v8a_success.txt opencv452_android_VideoCapture_armeabi-v7a_crash.txt

Also, it seems that the same issue was talked about in this thread. https://github.com/opencv/opencv/issues/11995#issuecomment-522449060

I am also looking for a solution to this problem.

[alalek]

that loads MJPEG video files in the VideoCapture class

not related to this feature (there is dedicated own MJPEG encoder, Media NDK I/O requires higher Android SDK level which is not enabled in the mentioned distribution).

Try to build Android SDK package with debug information. Check logs from original build on public CI.

[utibenkei]

The crash in the 32 bit app seems to occur regardless of the type of video file. It may not have been appropriate to post this topic in this thread, but it seems that VideoCapture class does not work with 32-bit apps.

[komakai]

@utibenkei The prebuilt binaries target an NDK version lower than the NDK version required for NDK media support. (Because Android's arm64-v8 support was added quite late the arm64-v8 build ignores NDK versions that are too low and automatically chooses a higher NDK version that it turns out does support NDK media.) To get a build that will work on armeabi-v7a and arm64-v8 please try building with the following command line: ./opencv/platforms/android/build_sdk.py --use_android_buildtools --no_samples_build --config ./opencv/platforms/android/ndk-18-api-level-21.config.py opencv_android_ndk21 opencv

Alternatively you could try to persuade @alalek to change the build configuration for the standard OpenCV Android releases such that the NDK target was set to 21 or higher

[MichaelBrasco]

@komakai Apologies for the late reply. My logcat output didn't show a particularly helpful stack trace and the object dumps of the armeabi-v7a binaries have references to the Android Media NDK just like the arm64-v8a binaries.

I'm also using VideoCapture to read and decode video frames from a file in a 32 bit app like @utibenkei , not a camera. I finally managed to stumble upon the fix by specifying the VideoCaptureAPI to CAP_ANDROID. This doesn't seem to cause a difference on arm64-v8a as it's the implicit default, but on armeabi-v7a you will get a crash if you don't provide this argument explicitly. I don't know why this is the case, but it is made even more baffling by the fact that the documentation lists CAP_ANDROID as "not used", which I assume is outdated? There is also a fairly significant performance hit on 32-bit in decoding time, but this is perhaps inevitable.

After finding the fix for this fatal issue, another stream of issues ensued. In fairness, I don't think these issues arise from OpenCV's implementation of the Media NDK; my only concern was some of the VideoCapture CAP_PROP_XXX properties were not properly set, so e.g. you cannot seek/rewind videos. On the Samsung S21 we saw a strange green frame flashing every time the video was reset. On some devices, it would just not read the videos at all. We finally had to resort to building OpenCV with FFMPEG for Android. This is a hellish process and is not technically permitted in the CMakeLists, but it is far more reliable and truly cross-platform unlike the Media NDK, where you just have to cross your fingers that the vendor's implementation will be compatible with whatever video format you're using (MP4 AVC is always a safe bet), or it might just not work at all. Here's a guide for anyone wishing to go down this path: https://www.programmersought.com/article/44306233033/

Sorry for hijacking this thread a bit, but this issue has cost me weeks if not an entire month of time on my project, so I hope no one else would have to go through all these steps again.

[komakai]

@MichaelBrasco Thanks for the write-up. It would be great if you could prepare a Pull-Request that addresses any of (or all of!) the troubles you encountered in your project.

[kikaxa]

The colors seem to be swapped when decoding at least mp4 files(android only) wrt. other platforms. As far as i know this stems from the historical swap of the U/V planes.

[komakai]

The colors seem to be swapped when decoding at least mp4 files(android only) wrt. other platforms.

Current implementation outputs BGR - if you are expecting RGB then yeah, the colors are swapped. I think the correct thing to do is implement the CAP_PROP_FOURCC property and then people can just choose the output format they want

[kikaxa]

I don't think this is the problem - we definetely expect BGR and the colors are right on linux, macos, windows and ios.

After some heavy digging around i found a pretty detailed discussion exactly on this convoluted badly-documented topic of android formats: https://community.khronos.org/t/omx-color-formatyuv420semiplanar-yuv420planar-fourcc-fmts/3807/5

Here now we have: COLOR_FormatYUV420SemiPlanar -> cv::COLOR_YUV2BGR_NV21 OLOR_FormatYUV420Planar -> cv::COLOR_YUV2BGR_YV12

The topic suggests: OMX_COLOR_FormatYUV420SemiPlanar — NV12 OMX_COLOR_FormatYUV420Planar — IYUV or I420 OMX_COLOR_FormatYVU420SemiPlanar — NV21 OMX_COLOR_FormatYVU420Planar — YV12

through fourcc.org we can verify this mapping seems to be correct e.g. https://www.fourcc.org/pixel-format/yuv-nv21/

credits for the right search direction: https://answers.opencv.org/question/61628/android-camera2-yuv-to-rgb-conversion-turns-out-green/

[komakai]

@kikaxa thanks for the very thorough research. It would be great if you could put together a PR to fix the mapping.