search

opencv / opencv

Open Source Computer Vision Library
https://opencv.org
Apache License 2.0
75.95k stars 55.62k forks source link

C++ cv::VideoCapture.open(0) always return false on Android #11952

Closed omatrot closed 3 years ago

omatrot commented 5 years ago
System information (version)
Detailed description
Context

I'm writing a cross platform OpenCV based C++ library. The consuming code is a React Native Application through a react native native module.

To be perfectly clear, there is no access from Java Code to C++ OpenCV on Android. There are events with the result of the OpenCV C++ code sent to Javascript through the react native bridge.

My native library is compiled on Android as a SHARED library. It is dynamically linked to the libopencv_world.so that is produced by the compilation of OpenCV C++ for Android.

What it does

Basically, it opens the device's default camera and take snapshots.

The outcome

This code is then ran on iOS and Android.

This is working perfectly well on iOS. It fails on Android.

Here is the failing part of C++ code on Adndroid:

Steps to reproduce
// cap is a cv::VideoCapture object    
if (cap.open(0))
        {
            cap.set(cv::CAP_PROP_FRAME_WIDTH, CAM_WIDTH);
            cap.set(cv::CAP_PROP_FRAME_HEIGHT, CAM_HEIGHT);
        }
        else
        {
            reject("false", "cap.open(0) returned false");
        }
alalek commented 5 years ago

C++ VideoCapture camera interface is not supported on Android platform:

Usage questions should go to Users OpenCV Q/A forum: http://answers.opencv.org

omatrot commented 5 years ago

So basically, what are my options on this platform ? FYI, I asked on the Q/A forum. No answer.

mshabunin commented 5 years ago

Native camera interface has been added in Android 7: https://developer.android.com/ndk/guides/stable_apis

potrykus commented 5 years ago

Following the previous comment, I think it is appropriate to turn issue 11952 into an [important] feature request.

Here is a relevant blurb on the Android 7 API (level 24):

The native camera API provides the native equivalent of the android.hardware.camera2 classes in the Java programming language. The native camera API lets you perform fine-grained photo capture and processing in your native code.

Instability, or non-constancy cross-android device, was an issue for OpenCV implementations from 2.4 on. This is a stable API, so there should be no impediment for OpenCV implementation.

Given android market shares, having (at least) the same capability as we have with iOS is hugely valuable.

This could be an experimental development, first just getting the CAP_ANDROID flag usable at the level of the videoCapture tutorial (e.g., through the >> operator). Further exposure of native-camera control bits can occur in follow-on (minor) versions. In some sense, the development would never have to leave "experimental" status (for the foreseeable future).

I don't know iOS native AVFoundation, nor do I know the openCV iOS videoCapture implementation, but given the Swift AVFoundation's handling of callbacks, I imagine a port of the openCV iOS videoCapture implementation (to resolve this issue) is "more than just" feasible. Further, the Android native-camera (level 24) API is actually cleaner than that of iOS. This makes implementation even easier. Here is a useful example of the 4 (only) callbacks necessary for hooking into the native-camera video stream: https://github.com/justinjoy/native-camera2/blob/master/app/src/main/jni/native-camera2-jni.cpp N.B., this code should not be intimidating (esp. for C++!): it is almost 100% glue code.

(Note SDK N = Nougat = Android 7 = API 24.)

As a last point, justinjoy's native-camera2 project above has the goal of gluing the native camera into a java driver through JNI. The parallel with the OpenCV goal is pretty strong: replace "through-JNI driver calls" with "through OpenCV impl to public cv::videoCapture API".

I'll save design for another time, but I think the only real issue is going from the push-based (native API) callback scheme to a pull-based stream (>>) operation. A performant bridge could go cross-thread via a std::deque protected by a tight condition/semaphore/mutex (safe, but avoid deadlock - on empty). This would essentially be a (bounded) blocking (double-ended) queue with no new dependencies (outside the android system lib).

potrykus commented 5 years ago

Great. Here is some more engineering detail on a blocking-queue "bridge." There is a cottage industry in these things (because they're useful, and the std library doesn't have one).

Importantly, not implementing a bound on the de queue will create a memory leak.

The only technical gotcha here is that the snippets rely [elegantly] on RAII C++ locking. I don't know if that is available in the Android native SDK, but with NDK r16 libc++ is no longer beta, so I suppose that means it's feature-complete.

Even more importantly: The code in the snippets should be re-written because - while it is really basic - none have copyright info on it: OpenCV needs its own source impl.

lapinozz commented 5 years ago

Would this let us use cv::VideoCapture on Android with both Camera2 and Camera1(LEGACY) cameras? That'd be awesome!

githubdoramon commented 5 years ago

I am a little lost here... does that means that for now we can't use OpenCV to capture camera images on Android newest versions?

email2jie commented 5 years ago

Would like to know if there are any other options at the moment as well.

mshabunin commented 5 years ago

You can use camera2 in Java (since 5.0) or NDK Camera in C++ (since 7.0).

komakai commented 5 years ago

14005 implements a native back-end for video file decoding (but not camera capture)

It requires API LEVEL to be set to 21 for example by building the OpenCV Android SDK with platforms/android/build_sdk.py --config ndk-18-api-level-21.config.py {...other arguments}

A possible improvement would be dynamically checking the presence of libmediandk.so at runtime using dlopen and then use dlsym to lookup all the necessary APIs. This approach would mean you could still get the native file decoding without having to set the API LEVEL to 21. You could also then add a Java implementation for pre-Lollipop Androids that would kick in if the native implementation wasn't available. A Java implementation should be pretty simple using the following for guidance: https://github.com/googlesamples/android-BasicMediaDecoder https://github.com/google/grafika (I haven't checked but OffscreenSurface.java might be useful)

alalek commented 5 years ago

@komakai Thank you for adding support for reading of video files!

dynamically checking the presence of libmediandk.so

There is attempt to reduce required external dependencies and make them optional with plugins approach (see #13677).

zmdsjtu commented 4 years ago

Is it possible to open camera using cv::videoCapture(0) in android now?

komakai commented 4 years ago

Is it possible to open camera using cv::videoCapture(0) in android now?

Not yet. If you are interested in this feature then please try out the branch below and let me know if it works for you: https://github.com/komakai/opencv/tree/android-ndk-camera

LankyBin-lb commented 4 years ago

Is there any ways to get camera buffer to instead "capture >> mat",i'v tried android camera buffer.But the data of buffer is ataxic when I use "Mat frame(720, 1280, CV_8UC1, buffer)"

AI-Decay commented 3 years ago

Did this feature appear after version version: 4.1.1 or higher?

komakai commented 3 years ago

@AI-Decay if you are interested in this feature then please try out https://github.com/komakai/opencv/tree/android-ndk-camera and give feedback.

jumpy88 commented 3 years ago

Has anyone tested @komakai code?

jumpy88 commented 3 years ago

Hello @komakai, I'm interested in camera capture feature and I'd like to try your branch. I'm working in linux. I've android-ndk-r21d installed but I can't figure out how to build your source code. Can you give me some hints? Thank you in advance.

komakai commented 3 years ago

Hello @jumpy88 The branch was quite old so I have synced it with master: 9315e45 In order to build of course you need the Android SDK and Android NDK installed and you need the ANDROID_SDK environment variable set up to point to the Android SDK directory. And of course you need to check out my branch, for example like: git clone https://github.com/komakai/opencv.git git co android-ndk-camera Then the following should build the OpenCV Android with NDK Camera support (call from the directory below the opencv checkout) ./opencv/platforms/android/build_sdk.py --use_android_buildtools --no_samples_build --config ./opencv/platforms/android/ndk-18-api-level-24.config.py android_build opencv

I will update later with info how to actually test once you get a build out

jumpy88 commented 3 years ago

Thank you @komakai for your support. I just need to build libopencv_java4.so to link it in a Godot application, so I'll not need to write any Java code line. Do I also need Android SDK and not only Android NDK? I see in python script name "ndk-18". Does it mean ndk r21 is still not supported and should I install a previous version? Sorry for silly questions, I'm very inexperienced in Android development.

komakai commented 3 years ago

Hello @jumpy88

Do I also need Android SDK and not only Android NDK?

Currently platforms/android/build_sdk.py has no option to just build the native part so you will need the Android SDK as well.

I see in python script name "ndk-18". Does it mean ndk r21 is still not supported and should I install a previous version?

It's no problem to use a newer NDK - the number in the script name was originally required because earlier versions of the NDK had different toolchain names. The "ndk-18" script doesn't specify a toolchain name and the build script just defaults to clang. As a result of this the "ndk-18" script works with newer NDKs as well.

For my own testing I created an app here: https://github.com/komakai/NdkCamtest feel free to use that to test as well. In my testing I found 2 issues:

jumpy88 commented 3 years ago

Hello @komakai, I'm close but something still goes wrong: it builds up to 880/890 (or to the end but I can't see because is too fast) but then it fails with this error

FAILURE: Build failed with an exception.

* What went wrong: A problem occurred configuring project ':face-detection'. > NDK not configured. Download it with SDK manager.

Obviously I installed NDK using SDK Manager inside Android Studio and I tried to set it both as ANDROID_NDK environment variable and --ndk_path parameter but nothing seems to work. What should I do to make it work?

komakai commented 3 years ago

I think if you pass --no_samples_build then that error will go away Alternatively try setting ANDROID_NDK_HOME - I think that also should solve it

jumpy88 commented 3 years ago

--no_samples_build is not accepted. It makes the script print the usage. I'm trying different configurations for ANDROID_NDK, ANDROID_NDK_ROOT, ANDROID_NDK_HOME, --sdk_path, --ndk_path...but can't find a working setup. Where are they supposed to point to?

komakai commented 3 years ago

--no_samples_build is not accepted.

Can you do git rev-parse HEAD and check the output starts 9315e45 ?

jumpy88 commented 3 years ago

Ops...you are right, I messed up my local repository after your recent merge with master. I have redownloaded the repository and checked out to the right branch (and checked the commit hash is correct 9315e45b54d4cb4fd14f021a29732946460265d9). Now it accepts --no_samples_build but I'm still doing something wrong:

FAILURE: Build failed with an exception.

* What went wrong: A problem occurred configuring project ':opencv'. > java.lang.NullPointerException (no error message)

komakai commented 3 years ago

Hmm - I believe that error happens when gradle can't find your JDK. Make sure you have the JDK installed and JAVA_HOME set

jumpy88 commented 3 years ago

Ok, I solved this by adding --sdk_path and --ndk_path together with --no_samples_build. Now I've my four libopencv_java4.so files (one for each ABI). I'll make you know if they do the work for my project. Thank you for your help

komakai commented 3 years ago

@jumpy88 OK let me know how you get on - please send LogCat output if you hit any issues

jumpy88 commented 3 years ago

@komakai my apk starts but it doesn't show any data from USB camera (the same project linked to x86_64 OpenCV library and exported for linux works fine). I also tried to require all permissions but, when I start it, it neither ask me for permission grant. I looked for a way to filter logcat output only to show my apk relevant information, but I couldn't find any solution. Do you have any suggestion to do it?

edit: further info when I connect my USB camera no new /dev/video device seems to appear, but I know this application can accede it through /dev/bus/usb/001/003

komakai commented 3 years ago

any data from USB camera

Ah - a USB camera. Then I don't think this will work. Android NDK Camera API (which is what I am using to implement VideoCapture.open()) doesn't know about external USB cameras. I recommend you take a look at https://github.com/saki4510t/UVCCamera

jumpy88 commented 3 years ago

Great, a weekend thrown in the rubbish (and uselessly stolen a lot of your time) 😆 Anyway, I learnt at least something new about OpenCV and Android C++ libraries building. For completeness: color blob detect sample from OpenCV ask for camera permission but it says "It seems that your device does not support camera (or it is locked). Application will be closed."

I recommend you take a look at https://github.com/saki4510t/UVCCamera

Yes, thank you for the suggestion. I already knew about it (it is the library used by the application linked above) and it was already my second planned attempt. It wasn't the first one because I'll also need OpenCV for computation, I know a little of OpenCV whereas never used UVCCamera and it seems not to be an active repository since October 2018. Thank you so much @komakai for all of your support.

jumpy88 commented 3 years ago

@komakai just to make you know: you could be interested in Android Camera2 API for further development of OpenCV VideoCapture Android support. It seems to be able to read data from USB cameras.

komakai commented 3 years ago

@jumpy88 Although UVCCamera has not been active recently I was able to get it to work. Please check out: https://github.com/komakai/UVCCamera/tree/komakai-fixes

Feel free to create an issue over in that repository if you need help with UVCCamera

jumpy88 commented 3 years ago

Oh, thanks @komakai, I was looking for an up to date UVCCamera fork. See you there.

jumpy88 commented 3 years ago

@komakai I couldn't create an issue in your repository, so I created it in the original one you forked. You find it here https://github.com/saki4510t/UVCCamera/issues/588

nwrkbiz commented 3 years ago

Hmm for me it works fine out of the box (opencv 4.5.0). At least if I use a cheap USB webcam which seems to be v4l2 compatible.

I cross compiled a completely statically linked executable using aarch64-linux-musl-g++ so no android specific code is used. (https://github.com/nwrkbiz/simple-webcam)

jumpy88 commented 3 years ago

@nwrkbiz do you mean plain C++ cross compiled source (the linked simple-webcam project)? How did run it without an apk?

nwrkbiz commented 3 years ago

@nwrkbiz do you mean plain C++ cross compiled source (the linked simple-webcam project)? How did run it without an apk?

Yes.

I did wrap the executable into following apk: https://github.com/nwrkbiz/android-xserver

You can unpack the executable from the apk res folder to /data, chmod +x it, set DISPLAY=127.0.0.1:0 env var and execute it directly via android JAVA APIs.

For my test I hardcoded the device number of my USB webcam.

See here: (slightly modified version of simple-webcam) image

nwrkbiz commented 3 years ago

If you have a rooted smartphone you can also perform these steps manually:

You can basically use any XServer within your network. By setting the DISPLAY IP-Address you may also send the picture to your linux desktop.

jumpy88 commented 3 years ago

Two more questions:

nwrkbiz commented 3 years ago

did you link against libopencv_java4.so?

No, I linked against all the opencv libraries you can see within this makefile: https://github.com/nwrkbiz/simple-webcam/blob/main/Makefile. And these libraries were built with the help of: https://github.com/nwrkbiz/static-build

do you see your usb camera device as /dev/video0 (or other number)?

Yes it has number 6, this is the number I hardcoded for testing purposes.

jumpy88 commented 3 years ago

Ok, so I don't think this fits my case: my device is not rooted and the usb device is not mounted as /dev/videox. UVCCamera accede to it through /dev/bus/00x/00y. Thank you anyway for sharing these information, it's interesting if OpenCV started supporting VideoCapture for Android. Maybe @alalek could confirm this.

nwrkbiz commented 3 years ago

Ok, so I don't think this fits my case: my device is not rooted and the usb device is not mounted as /dev/videox. UVCCamera accede to it through /dev/bus/00x/00y. Thank you anyway for sharing these information, it's interesting if OpenCV started supporting VideoCapture for Android. Maybe @alalek could confirm this.

Having no root should not be an issue. Hmm,... what Android version do you have? Do you have any /dev/videoX devices at all? Did you try different USB-Webcams? In my example OpenCV accesses the cam via v4l2 which is a linux feature.

jumpy88 commented 3 years ago

$ getprop ro.build.version.release

7.1.1

$ getprop ro.build.version.sdk

25

I can't ls them but I can see /dev/video0 - 6 exist (they return permission denied, /dev/video7, 8...return no such file or directory). I tried with 2 different USB cameras, but no one of them seems to add a /dev/videoX, despite I can accede them with this application (they only seem to appear as /dev/bus/usb/001/00X devices)

ps what does it mean "akk"?

nwrkbiz commented 3 years ago

ps what does it mean "akk"?

Sorry typo, I meant *all

jumpy88 commented 3 years ago

Don't worry, I just thought it was some strange slang and I was missing something 😂😆 Anyway, the device is an Oculus Quest, which has 4 physical built-in cameras. I don't know why there are 6 /dev/videoX. Also, I don't know what happens with other not rooted Android devices, but maybe OQ doesn't add /dev/videoX for USB cameras to avoid messing up the localization.

nwrkbiz commented 3 years ago

@jumpy88 I am very sorry if I caused confusion, but saw that I granted my wrapper app root permissions, thats why it worked out for me.

jumpy88 commented 3 years ago

@nwrkbiz no problem, we are investigating, any clue can be useful 🙂

splintersilk commented 3 years ago

Hey there! @komakai the combination of your 9315e45 commit as well as the python build script is a thing of beauty! Massive thanks for that. I recently hit the lack of access to the camera issue and happened across this thread - your solution which works a treat! Thanks also to @jumpy88 for documenting your journey, it was informative and helpful too.

I can confirm that when backgrounding my test app and returning I am unable to revive my camera stream - it appears frozen. Is this what you meant by this @komakai?: o "stopping and restarting doesn't work - looks like some extra clean-up is required when stopping the camera"