More general camera item

[qbonnard]

The Camera Item provided by the multimedia package of QML is easy to use, but somewhat limited. Most importantly, there is no way to choose which hardware should be used if there are several camera connected. That's especially a problem on android, where front and back cameras are often available, or on laptops with an integrated camera.

[ayberkozgur]

Also, this: http://qt-project.org/wiki/Qt5ForAndroidQtMultimedia Related issue report: https://bugreports.qt-project.org/browse/QTBUG-35416

[qbonnard]

Also, this: http://qt-project.org/wiki/Qt5ForAndroidQtMultimedia Last edit: March 6, 2013

I think that was fixed... At least, a simple feedback of the camera works easily on android.

[ayberkozgur]

I think we will be most happy with a QML wrapper around the OpenCV camera. It should work seamlessly across desktop and Android. It will also be faster to integrate with Chilitags since it takes cvmats.

[qbonnard]

I agree with everything, except the "It should work seamlessly across desktop and Android". Last time I tried, getting the android camera to work via opencv was hell... But maybe I missed something.

[ayberkozgur]

Well, Lorenzo has been testing this for the past week and it seemed to work. I'll test it myself today.

[qbonnard]

If that's the exciting stuff Séverin was talking about, I'm indeed thrilled !

On Mon, Sep 22, 2014 at 11:33 AM, Ayberk Özgür notifications@github.com wrote:

Well, Lorenzo has been testing this for the past week and it seemed to work. I'll test it myself today.

— Reply to this email directly or view it on GitHub https://github.com/chili-epfl/qimchi/issues/7#issuecomment-56349634.

[ayberkozgur]

Yes, indeed :)

[ayberkozgur]

I can confirm that I am able to allocate the camera and get 30FPS image data on both Android and desktop uniformly with cv::VideoCapture. I linked libandroid_camera.a exclusively in the android build and included libnative_camera_r4.3.0.so in the apk. I'll test the configurations (like image size and format) and surface output on screen tomorrow.

[qbonnard]

:+1: :like: ;) I'm curious about the color format on android... I'm still wondering whether Chilitags should take care of the color conversion

[ayberkozgur]

We should avoid conversion as much as possible, especially on Android where resources are very limited. The default color format on Android is YUV NV21, which is very good since we can access the grayscale channel directly which is the only required channel for Chilitags. On desktop it is RGB AFAIK, and it shouldn't hurt too much to convert it to grayscale once and expose the untouched RGB image for screen rendering.

[qbonnard]

I don't recall the details, but I remember thinking that it would have been nice to have been able to give whatever image I got from the camera, calling something like Chilitags::setPixelFormat, and let it worry about the grayscale converstion more efficiently. IIRC, the quick and dirty way I ended up doing involved a conversion to rgb for display, and another to bgr, or something like that. The idea is to use OpenCV's conversion, which I assume to be smart enough to convert YUV to grayscale by taking the Y channel, among other optimised goodness it could provide in other cases. We'll see...

[ayberkozgur]

Well the picture now is a bit bleak. Here's the situation: The guys who wrote the cv android camera capture thought it would be a good idea to hide the color format output functionality from the user altogether; once you use VideoCapture it is output as RGB and there's nothing you can do about it; you can only convert it once more to your desired format.

I've been looking into the androidcamera source and it's not as bad as it sounds: You have to get the raw buffer copied to an external buffer to be presented to the user, why not do the conversion from YUV to RGB on the way. The bad part is this: Qt's VideoOutput presents you a QAbstractVideoSurface for you to draw your frames on and this surface doesn't accept RGB neither on Android nor on desktop (does not accept YUV NV21 either)... The closest it does accept is RGBA and the conversion from RGB to RGBA on Android (with NEON instructions enabled) can take up to 7 ms. At least they have the code ready to convert raw YUV frames to RGBA instead of RGB.

For now I'm forking the opencv VideoCapture code to retrieve frames as RGBA instead of RGB in order to not do another conversion just to be able to draw it on screen. For maximum speed, we need to do these in the future:

• Implement a child of VideoOutput that accepts RGB and YUV NV21 frames and draws them on the screen via GL shaders. This could happen by itself in future releases of Qt.
• Hack the code of androidcamera to expose the raw YUV NV21 frames in order to be used in things like Chilitags. Shouldn't be too hard as long as we pay proper attention to the existing mutexes.

[ayberkozgur]

In fact, they also have YUV to grayscale conversion code ready, it shouldn't be too difficult to get both RGBA and grayscale out.

And they seem to expose the YUV buffer's Y channel to outside directly in case of grayscale conversion, so it should also be safe to get this buffer out whenever we want.

[ayberkozgur]

Well, it turns out that the channel parameter in VideoCapture's retrieve() actually chooses which color format conversion is done on Android, and is totally ignored on desktop. Can someone please tell me that the documentation (http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture-retrieve) does not imply this, just as a sanity check for me?

[ayberkozgur]

Also, @severin-lemaignan detected that cv::VideoCapture can set native Android camera properties, they just lack the documentation for it: https://github.com/Itseez/opencv/blob/master/modules/videoio/src/cap_android.cpp#L260

EDIT: Oops, it seems that you can't set the preview format, only get it.

[ayberkozgur]

Well, we have https://github.com/chili-epfl/qml-cvcamera now, so closing this issue.