search

kivy / kivy

Open source UI framework written in Python, running on Windows, Linux, macOS, Android and iOS
https://kivy.org
MIT License
16.95k stars 3.03k forks source link

Refresh `kivy.core.camera.camera_android` to use a recent Android API #8583

Open misl6 opened 3 months ago

misl6 commented 3 months ago

Our current implementation of the camera provider for Android works on top of the old and deprecated Camera API (https://developer.android.com/media/camera/camera-deprecated/camera-api). We should consider migrating to CameraX or Camera2 instead, as suggested by Android docs.

These new APIs will also make it possible to fulfill the new camera provider "abstraction" (See: https://github.com/kivy/kivy/issues/8582)

Samael-TLB commented 3 months ago

Recently, for an app I have built a camera provider of android based on camerax as Camera2 needs a lot of device specific edge cases to be dealt with. Now, I'm experimenting with the video use case of camerax as that is still experimental.

So, I would like to work on this as I already have the code ready just need a few discussions and experimentations for video use case.

Samael-TLB commented 3 months ago

Thanks for the assignment. In this regards, I would like to express a few of my findings in the implementation. So, if we go for a python implementation through pyjnius there's no signficant performance penalty even with preview+imagecapture+imageanalysis+video (note video is not supported in all devices along with imagecapture or imageanalysis) in working while there's slight extra latency in startup during initialization. So for any normal work this extra latency is allowable as its typically under 1-2 seconds, measured from getting the camera provider upto preview texture viewing in ui. This is inflated as I'm considering the total time to first start to see the preview from the initial configurations. While the individual components like preview texture binding with camera source is typically within 500-800ms, while other parts are also low latency. While pure java implementation is on average faster in startup by about 300-500ms in total, while there's no significant difference in performance afterwards between python and java implementations.

In this regard, one thing to mention is that camerax also allows to capture photos in memory and only for the callback of this process we need to subclass an abstract class. In short, if we want to support this in memory capturing of photos then we need to have a java file inclusion.

In view of the above, I would like to understand what the core developers would think regarding the implementation should we have a java implementation and use it through pyjnius or a completely python pyjnius side implementation with just one callback being defined within java due to being an abstract class (only if we want to support that).

RobertFlatt commented 1 month ago

My experience is that a pure Java camera provider works well https://github.com/Android-for-Python/camerax_provider (the cameras video api has been updated since this was written, but that will not change the experience).

Aside from performance, some of the newer Android apis implement callbacks as a reference to an instance of a "provider" Java class. It is not possible to pass a reference to PythonJavaClass in this case as this is Python not Java. The workarounds compound, it is much simpler to implement the full provider in Java or Kotlin (there are examples!). The provider probably belongs as a recipe. Try it here https://pypi.org/project/camera4kivy/ , Here is a simple test case https://github.com/Android-for-Python/c4k_photo_example

The camera widget api must be extended as file capture is performed by the provider not the widget (except screenshot). In addition users expect an image analysis api - they want a camera as part of some other behavior. Also the widget must handle various aspect ratio, orientation, pause/resume , pinch-spread zoom, and tap focus.

Personally I view Kivy camera widget as great for taking screenshots from a web cam and not much else, very 2000. I choose to ignore Kivy camera widget and implement my own widget, my thinking at the time was backwards compatibility would be hard and unfortunately might be expected. For example Camerax is not usable (including connect) till one time-step after Kivy on_start(), whereas Kivy camera assumes connect at build(). This and the threaded implementation in Camerax means the api must be at least partially event driven.

Finally for performance a major consideration is the size of any Kivy frame buffer, the hardware camera and screen lead to user expectations of high resolution, too large a Kivy frame buffer may lead to preview latency issues. The Kivy camera bypasses this by defaulting to 640x480, this would not make a mobile user happy. As far as I remember it worked up HD buffer, maybe my Kivy buffer implementation could have been optimized I didn't look at it.

Have fun with your project. This comment is just a drive by, I do not typically visit here.

Samael-TLB commented 2 weeks ago

Thanks for your suggestion @RobertFlatt. First of all sorry for the delayed response. Was away from work due to health issues. I took inspiration from your works too.

Aside from performance, some of the newer Android apis implement callbacks as a reference to an instance of a "provider" Java class. It is not possible to pass a reference to PythonJavaClass in this case as this is Python not Java. The workarounds compound, it is much simpler to implement the full provider in Java or Kotlin (there are examples!). The provider probably belongs as a recipe. Try it here https://pypi.org/project/camera4kivy/ , Here is a simple test case https://github.com/Android-for-Python/c4k_photo_example

I agree, but as I mentioned this is the case for just one callback currently that is for in memory photo capture, which has an alternative of photo capture via save to disk directly. But I tend to see a shift in the android apis from normal interface classes to abstract classes which necessitate the use of Java or Kotlin.

The camera widget api must be extended as file capture is performed by the provider not the widget (except screenshot). In addition users expect an image analysis api - they want a camera as part of some other behavior. Also the widget must handle various aspect ratio, orientation, pause/resume , pinch-spread zoom, and tap focus.

The current goal for 3.0 is complete rewrite of the api so as to enable incorporation of future technologies easily, image analysis and gesture handling are very important and are gonna be included by default in the api.

Personally I view Kivy camera widget as great for taking screenshots from a web cam and not much else, very 2000. I choose to ignore Kivy camera widget and implement my own widget, my thinking at the time was backwards compatibility would be hard and unfortunately might be expected. For example Camerax is not usable (including connect) till one time-step after Kivy on_start(), whereas Kivy camera assumes connect at build(). This and the threaded implementation in Camerax means the api must be at least partially event driven.

Yes, the current implementation is not very feature rich but the new proposed one is targeting to be much more useful to the end users. Using CameraX we are safe till Android 5.0 and above which is what officially supported by Kivy and suggested by Android. Though some features if used would raise the minimum supported version to 6 or higher. I am going forward with a lazy permission based approach, where you need permissions first explicitly only when you use the guarded apis, before that you are free to do whatever android allows you to do without permissions, following android guidelines of asking permissions. So this allows for at least no jank in my tests.

Finally for performance a major consideration is the size of any Kivy frame buffer, the hardware camera and screen lead to user expectations of high resolution, too large a Kivy frame buffer may lead to preview latency issues. The Kivy camera bypasses this by defaulting to 640x480, this would not make a mobile user happy. As far as I remember it worked up HD buffer, maybe my Kivy buffer implementation could have been optimized I didn't look at it.

As said above, I have implemented the api and tested it from Android 8-13, with no potential drawback or performance issues or significant difference noticed between pure python and java implementations. I didn't have any issues with the Kivy buffer which supported upto the highest size available on the device during my tests. Though there might be some issues for Android 5-7 (as untested by me).

misl6 commented 2 weeks ago

Hello everyone!

Let's start by saying that as @Samael-TLB pointed out, 3.0.0 is the right moment for us to rewrite what needs to be rewritten from scratch. Camera API is one of these components. It was faulty and buggy since the beginning. Nope. Tech just evolved, we now have camera phones that can compete with professional-grade cameras, and people (and apps) are using cameras more than ever.

I will work out to have a proposal (initially just drawings on paper) for #8582, based on my experience and suggestions taken from this issue, GitHub issues in general, discussions (both on Discord and IRL), etc ... (so please keep an eye on #8582, and feel free to make suggestions as soon as you see something).

Regarding Android:

A quick eagle eye on what I expect to have in #8582: