search

airsdk / Adobe-Runtime-Support

Report, track and discuss issues in Adobe AIR. Monitored by Adobe - and HARMAN - and maintained by the AIR community.
200 stars 11 forks source link

[Feature Request] Camera frames access by native extension #2029

Open itlancer opened 2 years ago

itlancer commented 2 years ago

Feature Description

AIR should provide some way to get camera frames by native extension. In many cases for image processing, computer vision and OpenCV usage you need fastest way to get "image bytes" from camera. Most of such operations should be done by native extensions. And getting frames from AS3 Camera and then pass it to native extension, then convert to bytes and so on cause huge performance overhead. Especially for low-performance Android devices. Moreover Camera usage with AS3 side for Android cause terrible performance: https://github.com/airsdk/Adobe-Runtime-Support/issues/1980 May be it related that AIR still uses legacy Android Camera API https://developer.android.com/guide/topics/media/camera, not CameraX https://developer.android.com/training/camerax or Camera2 https://developer.android.com/reference/android/hardware/camera2/package-summary.

If we get some way to get frames from camera by native extension it could increase overall performance for such applications.

For example, with Android native extension it could be new callback "on new image availability" which available through FlashRuntimeExtensions. With such callback we could get https://developer.android.com/reference/android/media/Image or frame YUV_420_888 bytes.

May be access to camera object from native side also could be useful. In such way we could also configure other camera parameters like torch/flash, auto focus and other. https://developer.android.com/reference/android/hardware/camera2/CameraManager

Related issues: https://github.com/airsdk/Adobe-Runtime-Support/issues/1895 https://github.com/airsdk/Adobe-Runtime-Support/issues/1348 https://github.com/airsdk/Adobe-Runtime-Support/issues/1344

Known Workarounds

Write own native extension for camera usage.

ajwfrost commented 7 months ago

Hi

Just reviewing this one, based on the other comments/updates made to linked issues recently .. and am curious: if you're looking to get hold of camera frames from within an ANE, then why do you not just directly request the camera via the ANE i.e. use the Android/Java code in the ANE to take a picture or kick off a video recording?

thanks

itlancer commented 7 months ago

@ajwfrost Firstly let me describe my use case. I need to display on screen camera view, process camera images (by native extension) and overlay some graphics over faces (such as masks). All in real-time.

Sometimes we are working with camera just from native extension (without AIR Camera). But there is a challenge to display camera view from native extension with other DisplayList and Stage3D. Especially on Android (performance issues) and using BitmapData transferring from ANE->AIR. We trying to optimize it by using Stage3D and passing VideoTexture from native extension to it. But still have some limitations (and it could works only for Android right now).

Basically all these issues around "how to improve performance?". 1) Camera usage cause terrible performance for Android: https://github.com/airsdk/Adobe-Runtime-Support/issues/1980 2) We need to make ARGB -> RGB conversion for camera images bytes. This feature could help: https://github.com/airsdk/Adobe-Runtime-Support/issues/3048 3) Camera access from Worker also could help. Right now it impossible. 4) Working with native extension from Worker could help. Right now it impossible. 5) Overlay something over StageWebView and other "native" layers could help. "Combining" Stage3D/DisplayList and "native" render. Discussed here: https://github.com/airsdk/Adobe-Runtime-Support/discussions/1170#discussioncomment-7656524