search

googlesamples / mlkit

A collection of sample apps to demonstrate how to use Google's ML Kit APIs on Android and iOS
Apache License 2.0
3.52k stars 2.92k forks source link

MLKit and Camera2 integration in Android #745

Open arianaa30 opened 11 months ago

arianaa30 commented 11 months ago

I have an Android camera app that uses Camera2 API and sends frames to a server. I recently did some work on top of MLKit's segmentation example code that generates masks and graphics overlay, etc. Now I want to integrate this MLKit demo into our app, but not sure how to integrate these two together.

Given my implementations are in segmentation example code, so my main question is how I can exactly call these functions in the demo from inside our camera app. I assume I should pass some variables like frames from our camera app to this MLKit demo and start processing them, and return something. Are these compatible together, given our app uses Camera2 API?

For example, I should first initialize a new VisionImageProcessor imageProcessor. Then just call processByteBuffer() or maybe processBitmap()? But we have graphicOverlay. What exactly these steps look like in my case? And is it best to pass frames as byteBuffer or media.Image or what for best performance?

Any tips or guidance on integration and tips would be appreciated.

ai-plays commented 11 months ago

Hi arianaa30 ,

Here are some pointers:

  1. CameraX library is more recommended for Camera applications. The ML Kit demo app contains an example of using CameraX for segmentation.
  2. The demo app doesn't contain camera2 examples.
  3. If the demo app example doesn't exactly fit your case, we can use the API based on documentation: https://developers.google.com/ml-kit/vision/selfie-segmentation/android creating an InputImage using one of the supported methods. https://developers.google.com/android/reference/com/google/mlkit/vision/common/InputImage
arianaa30 commented 11 months ago

Ok thanks. Looking at the demo, looks like for processing live preview frame from Camera1 API, VisionProcessorBase uses processByteBuffer() while in CameraX it defines processImageProxy(). Now that I have Camera 2, I have to define a new function right? Which of the two do you think can I reuse with minimum changes?

Harshad0011 commented 8 months ago

@arianaa30 hey I'm also trying to integrate pose detection with my app which uses camera2 api if you've found the solution to handle bufferQueueProducer error, please share your insights and help me perform in my app. It would be really appreciated