Open c-goettert opened 2 months ago
Hey - impressive work that you can draw it to a skia canvas! Is that running smooth? What FPS?
The VisionCamera Skia integration will use GPU Hardware Buffers, so it's gonna run really smooth, but I was wondering how smooth your approach is.
Regarding your issue; hm, well to be honest I don't have a lot of time to investigate this right now, but maybe it's due to the buffers just always being in device native orientation on Android?
I'd gladly accept PRs!
Hi, thanks for you answer! Regarding performance: My current approach is only suitable for debugging purposes. The app only runs with approx. 2-3 frames and crashes after a few seconds (probably a memory leak somewhere). However, it is sufficient for checking the frame content.
Regarding the issue:
maybe it's due to the buffers just always being in device native orientation on Android?
That would also be my guess, is there an easy way to check the native orientation of the device?
device.sensorOrientation
- but this is not guaranteed to be correct by Android. Some vendors choose to not implement it properly, specifically on Samsungs it's 180 deg rotated.
CameraX has workarounds for all such quirks, in VisionCamera V4 that should be fixed.
We now have rotation support in vision-camera-resize-plugin - thanks @rodgomesc! ❤️
So you can now use rotation: '90deg'
to fix the rotation issue.
I'm not sure why it's blueish though. This might not be a problem of the resize plugin, but rather of the way you display it maybe? Because when I inspect the UIImage of the ARGB 8888 array on the native side it looks correct
I'm not sure why it's blueish though. This might not be a problem of the resize plugin, but rather of the way you display it maybe? Because when I inspect the UIImage of the ARGB 8888 array on the native side it looks correct
it's blueish for me as well if i use rgba
on ios, since i don't have time to investigate now my workaround is
pixelFormat: IS_ANDROID ? 'rgba' : 'argb'
Strange for me it seems the other way around..
My request format for the resize plugin is pixelFormat: 'rgb'
. I interpret it as described in my first post:
const data = resize(frame, {
scale: {
width: 320,
height: 320
},
pixelFormat: 'rgb',
dataType: 'uint8'
})
const arrayData = new Array(width * height * 4).fill(255)
for (let i = 0, j = 0; i < data.length; i += 3, j += 4) {
arrayData[j] = data[i] // R
arrayData[j+1] = data[i + 1] // G
arrayData[j+2] = data[i + 2] // B
arrayData[j+3] = 255 // A
}
If I use the same code on iOS and Android (same display method), it looks fine on iOS (tested on iPhone X) but blueish on Android (tested on Pixel 7A). I can fix it for android by switching the blue and red channel (= interpret values as BGR):
const arrayData = new Array(width * height * 4).fill(255)
for (let i = 0, j = 0; i < data.length; i += 3, j += 4) {
arrayData[j] = data[i + 2] // R
arrayData[j+1] = data[i + 1] // G
arrayData[j+2] = data[i] // B
arrayData[j+3] = 255 // A
}
Maybe it is a bug then in this method: https://github.com/mrousavy/vision-camera-resize-plugin/blob/8a190a320cad637b917ab90d24f964ae3ece540e/ios/ResizePlugin.mm#L183-L250
You could try playing around with those values or hitting breakpoints to see if all values are correct, it seems like some channels are swapped wrong?
If I use the same code on iOS and Android (same display method), it looks fine on iOS (tested on iPhone X) but blueish on Android (tested on Pixel 7A). I can fix it for android by switching the blue and red channel (= interpret values as BGR):
you are right! the rgb buffer is incorrect, right now it's returning bgr
we can validate that creating the bitmap right before the return of the SharedValue
in ResizePlugin.kt
diff --git a/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt b/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
index 8c8c0c5..b018bf4 100644
--- a/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
+++ b/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
@@ -1,5 +1,6 @@
package com.visioncameraresizeplugin
+import android.graphics.Bitmap
import android.graphics.ImageFormat
import android.media.Image
import android.util.Log
@@ -156,7 +157,25 @@ class ResizePlugin(private val proxy: VisionCameraProxy) : FrameProcessorPlugin(
targetType.ordinal
)
- return SharedArray(proxy, resized)
+ val result = SharedArray(proxy, resized)
+
+
+ val bitmap = Bitmap.createBitmap(scaleWidth, scaleHeight, Bitmap.Config.ARGB_8888)
+ val intArray = IntArray(scaleWidth * scaleHeight)
+
+ result.byteBuffer.rewind()
+
+ for (i in 0 until intArray.size) {
+ // note: use bgr instead of rgb
+ val b = result.byteBuffer.get().toInt() and 0xFF
+ val g = result.byteBuffer.get().toInt() and 0xFF
+ val r = result.byteBuffer.get().toInt() and 0xFF
+ intArray[i] = (0xFF shl 24) or (r shl 16) or (g shl 8) or b
+ }
+
+
+ bitmap.setPixels(intArray, 0, scaleWidth, 0, 0, scaleWidth, scaleHeight)
+ // breakpoint the line above to visualize the correct preview
+ return result
}
private enum class PixelFormat {
Hi @rodgomesc @mrousavy I think I was able to track the bug down to libyuv library and fix it - check out PR: https://github.com/mrousavy/vision-camera-resize-plugin/pull/53
To verify / understand the content of the frames, I created a simple demo app that stores the content returned from the resize plugin and just draws it right below the original image (via react-native-skia).
The
resize
call within my frame processor looks like follows:My camera settings look like this:
I can could verify my approach is generally working on iOS, where my test-frame is rendered correctly:
iPhone X:
However, on my Android phone (Pixel 7A, Android 14), the frame seems to be rotated by -90°. Also, as you can see in the image below, the format is returned in
bgr
insteadrgb.
.Pixel 7A:
Furthermore, I don't actually want to use the default centre-crop, but the upper square of the image. I have therefore adapted the resize call as follows:
This setting strangely cuts the result into 3 image-areas:
To summarize the issues:
bgr
despitergb
was requestedI use the following configuration:
If it helps to trace the issue I can gladly add my complete test component.