brendan-duncan
commented
11 months ago I'm really sorry for the very late reply.
It seems that Dart is quite slow at doing conditional operations in large for loops, like iterating through pixels. The Pixel class (from the img iterator) is doing checks to make sure your setting correct values for the format of the image. It seems to add some overhead.
One way around it is to get the raw bytes of the image. This is essentially what the 3. provided. The reason I couldn't do that in general for 4. is because of the expanded support for many different image formats. But you can still access the bytes directly from the image.
final imgBytes = img.toUint8List();
for (var i = 0, l = imgBytes.length; i < l; i += 3) { // 3 bytes per pixel for a default format Image
imgBytes[i] = r;
imgBytes[i + 1] = g;
imgBytes[i + 2] = b;
}I did a timing test setting pixels on a 1000x1000 image and using the pixel iterator it was ~1100ms, and using the raw byte buffer as above it was ~859ms. It should be in line with 3.* performance.
Another optimization is that instead of physically rotating the pixels for iOS portrait orientation, you could set the image's orientation in the EXIF data.
img.exif.imageIfd.orientation = 7; // orientation of 7 is equivalent to rotating by 270
Hello, guys. I see a processing time significantly increased after updating to 4.0.17.
Here are the functions I used
Details in
processCameraImage3.3.0Details
```dart import 'dart:io'; import 'package:camera/camera.dart'; import 'package:image/image.dart' as imageLib; /// ImageUtils class ImageUtils { /// Converts a [CameraImage] in YUV420 format to [imageLib.Image] in RGB format static imageLib.Image? convertCameraImage(CameraImage cameraImage) { if (cameraImage.format.group == ImageFormatGroup.yuv420) { return convertYUV420ToImage(cameraImage); } else if (cameraImage.format.group == ImageFormatGroup.bgra8888) { return convertBGRA8888ToImage(cameraImage); } else { return null; } } /// Converts a [CameraImage] in BGRA888 format to [imageLib.Image] in RGB format static imageLib.Image convertBGRA8888ToImage(CameraImage cameraImage) { imageLib.Image img = imageLib.Image.fromBytes(cameraImage.planes[0].width!, cameraImage.planes[0].height!, cameraImage.planes[0].bytes, format: imageLib.Format.bgra); return img; } /// Converts a [CameraImage] in YUV420 format to [imageLib.Image] in RGB format static imageLib.Image convertYUV420ToImage(CameraImage cameraImage) { final int width = cameraImage.width; final int height = cameraImage.height; final int uvRowStride = cameraImage.planes[1].bytesPerRow; final int? uvPixelStride = cameraImage.planes[1].bytesPerPixel; final image = imageLib.Image(width, height); for (int w = 0; w < width; w++) { for (int h = 0; h < height; h++) { final int uvIndex = uvPixelStride! * (w / 2).floor() + uvRowStride * (h / 2).floor(); final int index = h * width + w; final y = cameraImage.planes[0].bytes[index]; final u = cameraImage.planes[1].bytes[uvIndex]; final v = cameraImage.planes[2].bytes[uvIndex]; image.data[index] = ImageUtils.yuv2rgb(y, u, v); } } return image; } /// Convert a single YUV pixel to RGB static int yuv2rgb(int y, int u, int v) { // Convert yuv pixel to rgb int r = (y + v * 1436 / 1024 - 179).round(); int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round(); int b = (y + u * 1814 / 1024 - 227).round(); // Clipping RGB values to be inside boundaries [ 0 , 255 ] r = r.clamp(0, 255); g = g.clamp(0, 255); b = b.clamp(0, 255); return 0xff000000 | ((b << 16) & 0xff0000) | ((g << 8) & 0xff00) | (r & 0xff); } } imageLib.Image? processCameraImage(CameraImage cameraImage) { imageLib.Image? image = ImageUtils.convertCameraImage(cameraImage); if (Platform.isIOS) { // ios, default camera image is portrait view // rotate 270 to the view that top is on the left, bottom is on the right image = imageLib.copyRotate(image!, 270); } return image; // processImage(inputImage); } ```
I know there is a problem that some devices will pad resolution for YUV420. So I update to 4.0.17
4.0.17Details
```dart import 'dart:io'; import 'package:camera/camera.dart'; import 'package:image/image.dart' as imageLib; /// ImageUtils class ImageUtils { /// Converts a [CameraImage] in YUV420 format to [imageLib.Image] in RGB format static imageLib.Image? convertCameraImage(CameraImage cameraImage) { if (cameraImage.format.group == ImageFormatGroup.yuv420) { return convertYUV420ToImage(cameraImage); } else if (cameraImage.format.group == ImageFormatGroup.bgra8888) { return convertBGRA8888ToImage(cameraImage); } else { return null; } } static imageLib.Image convertBGRA8888ToImage(CameraImage image) { return imageLib.Image.fromBytes( width: image.width, height: image.height, bytes: image.planes[0].bytes.buffer, order: imageLib.ChannelOrder.bgra, ); } static imageLib.Image convertNV21ToImage(CameraImage image) { return imageLib.Image.fromBytes( width: image.width, height: image.height, bytes: image.planes.first.bytes.buffer, order: imageLib.ChannelOrder.bgra, ); } static imageLib.Image convertYUV420ToImage(CameraImage image) { final uvRowStride = image.planes[1].bytesPerRow; final uvPixelStride = image.planes[1].bytesPerPixel ?? 0; final img = imageLib.Image(width: image.width, height: image.height); for (final p in img) { final x = p.x; final y = p.y; final uvIndex = uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor(); final index = y * uvRowStride + x; // Use the row stride instead of the image width as some devices pad the image data, and in those cases the image width != bytesPerRow. Using width will give you a distored image. final yp = image.planes[0].bytes[index]; final up = image.planes[1].bytes[uvIndex]; final vp = image.planes[2].bytes[uvIndex]; p.r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255).toInt(); p.g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91).round().clamp(0, 255).toInt(); p.b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255).toInt(); } return img; } } imageLib.Image? processCameraImage(CameraImage cameraImage) { imageLib.Image? image = ImageUtils.convertCameraImage(cameraImage); if (Platform.isIOS) { // ios, default camera image is portrait view // rotate 270 to the view that top is on the left, bottom is on the right // image ^4.0.17 error here image = imageLib.copyRotate(image!, angle: 270); } return image; // processImage(inputImage); } ```
Runtime for iPhone 13 pro max. I process frames at 6 fps
3.3.04.0.17Runtime for OnePlus 7pro. I process frames at 6 fps
3.3.04.0.17Runtime for Samsung A23 (The device will pad for resolution). I process frames at 6 fps
3.3.04.0.17We can see the runtime for 4.0.17 is higher than 3.3.0, especially in ios devices.
Even though the Android runtime for OnePlus 7 pro is not significantly increased, my other app has got a huge impact. I think there is something that makes the CPU busy and then triggers conflicts with other components(tflite). like 107ms for each frame increased to 170ms for each frame.