cv.merge causes segmentation fault

[TimoReusch]

I'm using the following implementation inside of an isolate:

Future<void> processFrame(CameraImage frame) async {
  // Create a ReceivePort for communication with the isolate
  final ReceivePort receivePort = ReceivePort();
  // Spawn the isolate
  await Isolate.spawn(_processFrameIsolate, receivePort.sendPort);
  // Wait for the isolate to send its SendPort
  final SendPort isolateSendPort = await receivePort.first as SendPort;

  // Convert FrameData
  final frameData = {
    'width': frame.width,
    'height': frame.height,
    'format': frame.format.raw,
    'planes': frame.planes
        .map((plane) => {
      'bytes': plane.bytes,
      'bytesPerRow': plane.bytesPerRow,
      'bytesPerPixel': plane.bytesPerPixel,
    })
        .toList(),
  };

  // Create a new ReceivePort for receiving the result
  final ReceivePort resultPort = ReceivePort();

  // Send the frame data along with the result port's SendPort to the isolate
  isolateSendPort.send([frameData, resultPort.sendPort]);

  // Wait for the result from the isolate
  final result = await resultPort.first as Map<String, dynamic>;

  // …
}

static void _processFrameIsolate(SendPort sendPort) async {
  print('Isolate spawned');
  // Create a ReceivePort to receive data from the main isolate
  final ReceivePort isolateReceivePort = ReceivePort();
  // Send the isolate's ReceivePort to the main isolate
  sendPort.send(isolateReceivePort.sendPort);

  // Listen for the frame data from the main isolate
  isolateReceivePort.listen((message) async {
    print('Frame data received');
    final frameData = message[0] as Map;
    final SendPort resultPort = message[1] as SendPort;
    // Convert the frame data to a Mat object
    cv.Mat mat = _convertFrameDataToMat(frameData);

    // …
  });
}

static cv.Mat _convertFrameDataToMat(Map frameData) {
  print("Processing frame");
  final int width = frameData['width'];
  final int height = frameData['height'];
  final List planesData = frameData['planes'];

  // Create a Mat for each plane
  List<cv.Mat> planeMats = planesData.map((planeData) {
    final Uint8List bytes = planeData['bytes'];
    cv.Mat mat = cv.Mat.fromList(height, width, cv.MatType.CV_8UC1, bytes);
    return mat;
  }).toList();

  // Create a VecMat from the list of Mats
  cv.VecMat vecMat = cv.VecMat.fromList(planeMats);

  // Merge the plane Mats into a single Mat
  cv.Mat mergedMat = cv.merge(vecMat);

  // Clean up
  planeMats.forEach((mat) => mat?.release());

  return mergedMat;
}

This results in the following error:

F/libc    ( 6969): Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x7041940000 in tid 7152 (DartWorker), pid 6969 (my_app)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/taimen/taimen:11/RP1A.201005.004.A1/6934943:user/release-keys'
Revision: 'rev_10'
ABI: 'arm64'
Timestamp: 2024-06-27 14:21:58+0200
pid: 6969, tid: 7152, name: DartWorker  >>> de.timo.my_app <<<
uid: 10190
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x7041940000
    x0  0000000000000000  x1  0000006fcf41d010  x2  00000000000e1000  x3  0000000000000003
    x4  0000000000000004  x5  0000000000000000  x6  0000000003000000  x7  0000000000000000
    x8  00000000000e0ff0  x9  0000006fdbf5f040  x10 00000070418cf040  x11 0000000000000000
    x12 0000006fe4dcc040  x13 00000000fffffff0  x14 0000000000070fd0  x15 0000000000070fc0
    x16 0000000000152f40  x17 0000000000000002  x18 0000006fd0420000  x19 00000000000e1000
    x20 0000006fccf5b040  x21 0000006fddb476c8  x22 0000000000000003  x23 0000006fddb49000
    x24 0000000000000003  x25 0000000000000000  x26 0000006fce88d7a0  x27 0000000000000003
    x28 00000000000e1000  x29 0000006fddb47560
    lr  0000006fce88caa8  sp  0000006fddb47540  pc  0000006fce88cbd0  pst 0000000080000000
backtrace:
      #00 pc 0000000000731bd0  /data/app/~~pDchOoXjTxLFXxbRgQsgFw==/de.timo.my_app-60au6JTGgpooivmfe4iyng==/lib/arm64/libopencv_dart.so (BuildId: e85889b8a44465372645f91bcb7f24887cd76169)
      #01 pc 00000000007327f4  /data/app/~~pDchOoXjTxLFXxbRgQsgFw==/de.timo.my_app-60au6JTGgpooivmfe4iyng==/lib/arm64/libopencv_dart.so (BuildId: e85889b8a44465372645f91bcb7f24887cd76169)
      #02 pc 0000000000733048  /data/app/~~pDchOoXjTxLFXxbRgQsgFw==/de.timo.my_app-60au6JTGgpooivmfe4iyng==/lib/arm64/libopencv_dart.so (BuildId: e85889b8a44465372645f91bcb7f24887cd76169)
      #03 pc 0000000000733914  /data/app/~~pDchOoXjTxLFXxbRgQsgFw==/de.timo.my_app-60au6JTGgpooivmfe4iyng==/lib/arm64/libopencv_dart.so (BuildId: e85889b8a44465372645f91bcb7f24887cd76169)
      #04 pc 00000000005e2bd0  /data/app/~~pDchOoXjTxLFXxbRgQsgFw==/de.timo.my_app-60au6JTGgpooivmfe4iyng==/lib/arm64/libopencv_dart.so (Mat_Merge+68) (BuildId: e85889b8a44465372645f91bcb7f24887cd76169)
      #05 pc 0000000000007bc4  [anon:dart-code]

Looks like a segmentation fault to me.

[rainyl]

Can't reproduce, please provide a minimal example.

[TimoReusch]

@rainyl I put together a minimal example you can clone and try to run here: https://github.com/TimoReusch/flutter_opencv_merge

[rainyl]

Got it, will try to fix some time, thx~

[rainyl]

Reproduced, but it's not the problem of opencv_dart, the following code worked fine:

  List<cv.Mat> planeMats = planesData.map((planeData) {
    // replace this line
    // final Uint8List bytes = planeData['bytes'];
    // with
    final bytes = List.generate(height * width * 1, (i) => i);
    cv.Mat mat = cv.Mat.fromList(height, width, cv.MatType.CV_8UC1, bytes);
    return mat;
  }).toList();

Thus, I am sure it's because the incorrect use of planeData['bytes'], I haven't used the camera plugin, so do not know what planeData['bytes'] exactly are, if they are encoded image data, please use cv.imdecode to decode, if they are raw pixel values, please ensure the length of planeData['bytes'] is exactly channels*rows*cols, where channels matchs the Mat you created.

I suppose the planeData['bytes'] contains RGB values of an image, then you can't use cv.MatType.CV_8UC1 but cv.MatType.CV_8UC3 should be used, and opencv expects BGR rather than RGB, so look into your code please.

[rainyl]

If no further problems, I am going to close this issue.

Feel free to reopen it or open new issues if you still have any problems.