How to convert Matft array to OpenCV mat array

[Eumentis-Jayashree]

I have created Matft array from float array let matftArray = MfArray(floatArrayValues) now I want to convert this array to OpenCV mat array so that I can use this array for processing.

[jjjkkkjjj]

@Eumentis-Jayashree According to OpenCV's code, how about like this? (The below is psuedo code.)

void  MfarrayToMat(const MfArray array, cv::Mat& m) {
    // you should assert array is row contiguous
    Int cols = array.shape[0], rows = array.shape[1], channel = array.shape[2];

    if (channel == 1)
    {
        m.create(rows, cols, CV_32FC1);
        memcpy(array.mfdata.data_real, m.data, array.storedSize);
    }
    else if (channel == 3)
    {
        m.create(rows, cols, CV_32FC4); 
        memcpy(array.mfdata.data_real, m.data, array.storedSize);
    }
    else
    {
        m.create(rows, cols, CV_32FC4);
        memcpy(array.mfdata.data_real, m.data, array.storedSize);
    }
}

[Eumentis-Jayashree]

Hey @jjjkkkjjj thank you for the response.

Actually, I am getting the result by the ONNX model. So the output I get is in ORTValue form, which has this shape [1, 359, 8400].

1. I converted that bytedata output to the float array let floatValuesForOutput = rawOutputData.withUnsafeBytes { (buffer: UnsafeRawBufferPointer) -> [Float] in let floatBuffer = buffer.bindMemory(to: Float.self) let count = rawOutputData.count / MemoryLayout<Float>.stride return Array(UnsafeBufferPointer(start: floatBuffer.baseAddress, count: count)) }

2. After that I have converted floatValues to Matft array let arr = MfArray(floatValuesForOutput)

So, according to your solution, if I print the shape of the arr, then I only get Shape: [3015600], which is equal to 359 * 8400.

[jjjkkkjjj]

@Eumentis-Jayashree

The reason of the shape: [3015600] is you pass the "1d" array to MfArray. Namely, floatValuesForOutput is 1d array.

So, when you want MfArray with shape: [1, 359, 8400], you should reshape it by;

let arr = MfArray(floatValuesForOutput, shape: [1, -1, 8400])

Caution: The conversion method you mentioned is asserted that rawOutputData is row contiguous. If you pass the NOT row contiguous rawOutputData, the MfArray will be different from it.

To prevent this, you should manage not only data but also shape and strides. Does ORTValue have strides property?

[jjjkkkjjj]

Oh, I'm sorry, the above code is invalid due to not supporting negative dimension in init... Instead of init, you can do it like this;

let arr = MfArray(floatValuesForOutput).reshape([1, -1, 8400])

[Eumentis-Jayashree]

@Eumentis-Jayashree

The reason of the shape: [3015600] is you pass the "1d" array to MfArray. Namely, floatValuesForOutput is 1d array.

So, when you want MfArray with shape: [1, 359, 8400], you should reshape it by;

let arr = MfArray(floatValuesForOutput, shape: [1, -1, 8400])

Caution: The conversion method you mentioned is asserted that rawOutputData is row contiguous. If you pass the NOT row contiguous rawOutputData, the MfArray will be different from it.

To prevent this, you should manage not only data but also shape and strides. Does ORTValue have strides property?

No, ORTValue doesn't have strides property. It only has tensorData(), typeInfo(), tensorTypeAndShapeInfo() and tensorStringData() properties.

[jjjkkkjjj]

I see, then you can get same data by above code. The ORTValue may preserve the row contiguous.

[jjjkkkjjj]

@Eumentis-Jayashree Did you solve it? And also, I'm interested in the actual code converting MfArray into cv::Mat. If possible, could you share it to me? (I'd like to integrate it into main branch)

[Eumentis-Jayashree]

@Eumentis-Jayashree Did you solve it? And also, I'm interested in the actual code converting MfArray into cv::Mat. If possible, could you share it to me? (I'd like to integrate it into main branch)

Yeah, I solved it. But I am not sure if it's right approach. I have converted the Matft array to a Swift array and then converted it into an OpenCV mat array.

[Eumentis-Jayashree]

Hey @jjjkkkjjj thank you for the response.

Actually, I am getting the result by the ONNX model. So the output I get is in ORTValue form, which has this shape [1, 359, 8400].

1. I converted that bytedata output to the float array let floatValuesForOutput = rawOutputData.withUnsafeBytes { (buffer: UnsafeRawBufferPointer) -> [Float] in let floatBuffer = buffer.bindMemory(to: Float.self) let count = rawOutputData.count / MemoryLayout<Float>.stride return Array(UnsafeBufferPointer(start: floatBuffer.baseAddress, count: count)) }
2. After that I have converted floatValues to Matft array let arr = MfArray(floatValuesForOutput)

So, according to your solution, if I print the shape of the arr, then I only get Shape: [3015600], which is equal to 359 * 8400.

Here I get rawOutputData from ORTValue using .tensorData() property which data type is Data. Can I directly convert this rawOutputData into Matft array? so that I don't need above conversion.

[Eumentis-Jayashree]

@jjjkkkjjj The output which I get is in shape of [1, 359, 8400]. Here 359 are rows and 8400 are columns. There is no channel like image.

[jjjkkkjjj]

Yeah, I solved it. But I am not sure if it's right approach. I have converted the Matft array to a Swift array and then converted it into an OpenCV mat array.

OK, but it is not efficient due to coping the data twice as you mentioned.

get rawOutputData from ORTValue using .tensorData() property which data type is Data. Can I directly convert this rawOutputData into Matft array?

That's why we should use same data or copy it once. Fortunately, I implemented borrowing data system by this;

let shape = // get from ORTValue
let size = // calculate the size from product all dimension = 1*359*8400
let floatValuesForOutput = rawOutputData.withUnsafeBytes { 
    (buffer: UnsafeRawBufferPointer) -> MfArray in 
    // copy data automatically
    let mfdata = MfData(source: nil, data_real_ptr: buffer.baseAddress!, data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
    let mfstructure = MfStructure(shape: shape)
    return MfArray(mfdata: mfdata, mfstructure: mfstructure)
}

If the ORTValue conform to MfDataBasable, you can share data without copying, so that it is very efficient!

[Eumentis-Jayashree]

Yeah, I solved it. But I am not sure if it's right approach. I have converted the Matft array to a Swift array and then converted it into an OpenCV mat array.

OK, but it is not efficient due to coping the data twice as you mentioned.

get rawOutputData from ORTValue using .tensorData() property which data type is Data. Can I directly convert this rawOutputData into Matft array?

That's why we should use same data or copy it once. Fortunately, I implemented borrowing data system by this;

let shape = // get from ORTValue
let size = // calculate the size from product all dimension = 1*359*8400
let floatValuesForOutput = rawOutputData.withUnsafeBytes { 
    (buffer: UnsafeRawBufferPointer) -> MfArray in 
    // copy data automatically
    let mfdata = MfData(source: nil, data_real_ptr: buffer.baseAddress!, data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
    let mfstructure = MfStructure(shape: shape)
    return MfArray(mfdata: mfdata, mfstructure: mfstructure)
}

If the ORTValue conform to MfDataBasable, you can share data without copying, so that it is very efficient!

@jjjkkkjjj Thank you :). But the this constructor is define as internal, I am unable to access it.

[jjjkkkjjj]

OMG! I'll change it into public ASAP. Wait for a moment...

[jjjkkkjjj]

@Eumentis-Jayashree I've changed it! Please check.

[jjjkkkjjj]

This one is no-copy (share version)

extension ORTValue: MfDataBasable{
}

let rawOutputData // ORTValue
let shape = // get from ORTValue
let size = // calculate the size from product all dimension = 1*359*8400
let floatValuesForOutput = rawOutputData.withUnsafeBytes { 
    (buffer: UnsafeRawBufferPointer) -> MfArray in 
    // share data
    let mfdata = MfData(source: rawOutputData, data_real_ptr: buffer.baseAddress!, data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
    let mfstructure = MfStructure(shape: shape)
    return MfArray(mfdata: mfdata, mfstructure: mfstructure)
}

[Eumentis-Jayashree]

let floatValuesForOutput = rawOutputData.withUnsafeBytes { 
    (buffer: UnsafeRawBufferPointer) -> MfArray in 
    // copy data automatically
    let mfdata = MfData(source: nil, data_real_ptr: buffer.baseAddress!, data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
    let mfstructure = MfStructure(shape: shape)
    return MfArray(mfdata: mfdata, mfstructure: mfstructure)
}

Yeah, I solved it. But I am not sure if it's right approach. I have converted the Matft array to a Swift array and then converted it into an OpenCV mat array.

OK, but it is not efficient due to coping the data twice as you mentioned.

get rawOutputData from ORTValue using .tensorData() property which data type is Data. Can I directly convert this rawOutputData into Matft array?

That's why we should use same data or copy it once. Fortunately, I implemented borrowing data system by this;

let shape = // get from ORTValue
let size = // calculate the size from product all dimension = 1*359*8400
let floatValuesForOutput = rawOutputData.withUnsafeBytes { 
    (buffer: UnsafeRawBufferPointer) -> MfArray in 
    // copy data automatically
    let mfdata = MfData(source: nil, data_real_ptr: buffer.baseAddress!, data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
    let mfstructure = MfStructure(shape: shape)
    return MfArray(mfdata: mfdata, mfstructure: mfstructure)
}

If the ORTValue conform to MfDataBasable, you can share data without copying, so that it is very efficient!

@jjjkkkjjj Thank you :). But the this constructor is define as internal, I am unable to access it.

I am getting error Cannot convert value of type 'UnsafeRawPointer' to expected argument type 'UnsafeMutableRawPointer' at _data_realptr: buffer.baseAddress! , because constructor accepts this data type data_real_ptr: UnsafeMutableRawPointer Do I need to convert this (buffer: UnsafeRawBufferPointer) to UnsafeMutableRawPointer?

[jjjkkkjjj]

@Eumentis-Jayashree Does rawOutputData have withUnsafeMutableBytes method? If so, when you replace withUnsafeBytes into withUnsafeMutableBytes, the error will be suppressed. Otherwise, I guess you can convert UnsafeRawPointer into UnsafeMutableRawPointer by;

let mfdata = MfData(source: nil, data_real_ptr: UnsafeMutableRawPointer(mutating: buffer.baseAddress!), data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)

[Eumentis-Jayashree]

I have converted UnsafeRawBufferPointer to UnsafeMutableRawPointer it's giving multiple errors at the time of build.

Showing All Messages ../Matft/core/util/typeconversion.swift:73:19: Incorrect argument label in call (have 'from:count:', expected 'repeating:count:')

../Matft/core/util/typeconversion.swift:73:40: Cannot convert value of type 'UnsafePointer' to expected argument type 'T'

../Matft/core/util/typeconversion.swift:134:24: Incorrect argument label in call (have 'from:count:', expected 'repeating:count:')

../Matft/core/util/typeconversion.swift:134:45: Cannot convert value of type 'UnsafeMutablePointer' to expected argument type 'Float'

../Matft/core/util/typeconversion.swift:144:18: Value of type 'UnsafeMutablePointer' has no member 'moveUpdate'

../Matft/core/util/typeconversion.swift:154:24: Incorrect argument label in call (have 'from:count:', expected 'repeating:count:')

../Matft/core/util/typeconversion.swift:154:72: Cannot convert value of type 'UnsafePointer' to expected argument type 'Float'

../Matft/core/util/typeconversion.swift:234:18: Value of type 'UnsafeMutablePointer' has no member 'moveUpdate'

../Matft/core/util/typeconversion.swift:241:24: Incorrect argument label in call (have 'from:count:', expected 'repeating:count:')

../Matft/core/util/typeconversion.swift:241:45: Cannot convert value of type 'UnsafeMutablePointer' to expected argument type 'Double'

../Matft/core/util/typeconversion.swift:256:24: Incorrect argument label in call (have 'from:count:', expected 'repeating:count:')

../Matft/core/util/typeconversion.swift:256:73: Cannot convert value of type 'UnsafePointer' to expected argument type 'Double'

../Matft/core/util/typeconversion.swift:269:27: Value of type 'UnsafeMutablePointer' has no member 'moveUpdate'

[jjjkkkjjj]

Which swift version was used? I guess mismatch swift version was caused

[Eumentis-Jayashree]

Xcode -> target -> Build Settings -> Swift Compiler Language -> Swift Language Version: swift 5

from terminal I get: swift-driver version: 1.62.15 Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51) Target: arm64-apple-macosx13.0

[jjjkkkjjj]

Hmm, It's strange. My code can build correctly.

let d: [Float] = [2, 3] // I didn't install ORTValue. So I use a simple array as workaround
let size = 2
let shape = [2,1]
let floatValuesForOutput = d.withUnsafeBytes {
                (buffer: UnsafeRawBufferPointer) -> MfArray in
                // copy data automatically
                let mfdata = MfData(source: nil, data_real_ptr: UnsafeMutableRawPointer(mutating: buffer.baseAddress!), data_imag_ptr: nil, storedSize: size, mftype: .Float, offset: 0)
                let mfstructure = MfStructure(shape: shape, mforder: .Row)
                return MfArray(mfdata: mfdata, mfstructure: mfstructure)
            }

[jjjkkkjjj]

@Eumentis-Jayashree Which Matft version did you use? Is it latest version? And, can you share the all codes about conversion?

[Eumentis-Jayashree]

I figured out what's happening. When I install Matft's specific version (Rules -> Exact -> 0.3.3) 0.3.3 then it's working fine. But when I install it using Rules -> Branch -> master, then it gives these errors.

<img width="1377" alt="Screenshot 2024-05-16 at 12 14 44 PM" src="https://github.com/jjjkkkjjj/Matft/assets/133974684/eefa3744-dd52-4bf2-82cc-e5ab410da985">

[jjjkkkjjj]

@Eumentis-Jayashree Namely, the codes updated after 0.3.3 caused this problem... This commit may be the reason. I reverted it on the revert_duplicate branch. Could you try the codes again in revert_duplicate branch?

[Eumentis-Jayashree]

@jjjkkkjjj It's working on the revert_duplicate branch. Thank you :)

[Eumentis-Jayashree]

I have converted Matft array to openCV mat. Steps:

1. Converted Matft array to NSData

   func matftArrayToNSData(mfArray: MfArray) -> NSData? { guard mfArray.mftype == .Float else { print("MfArray is not of type Float. Actual type: (mfArray.mftype)") return nil }

   // Flatten the MfArray if it's multidimensional let flatMfArray = mfArray.flatten()

   // Convert MfArray to Swift array of type [Float] guard let swiftArray = flatMfArray.astype(.Float).toArray() as? [Float] else { print("Failed to convert MfArray to [Float]") return nil }

   // Convert the Swift array to Data let data = NSData(bytes: swiftArray, length: swiftArray.count * MemoryLayout.size)

   return data 

   }

2. Created function in objective c to convert NSData to mat

   +(NSArray *) matFromNSData:(NSData *)data rows:(int)rows cols:(int)cols{
   // Convert NSData to a byte array
    const unsigned char *byteArray = (const unsigned char *)[data bytes];
   
   // Create a cv::Mat object from the byte array
   cv::Mat mat(rows, cols, CV_8UC4, (void *)byteArray);
   
   // ........performed some operations & converted mat to NSArray
   // return NSArray 
   return nsArray
   }

[jjjkkkjjj]

@Eumentis-Jayashree Thank you for sharing your codes! I'll try to implement the method to convert MfArray to opencv's mat later referencing your code!