Read RGBA int* using ccv_read

[codeclown]

Thanks for the fantastic library. I have it compiled in a iOS project.

I have an RGBA array of pixels like this (origin is a CVPixelBuffer, which I turn into const int* base_address):

// pixel 1
base_address[0] // 246
base_address[1] // 231
base_address[2] // 224
base_address[3] // 255
// pixel 2
base_address[4] // 231
base_address[5] // 224
base_address[6] // 246
base_address[7] // 255
// ...

I try to read it:

ccv_read(base_address, &image, CCV_IO_RGBA_RAW | CCV_IO_GRAY, rows, cols, scanline);

But running ccv_swt_detect_words produces no results. When I load the same image as a PNG, I get expected results. So my method of reading the image from memory must be off.

I've also looked at the unit test but unfortunately my understanding of the underlying data format is lacking so I haven't been able to figure this one out myself.

I am open to organizing the CVPizelBuffer data in a different format (e.g. char * base_address) if my current approach does not make sense.

Is there a way to read an image from an array of ints?

[liuliu]

Yeah, this looks alright. Make sure your scanline is correct (probably 4 * rows, but sometimes CVPixelBuffer can be weird, you can get it from their API: https://developer.apple.com/documentation/corevideo/1456964-cvpixelbuffergetbytesperrow?language=objc

Also make sure your CVPixelBuffer is RGBA format.

Last, to validate the orientation is correct, you should use ccv_write function to write it out as a png file and check if the file is what you expect.

[codeclown]

Yeah... after all, the issues I had (no results from SWT) were due to several problems like image orientation and my pixel array being converted incorrectly (my mistake). Here's the important pieces for anyone who might stumble upon here when searching for how to use ccv in an iOS project:

In captureOutput:

// Make sure your output is BGRA:
assert(CVPixelBufferGetPixelFormatType(imageBuffer) == kCVPixelFormatType_32BGRA);

// Get relevant information from buffer
CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)

// Cast to Int32 array so it can be passed to Obj-c++
let bgra = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Int32>.self)

// Call your Obj-c++ wrapper (you need to implement this yourself)
SwtWrapper().detect_wrapper(bgra, Int32(height), Int32(width), Int32(bytesPerRow));

// Unlock base address
CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)

Signatures and the flow from Swift -> Obj-C++ -> C++:

// .mm
- (NSMutableArray *) detect_wrapper:(int*)base_address : (int)height : (int)width : (int)bytes_per_row

// .cpp
detect(const int* base_address, int rows, int cols, int scanline)

// calling ccv
ccv_read(base_address, &image, CCV_IO_BGRA_RAW | CCV_IO_GRAY, rows, cols, scanline);

Helpful tip! Alter the output resolution to achieve better results. Example:

avCaptureSession.sessionPreset = AVCaptureSession.Preset.vga640x480

Also, make sure to select an appropriate orientation for your capture output:

avCaptureConnection.videoOrientation = .portrait

That's it! With these snippets in place I was able to call feed iOS camera output to ccv and get results back.