Getting data out of a CVPixelBuffer

[jmbldwn]

I can extract the base address from a CVPixelBuffer using CVPixelBufferGetBaseAddress like this:

let baseAddress = c.CVPixelBufferGetBaseAddress(imageBuffer);

I also know the format, size, and width, and can lock down the base address.

I am having a mental block on getting the data out of the runtime so I can manipulate it in Node. Ideally, I'd like to get the pixel data into a Node Buffer object. I have a pointer now (baseAddress), how do I marshal the data over to Node?

[lukaskollmer]

tbh i don't have any experience with CoreGraphics, so i can't really help you with that. depending on what the pointer points to you might be able to define a struct with that same layout to access the data?

The CVPixelBufferGetBaseAddress docs describe which kind of pointer is returned under certain situations, that might be helpful to you.

[jmbldwn]

How would I access the bytes of an NSData object? I believe I can get the data I want into an NSData object.

Here's the flow I'm working on to do that:

let imageBuffer = c.CMSampleBufferGetImageBuffer(sampleBuffer);
c.CVPixelBufferLockBaseAddress(imageBuffer, 0);
let baseAddress = c.CVPixelBufferGetBaseAddress(imageBuffer);
let bytesPerRow = c.CVPixelBufferGetBytesPerRow(imageBuffer);
let height = c.CVPixelBufferGetHeight(imageBuffer);
let size = bytesPerRow * height;
let data = NSData.dataWithBytes_length_(objc.wrap(baseAddress), objc.wrap(size));

The definition of the last call is: + (instancetype)dataWithBytes:(const void *)bytes length:(NSUInteger)length; I'm not certain I'm wrapping the inputs to this call correctly. For the size, it takes NSUInteger, not NSNumber, so I can't convert it with objc.ns(). What's the right way to pass in an NSUInteger?

FYI, my ffi library configuration is:

const c = new ffi.Library(null, {
    CMSampleBufferGetImageBuffer: ['pointer', ['pointer']],
    CVImageBufferGetDisplaySize: ['pointer', ['pointer']],
    CVPixelBufferLockBaseAddress: ['int', ['pointer', 'int']],
    CVPixelBufferGetBaseAddress: ['pointer', ['pointer']],
    CVPixelBufferGetBytesPerRow: ['int', ['pointer']],
    CVPixelBufferGetWidth: ['int', ['pointer']],
    CVPixelBufferGetHeight: ['int', ['pointer']],

    NSStringFromSize: ['pointer', [CGSize]],
    dispatch_queue_create: ['pointer', ['string', 'pointer']]
});

[lukaskollmer]

-[NSData bytes] returns const void*, which isn't (yet?) officially supported in the objc module.

However, you can easily fix this, simply include the following line directly after the require('objc') call:

require('objc/src/types')['r^v'] = 'pointer';

Calls to -[NSData bytes] will now return a Buffer instance, and you can use the ref module to work w/ that buffer

[jmbldwn]

My update passed your comment in flight. I'll try this as soon as I can get the NSData object created.

[lukaskollmer]

The +[NSData dataWithBytes:length:] looks wrong. Since bytes and length are primitives, both should be passed w/out the wrap call

[jmbldwn]

Yes, a bit of whack-a-mole on my part. What do you mean by using ref to work with the buffer? is the Buffer instance out of - [NSData bytes] a reference to the array of bytes?

[jmbldwn]

I tried this:

let data = NSData.dataWithBytes_length_(baseAddress, size);
let refBuffer = data.bytes();
let buffer = ref.readPointer(refBuffer, 0, size);

But, it's returning null for buffer.

As a sanity check, I am able to write the data to a file using:

data.writeToFile_atomically_('foo.rgb', true);

and the file looks like rgb data.

So, I just need to get that NSData object into a node Buffer object and the terrorists lose.

[jmbldwn]

It looks like when I deref void *, ref gives me null.

It's not clear how the bytes are going get into the memory accessible by JS this way. Getting - [NSData bytes] to return a void * just gives me a pointer, but the data it's pointing to is still in the runtime.

What we really need is a way to copy a chunk of memory from the objective-c runtime to a node Buffer.

Any way to do that?

[lukaskollmer]

Hmm ok so this is all i could come up with so far:

const objc = require('objc');
require('objc/src/types')['r^v'] = 'pointer';

const NSUTF8StringEncoding = 4;

const data = objc.ns("Hello World").dataUsingEncoding_(NSUTF8StringEncoding);
const bytes = data.bytes()

console.log(bytes.readCString());
// -> "Hello World"

const bytes2 = bytes.reinterpret(data.length(), 0);

console.log(bytes2.toString());
// -> "Hello World"

bytes2[6] = 119; // replace 'W' with 'w'

console.log(bytes2.toString());
// -> "Hello world"

console.log(bytes.readCString());
// -> "Hello world"

As you can see, any modifications made to bytes2 are also present in bytes (because they're both pointing to the same address)

Here's another example, creating a new NSData instance and writing to its bytes:

// Helpers
require('objc/src/types')['^v'] = 'pointer';
const a = char => char[0].charCodeAt(0);
const dataToString = data => NSString.alloc().initWithData_encoding_(data, NSUTF8StringEncoding);

const data = NSMutableData.dataWithLength_(3);
const bytes = data.mutableBytes().reinterpret(data.length(), 0)

console.log(dataToString(data));
// -> "" (data is empty so far)

bytes[0] = a`F`;
bytes[1] = a`o`;
bytes[2] = a`o`;
console.log(dataToString(data));
// -> "Foo"

[jmbldwn]

reinterpret seems to do the trick. Here's the sequence that works for me, and I'm actually getting camera data out of a node Buffer:

let data = NSData.dataWithBytes_length_(baseAddress, size);
let bytes = data.bytes();
let bytes2 = bytes.reinterpret(data.length(), 0);
let buffer = Buffer.from(bytes2);
// buffer is my image data.  woot.

Can I assume when I release these objects memory will get cleaned up?

[lukaskollmer]

Yeah i think so. You might however want to switch to +[NSData dataWithBytesNoCopy:length:] to make sure you're reading from the exact same pointer you got from CVPixelBufferGetBaseAddress

[jmbldwn]

Tried that:

let data = NSData.dataWithBytesNoCopy_length_freeWhenDone_(baseAddress, size, false);
// also tried:
let data = NSData.dataWithBytesNoCopy_length_(baseAddress, size);

Getting not-obvious error:

Exception has occurred: TypeError
TypeError: could not determine a proper "type" from: undefined
    at coerceType (/Users/jim/development/node/objc/test/node_modules/ref/lib/ref.js:397:11)
    at Array.map (<anonymous>)
    at Object.ForeignFunction (/Users/jim/development/node/objc/test/node_modules/ffi/lib/foreign_function.js:30:23)
    at Object.msgSend (/Users/jim/development/node/objc/test/node_modules/objc/src/runtime.js:71:14)
    at Instance.call (/Users/jim/development/node/objc/test/node_modules/objc/src/instance.js:97:29)
    at Object.apply (/Users/jim/development/node/objc/test/node_modules/objc/src/proxies.js:21:19)
    at captureOutput:didOutputSampleBuffer:fromConnection: (/Users/jim/development/node/objc/test/video.js:89:31)
    at /Users/jim/development/node/objc/test/node_modules/objc/src/block.js:59:30
    at /Users/jim/development/node/objc/test/node_modules/ffi/lib/callback.js:66:25

[lukaskollmer]

you have to add

require('objc/src/types')['^v'] = 'pointer';

[jmbldwn]

OK, that works.

I already had:

require('objc/src/types')['r^v'] = 'pointer';

Nuance of the difference is not obvious to me. What does the 'r' mean?

[lukaskollmer]

it indicates that it's a const pointer

https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtTypeEncodings.html

[jmbldwn]

Ah, got it. Should these get added to the types table in your module?

[lukaskollmer]

i'll add them eventually, i just haven't gotten around to that yet. i'd also like to support structs, which would require replacing the current implementation (simply hard-coding the different types) with a parser, so that's probably when i'll officially add support for r^v and friends

[jmbldwn]

Cool. I'll eventually make an npm module out of my code for anyone wanting to capture camera frames on a mac. The current options rely on spawning command-line tools and seem to be broken. This approach is definitely more robust.