Support for the imaging interface in Unity

[zachkinstner]

As far as I can tell, there is not a way to access the imaging data in the current Unity client. I posted a ticket at the OSVR-Unity repository: https://github.com/OSVR/OSVR-Unity/issues/66.

I'm adding a ticket in this repository to notify the Leap Motion team. The imaging interface is important for features like image pass-through and Image Hands in Unity.

[jdonald]

It appears this was implemented three days ago in OSVR/Managed-OSVR#27. Thanks @JeroMiya!

[JeroMiya]

@jdonald There is still work to be done with imaging in .Net. The current bindings are thin - the data is still hidden behind an IntPtr that you'll need to marshal manually at this point (this is to prevent unnecessary copies). Once the data is marshaled, you'll also need to do any conversions needed to efficiently upload the data into a texture in the target framework (e.g. Unity).

[zachkinstner]

I was able to get this working in the "Unity test client" code. See the commit referenced above, and the new LeapImagingCamera class.

Notes:

• Updated to the latest snapshot build of OSVR-Unity (0.2.122-g2ff3c40).
• The ImagingInterface.GetState() method throws a NotImplementedException
• Used Marshal.ReadByte() to obtain each element of the image data (byte[])
• Converted/saved these bytes to a Color32 array.
• On Update(), used Texture.SetColors() and Texture.Apply() to update the texture.
• Used a "changed" flag to ensure that
  • A) the image data is not read more than once per Update() cycle
  • B) the texture is only updated when the image data has changed
  • Without this, the test client was very laggy and/or completely froze the Unity editor

[JeroMiya]

GetState will likely remain unimplemented in ImagingInterface. The current implementation frees the native buffer after the callbacks are invoked (this prevents the server-side image ring buffer from getting blocked). The IntPtr may be invalid by the time GetState is called.

The intent was to implement a Unity wrapper that subscribes to the callbacks, makes the copy then (into a ring buffer), and also converts the image to a texture (a different ring buffer). That being said, some kind of wrapper around ImagingInterface in Managed-OSVR might be useful if everyone is going to at least make that first Marhsal.ReadByte call. Thoughts?

[zachkinstner]

Yes, a wrapper for obtaining raw data and/or Unity textures from ImagingInterface seems like it would be helpful.

The Leap Motion "distortion" images (see latest commit referenced above) use float arrays instead of byte arrays, and there isn't a Marshal.ReadFloat(). I used Marshal.Copy() to obtain the data, but I'm not sure if there is a performance/memory penalty for copying the data this way -- perhaps there is a more suitable approach?

This seems like a good task for ImagingInterface to perform, where it's always done correctly/efficiently given some parameters like: float data, 64x64 dimensions, three channels.

[JeroMiya]

Actually, instead of using Marshall.ReadByte or Marshall.Copy, you can use IntPtr.ToPointer in an unsafe & fixed block (actually I'm not sure if fixed is necessary since the data is in unmanaged memory), and use the data directly in your conversion to a Unity texture. This will save you one copy. I think this will be the fast path.

[zachkinstner]

you can use IntPtr.ToPointer in an unsafe & fixed block

I'll defer to @jdonald on this, as the Unity client for this device plugin was going to be built in-house by Leap Motion. The "test client" code that I've written so far should get them started, but the primary intent of that code is to ensure that all the data is flowing (from plugin to OSVR to Unity) as expected.

[jdonald]

I asked our Frontend team, and we're not sure if unsafe can be used by default in Unity. We could investigate forcing our developers to turn it on.

As we already pay the cost of the extra copy for our Oculus-compatible assets, we can stick with the Marshall.Copy for now.

[zachkinstner]

@jdonald, should we close this issue?

[jdonald]

Sounds good, although it looks like I don't have access to close issues here.