Accessing CVRCompositor

[mollybeam]

Hi

I'm using OVRSharp to power an Overlay, but I need to access CVRCompositor.GetMirrorTextureD3D11 (in a manner similar to OBS-OpenVR-Input-Plugin) for some application functionality.

I understand OVRSharp doesn't currently implement the compositor, so I'm wondering if you might be able to provide me some guidance on a horrible hacky way to access it?

I'm very new to developing with VR, so I'd like to avoid rewriting an entirely new setup with Valve.VR if possible, though I recognise avoiding it entirely isn't possible here

Many thanks

[tjhorner]

This shouldn't be too hard to implement; I can probably push a pre-release version with this API exposed later today.

The hacky way to do it would be to access Valve.VR.OpenVR.Compositor directly, which should be possible because it's public inside of OVRSharp:

OpenVR.Compositor.GetMirrorTextureD3D11(/* whatever params */);

Just make sure you do this after OpenVR is initialized, which should be the case after you've constructed an OVRSharp.Application. You should be able to do this for any OpenVR API not exposed by OVRSharp.

[tjhorner]

This shouldn't be too hard to implement; I can probably push a pre-release version with this API exposed later today.

I might have accidentally lied here. Since this deals directly with pointers, different graphics APIs, and manually allocating/releasing resources, this requires some special consideration to make using this API at a higher level simple to use rather than it just being a thin wrapper.

So, for now, I think using OpenVR.Compositor directly should be fine.

Edit: Once you're done implementing GetMirrorTextureD3D11 in your app, could you send me some snippets of how you used it? It would really help inform how to add it to OVRSharp.

[mollybeam]

https://gist.github.com/sasshunter/2fd045565bc3241ab1207d9ab0d0ec61

This is roughly what I'm trying to do, though this example doesn't actually work and is absolutely the wrong way of doing it. I plan on then firing this image at a Google API (and later a foss alternative or two) and scanning for text, QR codes etc.

[tjhorner]

Thanks! I'll tinker with this over the weekend to see what the most ergonomic way of working with this at a higher level is. I haven't actually worked with DirectX in C# before, so I'll have to read up on that.

Also, that's a pretty cool project idea. QR code scanning sounds especially useful because I've seen QR codes scattered around VRChat worlds and such.

[tjhorner]

I think this would probably be easier with OpenGL, since there is a well supported library, OpenTK, which supports that. I'm working on supporting that in this branch. So far it seems the implementation should work, but I'm running into a weird issue where the OpenVR API always gives me an InvalidTexture error, and it seems like I'm not the only one. I commented on that issue, so hopefully Valve will respond and provide guidance. If they don't respond in around a week I will reach out to one of my Steamworks contacts to see if they can help here.

I found this project which seems to use that method successfully. I can't find any real differences in how it uses that versus my implementation.

---

As for the OVRSharp implementation, I decided to leave open an ICompositorAPI interface that is graphics API-agnostic. This way, I can ship the OpenGL and DirectX implementations as separate libraries (I've already created OVRSharp.Graphics.OpenGL), and anything that depends on this functionality can just use the interface and not worry about which graphics API is being used under the hood. Right now it just implements GetMirrorImage which returns a Bitmap.

The downside to this is that it's copying the texture into memory and turning it into a Bitmap, so there is probably going to be a big performance hit if you are going to be using this method a lot (i.e., if you need a constant stream of the mirror). But based on your use case it sounds like it will mostly just be one-off captures, so it should be fine.

[mollybeam]

This is brilliant - you're a star and you've probably saved me a couple of weeks of confusion. Thank you so much

[tjhorner]

No problem! I'm glad people are finding this library useful.

I implemented a DirectX-based version of the ICompositorAPI; you can find it in the mirror-api branch. It seems to work, unlike the OpenGL implementation. So after I do some cleanup I will publish this new version (probably sometime tomorrow).

But as I noted before, that method is really slow. It does a lot of work copying the texture from the GPU to the CPU, from the raw texture into the bitmap, etc. Basically, just make sure you don't fire calls to it rapidly, or else your app might lag 🙃

I tested it against SteamVR today and it works like a charm. Here's a sample:

[tjhorner]

Ok, I just published a pre-release version with the new API and the DirectX-based implementation to NuGet. They're indexing right now, but once that's done you should be able to install both. Let me know how it works for you.

Should be able to do:

var compositor = new DirectXCompositor(); // make sure you only have a single instance (I might turn this into a singleton)
var bitmap = compositor.GetMirrorImage(); // optionally, you can provide the eye to grab

[tjhorner]

@sasshunter any updates? If I don't hear back in a few days I'll just assume that it works and I'll close this issue lol (PS: this is in v1.2.0 stable now!)

[mollybeam]

Hi! Sorry! I've got the attention span of a goldfish and completely stopped working on my project for a few days >_>

I did set up GetMirrorImage and it hasn't thrown any errors so I assume it works, but i've not had the chance to actually check the output yet. Once again, many thanks for making this!