Non-HMD mode

[DerLeole]

Could this plugin be extended to include a non-hmd mode, in which the VR controllers are used to emulate the wii motes, but the user doesn't wear the headset and is looking on the normal screen.

Calibration could be done by placing one controller where the wii sensor bar/the screen in real life would be, the other would be used to point at the screen from the position the user sits at. If the hmd is worn during setup it could show a model of screen where the first controller is and line with which to aim the second controller.

Something like this would be amazing and possibly really useful.

[MichaelJW]

Hi @Leolele99, I've actually been considering a non-HMD mode (for a slightly different reason: mapping whatever combinations of controllers you have to whatever inputs you like). I wasn't sure if anyone would be interested, so it's great to hear that someone is!

The main restriction I see is that this couldn't work for VR setups that use inside-out tracking, like the Oculus Quest; it could only work for setups that use external sensors, like Vive's Base Stations.

I'll let you know if/when I start work on this :) Thanks for commenting.

[DerLeole]

The outside in restriction I hadn't considered yet. A disclaimer and mode switch at startup, or making a separated release for the non hmd version could limit confusion.

I have a Valve Index and would be happy to help test any future versions.

Thanks for the quick reply. Looking forward to seeing future versions, as inside out tracked vr controllers should be the ultimate wiimote for emulation.

[Jaitsu]

This is something I'd definitely like to see, and was what I had in mind when I went looking for Dolphin motion control support and came across this. It'd be great to just sort of map a rectangle in 3D space to my monitor and be able to use motion controllers (in my case Index controllers) like a real Wiimote/Nunchuk that way.

(...in the interim, my alternative has been using a real wiimote with a cardboard sensor bar made with IR LEDs scavenged from old TV remotes and a resistor wired to USB)

[MichaelJW]

I just tested this out in the latest build - which is very close to being ready to release - and this does kind of work, after a fashion :) I can't test it on an Index but I'll let you both know when this is ready and you can have a go, if you're interested.

[MichaelJW]

Hi @Leolele99 and @Jaitsu, I've just released the latest build: https://github.com/MichaelJW/DorsalVR/releases/tag/v1.0.0

It's more flexible than before, but a bit less user-friendly, as a trade-off; please see the instructions here: https://github.com/MichaelJW/DorsalVR/wiki/Dolphin-Usage-Instructions

I've only been able to test this in the Oculus Quest - if you have the time, please could you let me know how (whether!) it works on the Index? Thank you!

You should be able to use this without the headset as-is, but I expect there will be some problems I'll need to fix before it works well.

[Jaitsu]

@MichaelJW Oh excellent! I'll give that a shot ASAP and let you know how it goes with the Index.

[Jaitsu]

@MichaelJW Uh, hm. Problem. Tried booting this up on my Index, but all I get is an empty grey cube around me. My controllers are invisible, and there's nothing to see around me.

My default.yaml is as follows:

%YAML 1.1
---
imports:
- example/WiiExample.yaml
dolphin:
  exePath: D:\Emulators\GCN and Wii\Dolphin\Dolphin.exe
  configDir: D:\Emulators\GCN and Wii\DolphinData\Config
...

Any ideas?

[MichaelJW]

Will take a look, sorry! Are you using 1.0.0 or 2.0.0?

[Jaitsu]

This is on 2.0.0. If there are logs somewhere, or if you want to add some logging, I'd be happy to help debug. (By the way, of note: Dolphin is not launching during this.)

[MichaelJW]

Thanks @Jaitsu, I appreciate that! I've uploaded a debug build here: https://github.com/MichaelJW/DorsalVR/releases/download/v2.0.1/DorsalVR_v2.0.1_DebugBuild.zip

This should show an message in the DorsalVR window on your desktop if an error occurs; that error also offers a button to open the log file. If not, that log file should be in the /AppData/LocalLow/MichaelJW/DorsalVR/ folder (one directory above the /Config/ folder where your YAML files go) and is called Player.log.

Could you attach that log file here please?

It will contain a little bit of info about your computer (like what graphics card you have) so if you're not comfortable sending that then just copy the last few lines here - they should give the error message and the line of code that caused it. For instance, I just ran it with a fake path to break it and the log ends with this:

DirectoryNotFoundException: Destination directory not found: C:\123FakePath\Config
  at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x00167] in <a1e9f114a6e64f4eacb529fc802ec93d>:0 
  at DolphinConfigManager.SetControlINIs () [0x0000d] in C:\Users\micha\Documents\UnityProjects\DorsalVR\Assets\External\Dolphin\DolphinConfigManager.cs:55 
  at SettingsManager.OnEnable () [0x00639] in C:\Users\micha\Documents\UnityProjects\DorsalVR\Assets\SettingsManager.cs:155 

(Filename: C:/Users/micha/Documents/UnityProjects/DorsalVR/Assets/External/Dolphin/DolphinConfigManager.cs Line: 55)

Thank you in advance!

[Jaitsu]

Well, uh, long story short: my fault lol

I mistakenly edited the example path to example/WiiExample.yaml, and correcting that to examples fixed the issue of it not starting! Dolphin now launches, and I'm seeing both of my Index controllers as white rectangles (which are aligned with about the same axis as the pointer in the desktop view, so that feels good btw). The pointer works! But might need some sort of anti-jitter or a range setting (i.e. how much of an angle from center reaches the sides of the screen; right now it's only about a 20-degree arc from the left to the right edge of the screen).

The bindings are a little odd right now, however. Buttons are okay (trigger for B/Z, buttons for A) and I see the grips are nunchuk C and recenter, though for the Index controller at least you might need to read the grip pressure - as it stands right now, holding the controllers with closed hands (with no pressure) is constantly holding C and recentering the pointer.

One other thing: is there a way to disable the automatic side-by-side setting for Dolphin for the sake of trying this out in on the desktop/regular monitor?

[MichaelJW]

I mistakenly edited the example path to example/WiiExample.yaml, and correcting that to examples fixed the issue of it not starting! Dolphin now launches, and I'm seeing both of my Index controllers as white rectangles (which are aligned with about the same axis as the pointer in the desktop view, so that feels good btw).

Ah, phew! I was worried it was going to be something really obscure and hard to fix :D I will add some kind of error in future versions, or a UI to select your files, or something. Glad it's basically working now though.

I've made some new YAML files for you to test here, since I haven't written any documentation yet: Jaitsu.zip Extract this to your /DorsalVR/Config/ dir (so e.g. you will have /DorsalVR/Config/Jaitsu/Jaitsu.yaml) and edit your default.yaml to simply say:

%YAML 1.1
---
imports:
- Jaitsu/Jaitsu.yaml
...

I'll explain what I've changed below:

The pointer works! But might need some sort of anti-jitter or a range setting (i.e. how much of an angle from center reaches the sides of the screen; right now it's only about a 20-degree arc from the left to the right edge of the screen).

Unfortunately I'm a little restricted by Dolphin, currently. Since their gyro aim code is designed for controllers like the Joy-Con and the DualShock, which don't have the ability to detect where they are in space, only how they're moving, there's no way for me to pass through any additional information about where the VR controller is. It's hard coded to be... I think 1 metre away, off the top of my head, maybe 2m.

But I just played with adjusting the size and position of the virtual TV and what I ended up with feels more natural, at least on my setup. You can edit these yourself in /Jaitsu/WidescreenTV.yaml - look at positionOffset and scale in particular.

The bindings are a little odd right now, however. Buttons are okay (trigger for B/Z, buttons for A) and I see the grips are nunchuk C and recenter, though for the Index controller at least you might need to read the grip pressure - as it stands right now, holding the controllers with closed hands (with no pressure) is constantly holding C and recentering the pointer.

Interesting - there's a difference between the Index and Oculus that I didn't know about, thanks.

I've changed the settings in /Jaitsu/WiimoteNunchuk.yaml to something that I think will fix it. If not, play around with them some more using this as a reference and see if that helps?

One other thing: is there a way to disable the automatic side-by-side setting for Dolphin for the sake of trying this out in on the desktop/regular monitor?

Yes - this happens because WiiExample.yaml imports Stereoscopic.yaml; by removing this import it should use your normal Dolphin stereoscopic settings (or lack thereof).

But just to be sure, I've added this option to Jaitsu.yaml which tells Dolphin explicitly not to use any stereoscopic effects:

dolphin:
  config:
    Graphics.Stereoscopy.StereoMode: 0

Hope that helps! Thanks for the feedback and please let me know how well (or not) this works for you.

[Jaitsu]

That got it into flat-screen mode, thanks. No dice on the control change, I'm afraid - I'll try out some other yaml edits based on that control reference page and see if I can figure anything out, though.

Also, odd quirk I discovered, but it might be a Dolphin bug - the last few times I started it up, the pointer kept drifting to the side (like the controller was rotating about 45 degrees over 6 seconds or so) even though both my real controller and the virtual wiimotes within DorsalVR were staying still, and it seemed to change speeds and directions depending on how fast the controller re-entered the screen after being pointed offscreen - but when I opened the Dolphin controller settings to see if I could see anything unusual, that problem mysteriously disappeared and now it's behaving normally and hasn't come back. Will also let you know if that one reappears.

Edit: Okay, using {RightHand}/gripForce instead of gripPressed does the trick and requires squeezing the grip to trigger the button - trouble now is it requires maximum pressure i.e. squeezing the life out of the controller. Still better than not being able to rest my fingers on the grips, but could get uncomfortable after a while. If you can use a float rather than a boolean, it looks like just "grip" likely returns a float between minimum and maximum (though I don't know the max value, I'd guess 0.0~1.0?) that could be used to set a threshold.

Edit 2: Actually, after swapping Z and C (so left trigger is Z and grip is C) even this might be fine given how rarely the C button is used (along with recenter, ideally). Also, the drift issue came back, but I think I've tracked it down to happening if the controller isn't fully tracking at the moment I start DorsalVR? I'd been putting it down in a spot where it couldn't see my base stations, but when I avoid doing that, the drift issue doesn't seem to happen for that session. Odd Unity tracking fallback perhaps?

[MichaelJW]

Thanks @Jaitsu, this is really useful.

Also, the drift issue came back, but I think I've tracked it down to happening if the controller isn't fully tracking at the moment I start DorsalVR? I'd been putting it down in a spot where it couldn't see my base stations, but when I avoid doing that, the drift issue doesn't seem to happen for that session. Odd Unity tracking fallback perhaps?

Nice find. I saw this bug happen exactly once but couldn't reproduce it. It reminds me of an earlier bug that was present in Dolphin where analog sticks and D-Pads would act as if they were pointing bottom-left when actually they were centred. Filoppi fixed that here: https://github.com/dolphin-emu/dolphin/pull/9771

Even if that's not it, it sounds like I can at least paper over the bug by waiting until the controllers are tracked before doing anything. I'll sort that in a future update :)

Okay, using {RightHand}/gripForce instead of gripPressed does the trick and requires squeezing the grip to trigger the button - trouble now is it requires maximum pressure i.e. squeezing the life out of the controller. Still better than not being able to rest my fingers on the grips, but could get uncomfortable after a while. If you can use a float rather than a boolean, it looks like just "grip" likely returns a float between minimum and maximum (though I don't know the max value, I'd guess 0.0~1.0?) that could be used to set a threshold.

Internally DorsalVR checks the float value of whatever's mapped to Recenter and passes true if it's more than 0.5, while for Z it passes the float converted to a byte (i.e. 0.0 -> 0, 1 -> 255), so in theory just mapping /grip or /squeeze/value should work.

Looking at Unity's input debugger for the Touch controllers, I can see that /gripPressed jumps from 0 to 1 when /grip becomes 0.25 - I guess the threshold for the Index is much higher, maybe even 1.0.

A Processor might fix this. For instance, I added an axisDeadzone processor to my Nunchuk Z binding like so:

    Z: 
      path: <XRController>{LeftHand}/grip
      processors: axisDeadzone(min=0.8,max=0.9)

(Again, sorry this isn't documented yet.) What this does is effectively change that threshold from 0.25 to 0.8. So maybe in your case something like this would help:

    Z: 
      path: <XRController>{LeftHand}/grip
      processors: axisDeadzone(min=0.3,max=0.7)

Hard to tell without testing, though. Please let me know if that works; if not, I'll look at creating a new build with some better debugging so we can see what's going on.

[Jaitsu]

After seeing how you implemented the D-pad, I actually tried exactly that, with axisDeadzone, after reading the documentation some more - both with \<XRController> and with \<ValveIndexController> in case it's implemented differently - but no matter what I do it always seems to respond at either the slightest touch or max pressure depending on whether I use grip or gripForce (or \<ValveIndexController>'s gripForce which is supposedly a float), regardless of axisDeadzone setting. Unless the capitalization of axisDeadzone matters (I used lowercase axisdeadzone), which I can check later.

(also, is this getting too far off-topic from the original 'non-HMD mode?' I could break this off into another issue if we want to keep this one focused on that in particular.)

Edit: No dice on the capitalization change, no matter what combination I use it seems to be either max pressure or no pressure. ...And I was wrong on the controller tracking causing the drift; it just seems to happen at random regardless of where the controller is or what I'm doing with it at startup time.

[MichaelJW]

(also, is this getting too far off-topic from the original 'non-HMD mode?' I could break this off into another issue if we want to keep this one focused on that in particular.)

Yes, good point - please do :)

In the meantime, I hacked in a quick debug UI to a build here: https://github.com/MichaelJW/DorsalVR/releases/download/v2.0.1/DorsalVR_2.0.1-InputDebugger.zip

You can add a new debug node to your YAML to use it like so:

debug:
  bindings:
    - <XRController>{LeftHand}/grip
    - <XRController>{LeftHand}/gripPressed

This will show you what's being read by the binding paths you specify. (No support for processors yet, or for seeing what gets passed to Dolphin.)

Please could you let me know what's actually being output when you squeeze the grip? And I guess that can go in the new issue to make this cleaner. Thanks!

[Jaitsu]

Just to chime in on this issue again since I've been doing some experimenting - while it's not quite a true "no-HMD mode", just removing the Stereoscopic/WidescreenTV.yaml (etc) from the import chain still has the controllers working fine while disabling the screen-capture portion, which helps performance when it's not needed while not using an HMD.

Might be the closest we can get, since I'm not aware of any way to have SteamVR track the controllers via the headset without having the display connected and the compositor running (because at the least otherwise I think you'd need to have tracker dongle for each controller and use a null HMD driver and that seems needlessly complex just to avoid using the HMD.)

[MichaelJW]

@Jaitsu That's a great point, I hadn't considered disabling mirroring when not using the HMD.

Is it easy enough to line up the real screen with the virtual screen? I can add a "fake" screen (i.e. just a rectangle that shows where the tracker acts as if the screen is for tracking purposes) if not.

And I think you're right about using controllers without headset - I don't think it's feasible within Unity at least.

[Jaitsu]

@MichaelJW Actually, just pointing at the real screen and hitting the "recenter" bind works perfectly, I don't even have to touch the HMD.

My only remaining issue that's getting in the way of "Non-HMD mode" is a strange performance problem (which might be more of a SteamVR/Unity thing, and also might be grounds for another, separate issue) - even though, with screen mirroring disabled (no WidescreenTV/Stereoscopic.yaml, no yellow rectangle), all DorsalVR is rendering is a simple grey cubic 'room' and the two controller rectangles, for some reason it's using 30% of my laptop's 1660 Ti according to Task Manager, which sometimes noticeably cuts into Dolphin's performance compared to running Dolphin alone. Will have to try it on my desktop and see if the same thing happens.

[MichaelJW]

@Jaitsu 30 percent! That's nuts.

It is possible to remove the controller cuboids via the YAMLs now but it's not possible to remove the "room" at the moment. When I get a chance I'll make a build that lets you do that and we'll see what that does to performance.

But it's good to know that "Recenter" does the job on its own. That might change if and when sensor bar simulation gets added to Dolphin but no sense worrying about that yet.

[Jaitsu]

@MichaelJW Yeah, oddly enough. You'd think three super-basic untextured cuboids (two controllers and the room) would be practically free performance-wise, but it's using quite a bit:

Seems to be down to 17% usage or so now, and 7-8% CPU, I might've had my laptop in "Silent" (low-power) mode when I last looked. Forcing DorsalVR's resolution down to 20% (minimum) through SteamVR bumped its usage down to 9-11%, CPU usage unchanged. Forcing its framerate down brought both usages down a bit further (to about 6% for both, with the low resolution also applied), but it seems the tracking is tied to framerate, as lower framerates seem to cause less-accurate tracking. (45 or so is interesting, though, as the change in accuracy is negligible but seems to lessen the pointer jitter)

[MichaelJW]

@Jaitsu I just tested this out and I see ~12% GPU usage on my system. Curiously, even if I completely disable the cameras so no rendering is required, this only drops to ~9%! Maybe this is some overhead that Unity or SteamVR reserves... not sure.

I haven't added this to a build yet (I'm a little pressed for time at the moment) but I will do when I get a chance.

[Sleepy9000]

I've been trying to get non-hmd to work on HTC Vive and have yet to get it working. The game wont take any input from the vive controllers.

[snowmobile2004]

Got it (mostly) working with my Rift S. I used the attached config (Jaitsu), and the cursor and stuff works fine. One thing - make sure youre blocking the proximity sensor on the headset itself (i use a microfiber cloth) or else the controller wont register any input. One issue im facing is i cant do any "pull back" motion in the game im trying to play - not sure why that is. Seems to be missing the 3d space data, as i can point the cursor and tilt from side to side but i cant seem to translate (maybe just 6dof is broken?). I'd appreciate any assistance, thanks.

[MichaelJW]

Got it (mostly) working with my Rift S. I used the attached config (Jaitsu), and the cursor and stuff works fine. One thing - make sure youre blocking the proximity sensor on the headset itself (i use a microfiber cloth) or else the controller wont register any input.

That's great to hear! And good call on covering the proximity sensor.

One issue im facing is i cant do any "pull back" motion in the game im trying to play - not sure why that is. Seems to be missing the 3d space data, as i can point the cursor and tilt from side to side but i cant seem to translate (maybe just 6dof is broken?). I'd appreciate any assistance, thanks.

I think - and this is just an educated guess; I might be wrong - that this is because Dolphin hard-codes the distance from the screen that the Wii Remote reports its position. More accurately: Dolphin uses accelerometer and gyroscope data to simulate the movement of a "Virtual Wii Remote" internally, which in turn simulates how the virtual Wii remote "sees" the sensor bar, but this does not simulate forwards or backwards movement.

So unfortunately I don't think I can fix this using the standard input method that Dolphin offers. Fortunately I am working on another way around this and I might be getting close - see #15 :) But it's going to be a while yet before I release that new version, I'm afraid.