dantman / elite-vr-cockpit

A SteamVR Overlay providing a virtual throttle and joystick and holographic buttons for ship functions, with custom integration into Elite Dangerous
Other
320 stars 36 forks source link

SteamVR Input API and support for non-Vive wand controllers #14

Open dantman opened 6 years ago

dantman commented 6 years ago

We should switch from the legacy input API to the new action based input API. By using a series of generic actions (e.g. TrackpadPress, JoystickPress, TrackButtonPress) we can support controllers with large trackpads, joysticks, and joystick/trackbutton pairs.

I already investigated this and partially got it working. The primary barrier at this point is that the SteamVR_Input code is far too complex to implement manually the way that I handled the event and pose update loop, but the SteamVR_Input code assumes that the SteamVR class has an instance it can read settings from (which isn't the case in the overlay).

I may want to work on a PR for the SteamVR Unity Plugin to try and get them to add basic overlay support, so I can drop my custom event loop code and use SteamVR_Input. (ValveSoftware/steamvr_unity_plugin#108)

zubrowska commented 5 years ago

Hey, is There a way to use it in steamvr with Oculus controllers ? VR cockpit Work but i can't use the touch in game. They are not conected and i can't bind anything on them.

dantman commented 5 years ago

Some people have suggested running ED in Windows 7 compat mode helps things work better. Most of the bindings should work, it's just the POV switches that only have trackpad bindings.

zubrowska commented 5 years ago

Thx for your answer, i managed to make the A / X / Y and the triggers to Work, but the two joysticks and the B button don't Work on the Oculus touch.Le 17 déc. 2018 22:28, Daniel Friesen notifications@github.com a écrit :Some people have suggested running ED in Windows 7 compat mode helps things work better. Most of the bindings should work, it's just the POV switches that only have trackpad bindings.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

flavourous commented 5 years ago

Running with rift here, it doesn't detect the game unless i run in compatibility for win7, which forces it to run through steamvr. Otherwise it appears to run natively through rift (doesnt show up on steamvr status).

Problem is, I can't navigate any menus using the rift controllers when running through steamvr it seems? The preset doesn't show up, and the thumbsticks don't navigate the menus/cockpit panels. "Occulus touch" preset is gone from controls options menu too.

I've cloned the repo and had a quick look...I'm a c# dev happy to help. Thinking skip steamvr watch processes directly to detect...or maybe there's a way to map the rift thumbstick to arrow keys or something...

flavourous commented 5 years ago

Got it detecting without the steamVR events, but maybe obviously that doesnt work :D

dantman commented 5 years ago

@flavourous My issues are becoming disorganized spaghetti. Could you make a new one for your issue.

Problem is, I can't navigate any menus using the rift controllers when running through steamvr it seems? The preset doesn't show up, and the thumbsticks don't navigate the menus/cockpit panels. "Occulus touch" preset is gone from controls options menu too.

Could you include display mirror screenshots and a screenshot of the overlay's desktop view.

dantman commented 5 years ago

I was part way through work on this. However it's currently completely blocked by ValveSoftware/steamvr_unity_plugin#455.

dantman commented 4 years ago

😫It turns out that my issue was actually caused by a race condition in the code. My SteamVR Input actions are now back to working.

I may get back to the project in a few weeks after I get over how such a stupid little bug slipped by and stopped me for so long.

dantman commented 4 years ago

SteamVR Input is getting closer to finished. However I still only have a Vive controller to work with. So it's difficult to create any *Joystick actions for Oculus/Index or tune the *Trackpad actions to work with the tiny WMR/Index trackpads.

dantman commented 4 years ago

Ok, I'm now at the point that the lack of hardware is a critical issue.

I have the SteamVR Actions system working with fully implemented support for the Vive Controllers, all parts of the legacy input system (except perhaps poses) completely replaced.

But I cannot create *Joystick actions because I don't have any controller hardware with Joysticks/Thumbsticks. And I don't have any controller hardware with tiny trackpads, so I cannot test how bad the sensitivity of the trackpads are and get controller name strings to auto-tune that sensitivity. And I can't experiment with the Index controllers to figure out where I should put the seated position reset gesture, since the Index controllers don't have an application menu button.

Would anyone with a Rift (S) or Index controller care to test out the feat/steamvr-input branch and check that at least I have not made the Rift/Index controller handling worse.

And would anyone with a WMR care to create a binding file for the WMR controllers. Because I cannot generate even an untested bindings file for WMR like I can Oculus Touch and Index. Which means if I release WMR support is going to go from "bad" to "nonexistent".


Ultimately things would be best if I were given access to a bunch of developer hardware so I could test and develop all of this myself.

OWL282820 commented 4 years ago

Why not get in touch with Oculus direct via their developer channel explaining your project. [https://developer.oculus.com/] I'm sure they'd not let a lack of a few little bits of their hardware prevent you from offering what you already have for Vive. Sell it to them.

dantman commented 4 years ago

Would anyone with a Rift (S) or Index controller care to test out the feat/steamvr-input branch and check that at least I have not made the Rift/Index controller handling worse.

I've made an experimental build for Rift and Index users to test and make sure I didn't break any of the Oculus Touch/Knuckles handling.

https://github.com/dantman/elite-vr-cockpit/releases/tag/v0.6.0-steamvr-input-experiment

jdawgzim commented 4 years ago

I've made an experimental build for Rift and Index users to test and make sure I didn't break any of the Oculus Touch/Knuckles handling.

I have an Index. I'll try to get to testing it tonight.

It would be amazing to be able to map the thumbpads, thumbsticks and buttons to whatever we want in Elite! Using the left hand thumbstick to control strafing would be amazing.

jdawgzim commented 4 years ago

I tried my Index on 0.6.0. I had to update Vjoy to use the 0.6.0 release. Seems to work mostly just as well as last release of Elite VR Cockpit.

  1. Touchpad usage is still as finicky as it was using the Vive touchpads meaning using the Index touchpads isn't any worse. I suffer with it as long as I can and then I end up switching to using keyboard in menus. Sometimes when I middle click it scrolls left or right. I think it'd take a lot of tweaking to get this right.
  2. SteamVR legacy mode is activated so users will need to remap Button A back to being Button A (Defaults to A button = Grip button). I found this to be an issue in any legacy game mode with Index controllers.
  3. Version 0.6.0 was effecting my framerates more. When I turned Elite VR Cockpit off then my framerates would come back in Elite. I have a RTX 2070, I'm running in 120Hz mode, and I'm testing framerates with fpsVR
  4. When I bring up GUI to add buttons it flickers. Out of sync with 120Hz maybe or maybe it's not part of the motion smoothing going on?
  5. Couldn't reset head to center for Elite VR Cockpit. Held down menu button (B button on Index) and it'd vibrate but then not recenter either Elite game or your overlay. I could reset overlay by restarting it and I could reset Elite by holding down Steam button and selecting center reset.
  6. Couldn't map a key for galaxy map rotate, zoom, or angle. Seems like that's still not fully implemented by you.
  7. When I hold a button to turn flight assist off for fast turning the 6 DOF flight grid thing pops up. Then it disappears when I let go. I don't use flight assist as a toggle so it's weird how it pops on and off.
  8. Using the Index controller touch grips works pretty good for grabbing controls and using them. It takes a little practice to use it this way though.
  9. I didn't test overlay object limit. Not sure if it still has the same limit. Does it? I still use VoiceMacro to overcome this.
  10. So main problems: touchpads hard to use in menus, I wish there were better ways to strafe during combat (like being able to map a thumbstick to strafing), framerate drop is worse, and galaxy map controls not working.

I'll have to mess around with it more and see if I can find more things to nit pick. 😉 Otherwise it's working pretty good, just needs more functionality, custom-ability, and polish.

jdawgzim commented 4 years ago
  1. I also wanted to note the touchpads have some kind of weird deceleration going on with the scrolling. If I try to scroll faster then it scrolls the same speed. Maybe it's a max scroll speed it's hitting? Is there a trackball API you can use in SteamVR? Being able to scroll fast by flicking and letting go would be cool and then normal scroll could be less sensitive... maybe
dantman commented 4 years ago
  • Touchpad usage is still as finicky as it was using the Vive touchpads meaning using the Index touchpads isn't any worse. I suffer with it as long as I can and then I end up switching to using keyboard in menus. Sometimes when I middle click it scrolls left or right. I think it'd take a lot of tweaking to get this right.

Yup the trackpad sensitivity is still tuned to the Vive trackpads. I'm thinking of getting the controller name and using that to apply different sensitivity levels. However that of course won't happen till I have Index controllers and WMR hardware so I can decide what the right values are and what strings OpenVR reports as the names of those devices.

  • SteamVR legacy mode is activated so users will need to remap Button A back to being Button A (Defaults to A button = Grip button). I found this to be an issue in any legacy game mode with Index controllers.

That's strange. The whole point of this is to use the new SteamVR Input API, so it shouldn't be using legacy mode. And I should already have Index bindings that assign A to Secondary and B to Alt.

  • Version 0.6.0 was effecting my framerates more. When I turned Elite VR Cockpit off then my framerates would come back in Elite. I have a RTX 2070, I'm running in 120Hz mode, and I'm testing framerates with fpsVR

This is also strange. I'm not sure if there's any other changes I've made since 0.5 that could be related.

  • When I bring up GUI to add buttons it flickers. Out of sync with 120Hz maybe or maybe it's not part of the motion smoothing going on?

That sounds like what happens when you hit the overlay limit. I also got flickers of the edit panel when I had too many overlays. Edit mode adds 4 overlays, so if you have 61-64 overlays already turning on edit mode will immediately exceed the overlay limit.

  • Couldn't reset head to center for Elite VR Cockpit. Held down menu button (B button on Index) and it'd vibrate but then not recenter either Elite game or your overlay. I could reset overlay by restarting it and I could reset Elite by holding down Steam button and selecting center reset.

B is now properly being used for Alt on joysticks and the Menu/UI back button.

The problem is I have no clue what button(s) I should map the seated position reset to for Index users, since it doesn't have an application menu button.

  • Couldn't map a key for galaxy map rotate, zoom, or angle. Seems like that's still not fully implemented by you.

No, I wasn't really planning to. The trackpads/sticks are going to be getting ui navigation bindings in the galaxy map. To control rotate/zoom/angle I made a whole map control UI you interact with using the trigger.

  • When I hold a button to turn flight assist off for fast turning the 6 DOF flight grid thing pops up. Then it disappears when I let go. I don't use flight assist as a toggle so it's weird how it pops on and off.

6DOF is enabled when landing gear is down or flight assist is on. That issue is #15, eventually we'll have options to control what situations the 6DOF controller is available.

  • Using the Index controller touch grips works pretty good for grabbing controls and using them. It takes a little practice to use it this way though.

That's good. The bindings panel does have options to tune the point when grip is activated.

  • I didn't test overlay object limit. Not sure if it still has the same limit. Does it? I still use VoiceMacro to overcome this.

Same limit, this limit isn't part of EVRC. It's built into SteamVR. The only thing I can do that affects this limit is to use more/less overlays for the non-button controls. So the limit will change between SteamVR versions, not between EVRC versions (unless I add a new feature, which will likely just make the limit smaller).

I wish there were better ways to strafe during combat (like being able to map a thumbstick to strafing)

I'm still not sure about using trackpads/thumbsticks for axises. But I still am thinking of adding a 3DOF controller (#51) both 6DOF and joystick styles; after I create a way to enable/disable individual controls.

  1. I also wanted to note the touchpads have some kind of weird deceleration going on with the scrolling. If I try to scroll faster then it scrolls the same speed. Maybe it's a max scroll speed it's hitting?

Yes, there is a hardcoded limit. It's not possible to send instantaneous keypresses to ED. To send an instantaneous keypress not associated with the user pressing/releasing a physical control (i.e. trackpad swipes), I have to hold down one of keyboard keys long enough for ED to hear the press event before I send the key release event, otherwise ED never receives the keypress.

So basically trackpad swipes are hard limited by how long I have to hold down a keypress for Elite Dangerous to hear it.

Is there a trackball API you can use in SteamVR?

Yes, there is a Use as Scroll input type that you can use for binding trackpads to actions. Long story short, it's shit (ValveSoftware/openvr#1229). It also won't be available in OpenXR if we ever get to use it. So I just gave up and added Trackpad{Touch,Position,Press} actions instead.

Being able to scroll fast by flicking and letting go would be cool and then normal scroll could be less sensitive... maybe

Long term this shouldn't be an issue. I do use the trackpad swipes for small distances of menu navigation. But I actually don't use it for long scrolls through the menu. Instead I press and hold a direction on the trackpad. Elite Dangerous scrolls faster through the menu when you hold down an arrow key than when you press an arrow key over and over.

So long term this won't be an issue since I plan to also map the Knuckles joystick to menu navigation in the future. So in the future you can fast scroll using the joystick instead.

jdawgzim commented 4 years ago
  1. I bet the legacy mode is being activated by the use of Elite Dangerous and it being the focus VR app.

  2. So if I relaunched EVRC and Elite the framerates are fine. If I configure and move around things a lot in Elite VR Cockpit (EVRC) then the weird flicker and framerate drops start to happen.

  3. I learned how to reset my seated position to both Elite and EVRC by holding down the SteamVR button for 3 seconds, selecting reset seated position and then doing it again. Sometimes multiple times to tweak it where I want it.

It'll be nice when SteamVR gets the overlay increase to 128 objects. Then I might not have to rely on using VoiceMacro or keyboard anymore. Well I'd still use VoiceMacro a little because it's fun and useful. Mostly not having to use keyboard since it's a pain to take my Index controllers off and on again to do menus.

dantman commented 4 years ago
  1. I bet the legacy mode is being activated by the use of Elite Dangerous and it being the focus VR app.

Wait, where do you actually see legacy mode enabled? The 0.6 experimental version is entirely action based, it doesn't use the legacy buttons API at all. If you can so much as press a holo button or grab a joystick then the actions are working and it's not running in legacy mode.

Are you going into the controller configuration area in SteamVR settings and looking at the Elite Dangerous configuration? That's the wrong location. The overlay and Elite Dangerous have completely separate controller configurations. ED having legacy mode enabled does not stop EVRC from using the new Input API.

However EVRC's configuration is not accessible from the SteamVR settings panel. To my understanding SteamVR pretty much won't show anything in the list of games there unless you install it from Steam, so you won't see the overlay there.

If you feel like installing the Unity editor, try out the latest version of the feat/steamvr-input branch. The SteamVR plugin/OpenVR API recently added a new API to open the bindings page for a game in VR from within the game. So in the unreleased version I have a new button to open the bindings panel.

  • So if I relaunched EVRC and Elite the framerates are fine. If I configure and move around things a lot in Elite VR Cockpit (EVRC) then the weird flicker and framerate drops start to happen.

Framerate drops in Elite Dangerous or just overlay slowness? The flickering still sounds like what happens when the overlay limit is exceeded.

  • If we can get access to using all the buttons, touchpad directions, and thumbsticks to map to whatever Elite button presses we want it'd also be amazing.

I've already got enough button press actions to map all controller buttons to joystick/throttle buttons. And the ability to freely map trackpad swipes and presses to 2 POVs individually. i.e. I have a full set of {Touch,Position,Press} actions for both POV1 and POV2 and also a POV#Button action to handle scenarios where press shouldn't be a d-pad style trackpad press. For example the Vive's trackpad binding works by binding POV1Position/POV1Press to a trackpad binding and POV2Position/POV2Touch to another trackpad binding. So swipes go to POV2 directions, edge presses go to POV1 directions, and center presses are a POV1 button press.

Though Cosmos is making me think I may need to add an Alt2 button, given its controllers have 2 triggers, and a set of A/B buttons. Meaning you could bind Primary/Secondary to triggers and Alt/Alt2 to A/B. Instead of mapping it like the Touch controllers where Primary is the trigger and Secondary/Alt is A/B.

Thumbstick will similarly be able to freely map to POV1 or POV2 (i.e. So you can map the Index thumbstick to POV1 directions and POV1 press, and the Index track button swipes to POV2 directions and press to POV2 press). But that won't happen till I have controllers with joysticks (anything Touch, Index, or WMR) so I can actually develop code to handle how SteamVR maps joysticks to 2D float vectors.

I still have no plans to allow for arbitrary bindings in ship mode when you're not grabbing anything. The number of buttons/POVs would explode, the trigger would conflict, and it might get away with alternatives I hope are possible to implement.

  • I also wish there was a haptic feedback when a joystick maxes out it's angle range. Sometimes I'm contorting my hands way to far because I'm not sure what the max controller angle is unless I constantly watch the animated hand in game. I'd rather keep my head up on the action. Maybe a little temporary haptic buzz each time I hit the max range angle of vJoy. Is that possible?

Haptic feedback is #6. It is something we're definitely missing.

dantman commented 4 years ago

Ok, I'm going to consider the feat/steamvr-input branch complete for now.

It doesn't sound like it's made anything critically broken. No-one seems to want to bother testing the Oculus controllers.

There is still no joystick support or full menu/galaxy navigation on non-Vive controllers yet. But I think it's time I finish working on SteamVR Input and get back to working on other features and bugs.

In particular I've been holding off on relicensing, switching InputSimulator, fixing a recenter bug, and debugging some issues in the Elite Dangerous detection code – because of this branch.

EmteZogaf commented 4 years ago

I have an Oculus Quest with Touch Controllers. I am currently using the Virtual Desktop app with SteamVR support (sideloaded) over WiFi which works pretty well.

I tried the v0.6.0 SteamVR input experiment with vJoy v219 yesterday for about an hour. The thumbsticks didn't respond at all in default settings, but I didn't check the control settings ingame if they were configured. I could only navigate the the UI menus when unlocking edit mode and switching to menu mode. Then i could tap/touch the POV hat directions for navigating. Other then this it worked pretty well and I was especially surprised about the good axis sensitivity when grabbing the virtual HOTAS and turning the ship in all directions.

One problem was the occasional disappearing of the main overlay menu which could only be brought back by guessing were it is and hitting it so it then appeared together with the tooltip. Almost the same happened to the actions overlay menu but it kind of flickered and then went back to normal after 1-2 seconds of strobing. This repeats about twice a minute. The action buttons I layed out in the cockpit didn't have this problem.

I will try it again tomorrow evening (CET) and verify the control settings first. Maybe in a month I will get a working USB cable to use Oculus Link which will present the Quest as a Rift to the system and maybe make a difference.

EmteZogaf commented 4 years ago

And the Galaxy map overlay (arrows for all directions to move) didn't work at all. I tried to hit it with the virtual controller and press the trigger. Maybe I have the wrong keyboard keys assigned to navigate the map?

dantman commented 4 years ago

The thumbsticks didn't respond at all in default settings

Thumbsticks are currently not supported. I do not own any controllers with thumbsticks yet so I cannot develop actions that work with joysticks.

One problem was the occasional disappearing of the main overlay menu which could only be brought back by guessing were it is and hitting it so it then appeared together with the tooltip. Almost the same happened to the actions overlay menu but it kind of flickered and then went back to normal after 1-2 seconds of strobing.

Could you check the desktop window. This sounds like what happens when SteamVR's overlay limit is exceeded.

And the Galaxy map overlay (arrows for all directions to move) didn't work at all. I tried to hit it with the virtual controller and press the trigger. Maybe I have the wrong keyboard keys assigned to navigate the map?

You do need to bind the galaxy map's axis to the map controls. Turn on edit+menu mode and press the galaxy map icon in the upper right corner, that will let you bind galaxy map axis in the controls panel.

EmteZogaf commented 4 years ago

The thumbsticks didn't respond at all in default settings

Thumbsticks are currently not supported. I do not own any controllers with thumbsticks yet so I cannot develop actions that work with joysticks.

Is it possible to map them to additional axes in vJoy? (https://docs.unity3d.com/Manual/OpenVRControllers.html)

dantman commented 4 years ago

Is it possible to map them to additional axes in vJoy?

Not really sure what you're asking.

If you're asking how the thumbsticks are going to work when I do support them. They are going to work pretty similarly to the trackpads:

EmteZogaf commented 4 years ago

I'm likely going to end up needing to disable mirror mode/symmetric input bindings and make it so the right controller's joystick is bound to ui navigation and the left controller's joystick is bound to category navigation.

I see, but what is the benefit of symmetric mapping? Wouldn't you use the same hand for the same action anyway?

EmteZogaf commented 4 years ago

The fundamental issue is simply I don't have any hardware with a thumbstick, so I do not know how the joystick will be mapped to actions and what kind of values a x/y vector binding will get when a joystick is bound to it. So it doesn't matter what I'm trying to make it do, I don't have the hardware to make anything work.

Maybe I could help and test this with a simple example project just for displaying the values of each input on the controllers.

dantman commented 4 years ago

I see, but what is the benefit of symmetric mapping? Wouldn't you use the same hand for the same action anyway?

It's mostly a collection of small issues:

Maybe I could help and test this with a simple example project just for displaying the values of each input on the controllers.

Perhaps, though it's going to be a pain to debug and test. And given the Index is now bing sold in Canada it'll probably only be a month or two before I have Index controllers with thumbsticks to test with.

EmteZogaf commented 4 years ago

OK, I'll try something at home. I need unity and the SteamVR unity plugin, correct?

dantman commented 4 years ago

Mostly just Unity and Git. I recommend installing Unity hub to make managing Unity versions easy. Visual Studio would be good for editing any C#. And VSCode for editing any non-C# files or looking at the json bindings files. And either SourceTree or VSCode with the Git Graph extension – for cloning the code, pulling any changes, and submitting anything back.

The SteamVR plugin code is bundled. Unity should install packages when you open the project. And TextMesh Pro should automatically detect the first start and offer to generate the TextMesh Pro assets needed. After that just open up the Assets/Scenes/EVRC scene.

EmteZogaf commented 4 years ago

Thanks, it has been some years since I last used unity and developed in C#, but I will give it a try.

dantman commented 4 years ago

Thanks, it has been some years since I last used unity and developed in C#, but I will give it a try.

😓I hope you did some web stuff at some point too. Cause like any good web developer, I've built this almost entirely using event driven programming and async coroutines; avoiding the Update loop as much as possible. Unlike a lot of Unity developers and example code.

dantman commented 4 years ago
  • So if I relaunched EVRC and Elite the framerates are fine. If I configure and move around things a lot in Elite VR Cockpit (EVRC) then the weird flicker and framerate drops start to happen.

Framerate drops in Elite Dangerous or just overlay slowness? The flickering still sounds like what happens when the overlay limit is exceeded.

You might actually be right. Going over the overlay limit does introduce its own type of flickering. But I do notice that some of the elements do have a very slight constant flicker. This seems specific to the continually rendering UI elements like the edit panel and 6DOF controller, which suggests it may be related to some sort of synchronization issue, i.e. the camera re-rendering it while the compositor uses it. The holo buttons that use static image textures are fine. And the main panel that is dynamic but only re-rendered on demand is also fine. I can either try fixing this by seeing if I can deal with the synchronization or I could implement some sort of double-buffer setup for the continually updating overlays.

jdawgzim commented 4 years ago

https://steamcommunity.com/games/250820/announcements/detail/3150762393108758897
New SteamVR release. Includes the 128 Overlays! Can we go crazy with the holo-buttons now?

dantman commented 4 years ago

I didn't notice any difference before on the beta. We'll just have to try it out and see if it's actually updated.