Feature Request: Add NullDriver-like Controller support using Ping pong ball tracked controller

[EtherealIntellect]

Basically as a mirror to https://github.com/cboulay/PSMoveService/issues/222

Adding a feature for head tracking and eventually controller tracking through the sensor fusion algorithms from PSMoveSteamVRBridge but from other sources like Colored ping pong ball or marker and/or/older phones. Afaik other implementations like https://hackaday.com/2016/12/16/revealed-homebrew-controller-working-in-steam-vr/ still have problems with trowing objests even though they technically work.

[ninjav69]

I'd be interested to take a stab at this if anyone can provide some clarity on how this should be approached.

[g-coded]

Could the third controller just be an emulated bluetooth device?http://btsim.sourceforge.net/btsim_faq.html
If psms isn't using any rotational/accelerometer data(hmd tracker), would a 'dummy' bluetooth device work?

[ninjav69]

The emulation of a bluetooth device has been suggested by someone else before, and one of the authors said that it's not the "correct" way to solve the problem. What he suggested was that a new device type be specifically coded for the application of tracking ping-pong balls - and hinted that a virtual bluetooth controller would be a nasty kludge.

There seems to be allot of interest in having controller-less tracking in the user base, but unfortunately that seems to not be a priority for the devs at this time.

I personally wouldn't mind having this feature myself and would gladly contribute such code to the project but the devs unfortunately doesn't show allot of interest nor provides allot of assistance.

[HipsterSloth]

@ninjav69 I'm sorry for not being very responsive on this issue for a while. I haven't really been available for any support for this due to a lot of other demands on my time lately (getting game ready for E3 at my day job, teaching prep, mentoring a high school student). I do care about this feature though. What do you need from me to work on this feature? Have you forked the psmoveservice repo and gotten that to build yet?

[ninjav69]

Greetings @HipsterSloth. I understand your dilemma - I'm a full time developer myself.

First and foremost I want to take the opportunity to say thanks for all your work - I'm currently using the HMD position tracking feature for position tracking in Elite Dangerous and I'm loving it.

I have forked the project and I've taken a peek at the code. It's been a while since I did any C++ but I'm not scared of it. I'm obsessive enough to make the project build - I've just been waiting for some motivation.

I suppose the most useful information for me and any other possible contributors would be to have a description of the general overarching architecture complimented by a roadmap of the steps involved to produce a new device type to the system.

If the information from above could be made available to anyone via your web site - then maybe even more people can sink their teeth into this.

Thanks for taking the time to respond.

[HipsterSloth]

@ninjav69 Documentation was actually one of my current work items. I've spent some time on and off working on getting the client interface documented in Doxygen. I was trying to get the docs to autogenerate for the github page here: https://codedocs.xyz/HipsterSloth/PSMoveService/. Unfortunately, I've had some trouble getting them to update automatically there (what's posted at that link is the doc build before I did my client api pass). If you have doxygen installed you can open the doxygen settings file [here] (https://github.com/cboulay/PSMoveService/blob/master/Doxyfile).

As for the rest of PSMoveService I still need to write a detailed description of how the system works. I'll start with the implementation roadmap and post that in the wiki. I'll take a stab at getting something posted for that tonight and then link that here.

More general documentation will take a while. Probably best to just start looking at the code and asking questions. I'll focus on documenting what you are getting stuck on first.

[ninjav69]

Thanks. That's a step into a direction. I was hoping for a high-level description of the architecture so that everyone doesn't need to reverse engineer those insights from the code - it would get us up and running allot faster.

I'm going to assume a plugable architecture where concrete device implementations implement a well-defined contract. I also suppose there's some kind of kernel containing the main control loop and delegating to device implementations through those well-defined contracts.

If you could point to the right core classes to start looking, that would be just splendid.

[HipsterSloth]

Turns out Chad (the other PSMoveService maintainer) had started some programmer notes awhile ago on the PSMoveService wiki that I had forgotten about. I added some more detail to it about high level structure of PSMoveService. There is still a ton more detail to fill in, but this will at least get you started:

https://github.com/cboulay/PSMoveService/wiki/PSMoveService-Programmer-Notes

I'm going to assume a plugable architecture where concrete device implementations implement a well-defined contract. I also suppose there's some kind of kernel containing the main control loop and delegating to device implementations through those well-defined contracts.

Yeah that's basically what the "DeviceManager" sub-system is. I'm going to continue working on filling that part of the documentation out tomorrow. Currently all I have in that part of the wiki is a bunch of outline stubs.

If you could point to the right core classes to start looking, that would be just splendid.

I'll make another post tomorrow night with some more specific links once I have more of the details about how the controllers get updated documented.

[ninjav69]

Oh that documentation is just brilliant. I'm sure the community is going to really appreciate it!

Can't wait to get my dev environment lined up. Thanks allot.

[HipsterSloth]

I updated the device manger documentation here: https://github.com/cboulay/PSMoveService/wiki/DeviceManager-Programmer-Notes

I'll fill out the controller and tracker manager notes tomorrow. That's where most of the relevant work for implementing a virtual controller will be done.

[zelmon64]

@ninjav69 I did start working on this a while ago in this branch but got rather muddled and waylaid by other issues so I decided to stop and wait patiently for the documentation. Are you happy to take over or shall I try to continue where I left off?

[ninjav69]

@zelmon64 I see no reason why you shouldn't continue with your implementation. Sounds to me like you're already familiar with the code and know what to do. I think I'm speaking on the behalf of the entire user base when I say "Please do!?". There are many DIY VR enthusiasts who just don't have the budget to source the full blown controllers - especially since the price hike due to PSVR. This could also open the door to multi-node tracking - something that the Vive is already starting to do.

So yes, please continue.?

[HipsterSloth]

@zelmon64 I'm happy to help out with this effort btw. I'll sync your branch tonight. Was there a set of documentation that you would find particularly helpful for you. I'm trying to fill out all of the new programmer notes section when I can, but I want to focus on parts that people are trying to reference now first.

[ninjav69]

@HipsterSloth I always find diagrams very intuitive when describing high-level systems design. A simple UML class diagram complemented with a sequence diagram would communicate the documentation on the wiki very well. Perhaps even a deployment diagram showing how a client (such as the freepie bridge) interfaces with PSMoveService. It's not obvious at first glance and I think such things helps to know up-front: even for the users: to understand the product and its parts a little better.

[zelmon64]

@ninjav69 broadly speaking I'm familiar with PSMoveService but I don't really have a handle on how the controllers are plumbed in yet (which is why I couldn't have helped when you asked - sorry).

@HipsterSloth thanks for your support as always - I'll be needing it. I think where I got lost was trying to figure out which parts I could cut out completely and which have to remain to give a connected response. I'll have to refresh myself before I can ply you with more specific questions.

[ninjav69]

@zelmon64 Yeah I had exactly the same question while reading the docs - it appears that controllers are kinda part of the core design and what's more controllers seem to assume bluetooth connectivity. I think the thread title "NullDriver-like Controller" says it all - one would need a NullController implementation that does nothing but sit there. It's either that or a refactor of the kernel to make it optional.

[HipsterSloth]

@ninjav69 Controllers are part of the core design, but being connected via bluetooth isn't required. What is needed is a new ControllerEnumerator type for the null controller. The documentation for the controller manager and device enumeration are still pretty heavily in outline form. But the basic idea is that you have an enumerator for a type of device connection (bluetooth, USB, or Gamepad API). When the controller manager gets notification that a new device has been connected, we create a ControllerEnumerator (which has child enumerators for each type of connection). If we can't find any pre-existing controllers opened with the matching enumerator signature we open a new device of the type corresponding to the enumerators device type.

So in summary we need:

• A new NullControllerEnumerator
  • Probably will read from the ControllerManagerConfig.json to get a list of pre-defined null controllers?
• A new NullController (implements IController interface)
• Add a new entry to the device type enum for the null controller type
• Update the google protobuf file to take into account the new controller type
• Update the client API to have structures for the null controller type
• Update the config tool to allow configuration of the null controllers (set the tracking color type?)
• Update the steamvr plugin to accept the new null controller type

[HipsterSloth]

Also I guess it's worth asking what the main goal for the null controller is. If we are just going to use this to track a headset, maybe the interface we should be creating is a null-hmd instead. Basically everything I said before would still hold except replace "controller" with "hmd"

[ninjav69]

@HipsterSloth It might be due to my poor understanding of the design, but I don't really understand why there's a distinction between HMD and controllers in the first place - I'd expect to rather see a position- and orientation source as the de-facto abstraction for both HMD and controllers. Both devices deal with orientation and position after all. In this case only position is required, yet it's kinda hard to represent that with the current design.

[HipsterSloth]

@ninjav69 steamvr does this. They just have tracked devices as a common base. However in psmove service not all controllers are tracked or have an in (the psnavi in particular). I kept the two device separate because it made it easier to ask questions like how many controllers vs hmd are connected, getting a list of controllers, polling for device connection at different rates (controllers connect/disconnect more frequently then hmds do). This is not to say that it couldn't be compressed into a single abstraction, but there are just enough differences that it made more sense to split them up.

[ninjav69]

I suppose the psmoveservice freepie bridge is going to have to add support for HMD then. It appears that they currently only support controllers. I'm kinda dependent on having this NullController device supported via their bridge.

[EtherealIntellect]

Not sure how they would interact or if it even matters, OpenVR Input Emulator v1.0 released a couple of days ago

[HipsterSloth]

@EtherealIntellect That is actually super interesting. OpenVR Input Emulator wouldn't affect how the null controller work is implemented on the PSMoveService side (still need to do all the work I outlined above). But it does affect how we might want to expose it to the steamvr side. I chatted briefly with matzman66 about his plugin. I'm going to write a PSMoveService hook for it because it looks like it should be pretty straight forward for me to do so. OpenVR Input Emulator might eventually eliminate the need for PSMoveSteamVRBridge, but I need to look further at his code to see what additional work would be needed to have feature parity.

[Nerdenstein]

Hi I'm looking into doing the same kind of thing. I have ping pong light thingy made and I hope this becomes a actual thing.

[Nerdenstein]

Has there been much progress on this?

[Sp4iK]

This ping pong ball thing may add the possibility of tracking hmd positions, but also the PS Navi to use it as an alternative PS Move controller, doesn't it? That would be great!

[ninjav69]

Everyone REALLY want this. That much is clear.

[Nerdenstein]

How hard is it to implement something like this?

Thanks

On Tue, May 16, 2017 at 7:01 PM, ninjav69 notifications@github.com wrote:

Everyone REALLY want this. That much is clear.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/HipsterSloth/PSMoveSteamVRBridge/issues/2#issuecomment-301720371, or mute the thread https://github.com/notifications/unsubscribe-auth/ASxfejs7z2eMVX8uivMFZ-IbQZp9KBSiks5r6WX5gaJpZM4M2yvb .

[HipsterSloth]

I started work on the virtual hmd stuff last night. No promises on when it'll be done. The virtual controller (bulb + pnavi) makes less sense to implement because you won't get an orientation off of a bulb. For the virtual hmd you get the orientation from the phone you are using in FreePie.

[Sp4iK]

Hey, that's good news! About the ps navi, I thought it had a gyro as his bulb brother, but if not the case it does not make sense to track it. Thanks!

[EtherealIntellect]

The idea/hope for the virtual controller was being able to use an old android phone with freepie/wireless imu and bulb tracking just like the head is tracked. Something like they used on the daydream hack but with a normal dualshock controller. Here is a very bad mspaint illustration (edit:the bulb should probably be over the right trigger instead of the middle). I know it looks clunky, but considering older phones are something a lot of people have left over from when they upgraded or from a brother/sister/mother, I still think it's worth looking into. And even the vive uses the same accelerometer they use in a lot of phones, like my s5

[g-coded]

I was thinking the wiimotes would be a good alternative. The buttons aren't all in an optimal location, but grip and thumb are where they need to be. Is there a limitation to the wiimote sensors that wouldn't work?

[Nerdenstein]

The Wiimote It self only has a accelerometer. It also has a camera on the front end. That's what the black piece is at the front That camera tracks the sensor bar which is basically a strip of leds. Knowing where the leds are in its view it can triangulate its position. So the Wiimote itself wouldn't work. But If we're talking about the Nunchucks thats a different story. The Nunchucks have gyros so It can track rotation which is good. Now to add tracking you would pair with Bluetooth two Wiimotes to your PC. Then you would plug a Nunchuck into both controller. You would then put the wimotes in your pocket or vest or something like that then you would use the Nunchucks as the controllers. Now the Nunchucks don't have position tracking so that's when your would stick a led in a ping pong ball on the end of the Nunchucks for tracking. Now currently the ping pong ball tracking does work which is a plus but the Wiimote isn't made to work with the Psmoveservice so the rotation and buttons wouldn't work. That being said somebody could make a program like Psmoveservice that uses the Nunchucks with ping pong balls and uses the Pseyes to track.

Hope that answers your question. Sorry its long.

[g-coded]

@Nerdenstein - I didn't realize the original wiimote only used an accelerometer, I am aware of the camera/leds used for position. It looks like the wii remote plus does have a gyro. Building on what you said, one could make a hybrid wiimote/nunchuk clip to get the buttons needed from the wiimote. Simlrh has an OSVR-Wiimote plugin that could be used along side the bulb tracking(maybe?) I currently have a set of ps move/navi controllers, so I'm not looking to replace them. However if this was a feature before I had bought them I would have tested this option out.

[Nerdenstein]

Well if just tracking a light bulb is added then the PsmoveService can provide the position tracking for the Wiimote for steamvr. But does the OSVR Wiimote plugin give steamvr the necessary rotation and button input data?

[HipsterSloth]

BTW for anyone interesting in the code so far being done on the virtual HMD work, the check-ins can be found over here: https://github.com/HipsterSloth/PSMoveService/commits/virtual_hmd

I haven't tested anything beyond making sure it compiles yet. Still a few more evenings of work here.

After I get the virtual HMD done, I'll do the virtual controller. That'll actually be a bit simpler.

[g-coded]

Thanks HipsterSloth, can't wait to try this out.

@Nerdenstein - I was remembering incorrectly, the OSVR-wiimote may not provide button data(video only shows hand tracking). It looks like a magnetometer is also needed(?) for yaw tracking(for 180* turns). You could probably use(haven't tested)FreePie-OSVR for button input.

[Nerdenstein]

@g-coded Hmmm'

Well if you do find a way please let me know cause I'm interested.

Thanks

[wizardwes]

Would it be possible to implement a controller like this http://www.feelthree.com/2014/09/6dof-handarmfinger-tracking/ Without the leapmotion once this is accomplished? Or would that be too much without adding specific code for the nunchuk. This caught my eye because it would only be about 25 dollars for 2 hands using the nunchuk's gyro and 3 cameras with this service.

[Nerdenstein]

Has there been much progress with coding the virtual controller?

Thanks :)

[HipsterSloth]

@Nerdenstein Still working on the virtual HMD code over in this branch: https://github.com/HipsterSloth/PSMoveService/commits/virtual_hmd.

I just got virtual HMD tracking working in the config tool this evening (tracking a ping pong ball). Still a few bugs to fix. After the psmoveservice work is done then I need to integrate the new virtual HMD API changes over to PSMoveFreePIEBridge.

It's been slow going the last few weeks because I've been crunching pretty hard getting a game ready for E3. I can only afford an hour or two an evening to work on this if I don't get home to late. Fortunately I'm in the home stretch.

[Nerdenstein]

Okay cool. :)

Also does anybody know how to get psmovefreepiebridge working with Steamvr without RiftCat?

I have a Pimax 4k and dont use Rift Cat.

Thanks

[HipsterSloth]

UPDATE: I have a pull request for the virtual HMD work on PSMoveService here: https://github.com/cboulay/PSMoveService/pull/424

I need to finish making a modified version of psmovefreepiebridge that work with the new virtual HMD interface.

[Nerdenstein]

That's great! How do I install this? Is there a pre-built release or do I have to build it my self? If so how do I do that? Or is it none of that and there's just a file I need to add to my PsMoveService folder?

[HipsterSloth]

I just merged the the virtual HMD stuff into the master branch of PSMoveService moments ago. For the moment you would have to get latest and build it yourself. I will have a release out shortly. For the moment the virtual HMD changes in PSMoveService won't be very useful without an updated PSMoveFreePIEBridge, which I'm working on now.

[HipsterSloth]

And if you really want to build PSMoveService from source yourself, you can follow the build instructions here: https://github.com/cboulay/PSMoveService/wiki/Building-from-source

EDIT: pasted wrong build instructions link

[g-coded]

This is awesome news HipsterSloth! You mention there will need to be an update to freepiebridge, I was planning on using the osvr-psms-connector, I'm assuming this doesn't need the freepiebridge update. Also, when trying to use the latest ps move service I was getting errors in steamvr, and switched to psmovesteamvrbridge, will this work on both? It may have been an update to steamvr, or something on my end, so I will test the virtual hmd build with the osvr connector when available.

[Paradise-My-Colors]

Okay wow, great job and thanks to everyone working on the project. I was afraid that you abandoned this issue but seams that I was looking in the wrong direction. you are the best, and I can't wait to see the new release. Goodluck.

[HipsterSloth]

New release of PSMoveService is out with VirtualHMD support: https://github.com/cboulay/PSMoveService/releases/tag/v0.9-alpha8.4.0

See release notes for details about configuring VirtualHMD tracking. I'll try to move this documentation into the Wiki with some pictures in the next few days (depending on how busy this next week is).

I submitted a PR to PSMoveFreePIEBridge to take advantage of this: https://bitbucket.org/hawkinse/psmovefreepiebridge/pull-requests/2/updating-to-support-hmd-client-interface/diff

Until @hawkinse integrates this PR in and puts out a new release it won't make much sense to download the new PSMS release.

@g-coded While osvr-psms-connector doesn't need the free pie bridge update, osvr-psms-connector will need to be updated to use the new hmd client interface to take advantage of virtual HMDs.

@infinitellamas feel free to ping me if you have questions about what updates are needed for this. The aforementioned PSMoveFreePieBridge PR should should serve as a good reference.

[ghost]

Hello, about VirtualHMD tracking, that's means i can use my chines headset PIMAX 4k, with this, to have headtracking ?

We just need : Psmoveservice + PSMoveFreePIEBridge and with that we can directly have head tracking ?

other things :

why we have two version of psmoveservice :

this : https://github.com/cboulay/PSMoveService/releases/tag/v0.9-alpha8.4.0

and this : https://github.com/HipsterSloth/PSMoveSteamVRBridge/releases/tag/v1.1.0

so what version need to use ?

thanks