Add sensor (gyro/accelerometer) support.

[agraef]

Ok, I gave this a go myself now. The option is enabled with -r or --sensors, which will also print the available sensor names and their update rates as SDL reports them. (I'd rather have used -s for the short option, but that's already in use for the --sleep option.)

This will generate a lot of messages under the /joyosc/devices/<device>/sensor prefix. Arguments are the sensor name (usually gyro or accelerometer) and three float values (x, y, z) as SDL reports them (cf. https://wiki.libsdl.org/SDL2/SDL_SensorType). E.g., here are two typical messages generated from an 8bitdo Pro 2 controller in Switch mode:

/joyosc/devices/gc0/sensor accelerometer 0.06733767 10.047412 0.8047845
/joyosc/devices/gc0/sensor gyro -0.0024435392 0.0012216782 0.001220674

Right now, there's no way to ignore or remap individual sensors, like you can do with the buttons and axes. I'm not sure whether this would make any sense in this context (and in any case I wouldn't know how to implement it). But at least the option now gives a way to read those sensor values and convert them to OSC messages.

This closes #19.

Example: For good measure, here is a little test patch which can be used to show the gyro data generated by joyosc -r: controller-test.zip. It also includes a TouchOSC template which can be used to emulate a real controller with gyro support, and some information about how gyro data is to be interpreted and the differences in the axis configuration between gamepads and smartphones.

NOTE: Just to be clear (and for future reference), this PR provides the raw sensor readouts, as SDL provides them, nothing more. nothing less. You still need to figure out how to process that data, which is not a trivial problem, see the gyrowiki by Jibb Smart (of Fortnite fame) for possible avenues. In the future we might want to consider adding Jibb Smart's GamepadMotionHelpers to process the sensor data to something more readily usable (as is done, e.g., in Jibb Smart's well-known JoyShockMapper program), but this is not included in this PR.

[agraef]

Looks like the Ubuntu 22.04 runner chokes on the extra (left/right) gyro and accelerometer sensors, so I'll have to comment these out. Stay tuned.

[agraef]

Got it right the second time. :)

[agraef]

Not sure whether there's an easy way to check the availability of the SDL_SENSOR_ACCEL_L et al values. Of course, we could add an autoconf check for them, but I'm not sure whether it's worth the extra work, because it seems that these are only there for the benefit of the Nintendo Joy-Con and Wii nunchuk controllers.

[agraef]

Ok, so I added the configure check for SDL_SENSOR_ACCEL_L et al now, to be on the safe side. There might be other split controllers which use the same kind of left/right sensor arrangement (mobile controllers maybe?), so it makes sense to have those covered if the SDL version supports it.

[agraef]

CI finished without any hitches, so this should be ready to go in now.

[danomatika]

Good work. I tried the API a little yesterday but I don't have a controller here that exposes the sensors to SDL on macOS. I remember when I first used a PS3 Sixaxis controller on Linux back in 2011 or so and it exposed the accel as a set of joystick axes. This is actually why I added the ignore mechanism... to silence this as I didn't need it at the time.

The move to SDL2 brought the controller type but by then I was on macOS so I didn't see that output and forgot about it. I agree, having this is a no-brainer ,especially giving us some easy options for experimental interactivity without having to build something custom or buy something expensive.

[agraef]

Fixed some glitches in the autoconf SDL_SENSOR_ACCEL_L detection on Apple Silicon Mac, and changed the sensor names as you suggested.

I agree, having this is a no-brainer ,especially giving us some easy options for experimental interactivity without having to build something custom or buy something expensive.

I'm sure the students will love this. :) Unfortunately, most controllers with that functionality are still rather expensive. Not as expensive as they used to be, thanks to 8bitdo, GameSir, and the like, but still. The cheapest that I could find are the 8bitdo Pro 2 and SN30 Pro, and these are somewhere around 45€ at present here in Germany. (~The GameSir Nova Lite is around 30€, but I haven't had the opportunity to test it yet.~ Scratch that, the Nova Lite doesn't have gyros.)

Also, I've had no luck running these in Switch mode on the Mac, only D input mode seems to be supported there. They work fine in Switch mode on Linux and Windows, though.

[danomatika]

Do the sensors appear as axes when opening devices as joysticks only? Also, if you do have a list of controllers which provide sensors, we can add this to the readme.

[agraef]

Do the sensors appear as axes when opening devices as joysticks only?

No, I only added the requisite code (pilfered from the SDL2 testgamecontroller.c example) to GameController.cpp. The SDL2 testjoystick.c example doesn't have gyro support, so I'm not sure whether a gyro API even exists at that level.

Also, if you do have a list of controllers which provide sensors, we can add this to the readme.

This is a fast-moving market, with all the Chinese manufacturers churning out devices like there's no tomorrow. The product lineups are also very confusing, with similarly named products which differ in features, some with gyros, some without. I don't think that there's a definite list anywhere. Generally, you'd want to look for Nintendo Switch-compatible controllers which have some way of switching to Switch mode on PC. (But you really have to look at the specs, not all Switch-compatible controllers have gyros.) Neither Microsoft's D and X input protocols support gyros, only the PlayStation and Switch controllers do, as well as compatibles.

From the top of my head, I know that the 8bitdo "Pro 2" and "Ultimate Bluetooth & 2.4g" controllers have gyros, as well as the GameSir Nova (but not the Nova Lite) and the Gulikit KK3 (Pro and Max). And of course the Nintendo Switch Pro controller (not sure whether that also works on PC, though) and Sony's DualShock and DualSense controller (these are supported on Linux at least).

[agraef]

Add to that the GameSir Cyclone and Cyclone Pro, these both seem to have gyros as well, and reportedly are better quality controllers than the Nova. ~I think that I might get one of these so that I can test it.~ (I decided against that because AFAICT Linux support for those devices seems to be a bit sketchy. Also, the GameSir Cyclone 2 is about to come out, so I'm probably going to get that one instead if it's as good as some early reviews indicate.)

I'd also be interested in the DualSense with its touchpad, but they are a bit pricey for what they are IMHO, and I'm weary of the non-hall effect sticks. I had too many of these potentiometer sticks fall victim to stick drift.

The cheapest among all of these seems to be the GameSir Nova right now, closely followed by the 8bitdo Pro 2 (which reportedly has much better build quality than the Nova, and has that nice retro look and horizontal stick placement).

[danomatika]

There is a lot of new options the past years. I have been using Steel Series controllers, originally for the iOS support as I generally perform using an iPhone running PdParty. They are pricey but good quality (and research budget) and I am happy to get them to work with joyosc again.

As with most of these, they can be cut down into something more interesting:

[agraef]

Interesting, I didn't know about these. (Which isn't saying much, given the sea of game controllers available these days.) But, yes, they're still selling some models (the Nimbus+ for iOS and the Stratus+ for Android/PC) on Amazon.

Another possibility is to just use TouchOSC on Android or iOS to get movement data and convert it to OSC. I think I'm going to add that to my TouchOSC gamepad template in the joyosc-pd package, and have it generate the same messages as joyosc does with this PR.

[agraef]

Ok, I should maybe still update the documentation in the README before you merge this. Stay tuned.

[agraef]

Done. Unless you still have some other suggestions, this should be ready to go in now.

[danomatika]

I have major refactors in the develop branch, but I will handle that on my end. I will try merging this today.

One nice thing I am remembering now about joyosc: in joystick mode, many of these controllers expose their pressure sensitive buttons as additional axes. I ignored those in the past but I think I will try to create something with this specifically for the next project.

[agraef]

Ok, thanks. Then I'll wait until you're done with this, because I'd still like to look into rumble. Of course, you don't really need this for sound design stuff, but I think that short vibrations are a great way to signal status changes without having to look at the Pd patch on the screen. Like, e.g., providing haptic feedback after pressing a button to engage motion tracking, or to change a filter parameter.

[agraef]

Rebased on current master.

Dan, I have another branch which implements touchpad support. I'll submit this as a separate PR, but it will be based on this PR to prevent merge conflicts.

I'm not sure I want to proceed much beyond this, because I see that you've done a lot of refactoring on your develop branch that will likely cause a plethora of merge conflicts when you try to incorporate my PRs. :) But I do need/want the touchpad functionality, so I'll submit that as it stands now.

[agraef]

Dan, I just added controller-test.zip with a test patch and some documentation to the description. Feel free to use that for the joyosc docs in any way that you see fit.

[danomatika]

For processing accel events, there are a couple patches for PdParty which do raw accel -> pitch & roll, etc:

https://github.com/danomatika/PdParty/tree/master/doc/composerpack/cookbook

If we add sensor processing, I would have an additional sending point ala "/motion accel" etc. This is the approach I took with PdParty to keep compatibility with existing projects.

[agraef]

For processing accel events, there are a couple patches for PdParty which do raw accel -> pitch & roll, etc:

Interesting. I'll have a look.

I already started turning Jibb Smart's algorithm into an external, though. That method was developed for view control, so it's not clear to me whether it's applicable to controlling sound, but since it works so well in games it's tempting to give it a go anyway and see how well it fares. There's an interesting approach in that algorithm to integrate both acceleration and gyro data in order to estimate the current gravity vector and calculate pitch and yaw/roll relative to that. They call that "sensor fusion". It also provides a way to use different coordinate systems (what they call local, world, and player space) and calibrate the motion sensors, so there are ample configuration options as well.

As Jibb has shoved this entire algorithm into a single C++ header file, it shouldn't be hard to just wrap that up in a little Pd external which takes both acceleration and gyro data and turns it into x, y outputs depending on the chosen coordinate system.

If we add sensor processing, I would have an additional sending point ala "/motion accel" etc.

I knew there had to be a better symbol for that data. :facepalm: I can still do that change. So I'll just rename sensors to motion, and the --sensors option to --motion, would that suit you? As always, coming up with a sensible short option is a challenge, though, as -m is already short for --multicast. How about -v (for "velocity")?

I can send you another PR for that, stay tuned.

[agraef]

I already started turning Jibb Smart's algorithm into an external, though.

Works like a charm, see https://github.com/agraef/gpmotion. Also available as a Deken package.

There is a lot of new options the past years. I have been using Steel Series controllers

NB: If you're looking for some cheap and cheerful Chinese DS4 clones for DIY purposes, these controllers from Chereeki are on sale on Amazon for 23€ right now: https://www.amazon.de/dp/B0CBR9BMHY. I already bought four of them for my sound design course, so that I can hand them out to the students. They're surprisingly good quality controllers with all the functionality of the original DS4 plus assignable rear paddles.

[danomatika]

This looks cool, thanks. I am thinking about integrating the processing but was distracted with libpd the last week. I will make a roadmap issue for joyosc 1.0 and invite you.

[agraef]

That would be useful to have in joyosc for sure, and it's probably not too difficult to port the algorithm. But the advantage of a separate external is that you can also hook it up to a virtual gamepad running in TouchOSC on a tablet.