Use iPhone as MFI controller

[fotiDim]

I need to make an iPhone act as an MFI controller. I want to control a 3rd party game that is running on the same or a different iPhone. Is that possible with this framework given that I cannot modify the game itself?

[robreuss]

Unfortunately no, it’s not possible - it is something that needs to be implemented by the game developer. Hopefully more developers will soon support this because virtual game controler support is now built into Apple’s game controller framework, which mostly makes my framework unnecessary.

On Dec 14, 2021, at 7:37 AM, Foti Dim @.***> wrote:

I need to make an iPhone act as an MFI controller. I want to control a 3rd party game that is running on the same or a different iPhone. Is that possible with this framework given that I cannot modify the game itself?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/robreuss/VirtualGameController/issues/37, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPXCXDQ3HC4AANILTADALUQ4273ANCNFSM5KAY2QZQ.

[fotiDim]

I suppose you are talking about GCVirtualController. Still I don't think it is possible to have a bridge for GCVirtualController. Can an app that implements GCVirtualController run the in the background and forward button presses to the game in the foreground? If not then I guess there is still a need for a 3rd party framework that augments Apple's framework.

[robreuss]

That’s right, GCVirtualController.

If an app implements GCVirtualController but is running in the background, how can the user be providing touches to it if the app is in the background? There would be no UI to touch. Perhaps I’m misunderstanding you. Maybe if you tell me more generally what your goal is I’ll better understand!

On Dec 14, 2021, at 12:14 PM, Foti Dim @.***> wrote:

I suppose you are talking about GCVirtualController. Still I don't think it is possible to have a bridge for GCVirtualController. Can an app that implements GCVirtualController run the in the background and forward button presses to the game in the foreground? If not then I guess there is still a need for a 3rd party framework that augments the Apple's framework.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/robreuss/VirtualGameController/issues/37#issuecomment-993799023, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPXCWDT73PDYBZHDDKJF3UQ53RBANCNFSM5KAY2QZQ.

[fotiDim]

That’s right, GCVirtualController. If an app implements GCVirtualController but is running in the background, how can the user be providing touches to it if the app is in the background? There would be no UI to touch. Perhaps I’m misunderstanding you. Maybe if you tell me more generally what your goal is I’ll better understand! …

Yep, I missed to mention an important detail. I want to have a controller over the network. So the background app would be listening for web requests and translating those into MFI controller actions.

[robreuss]

Interesting. Now you are getting into the domain that VGC might provide assistance with, because I chose a network-based solution rather than Bluetooth.

Are you talking about a gaming scenario? It sounds like not, because your background app receiving the virtual controller input is…in the background, so the user is not seeing any response to the controller input. Am I understanding correctly?

FYI, I really stripped down the VGC framework into something more generic, and taking GCController out of the picture: https://github.com/robreuss/ElementalController

My primary person use case for the framework is controllering robots - that includes not only actually moving the robot around (using either on-screen controllers or a physical game controller), but also sending back telemetry from the robot, such as accelerometer data. I also use the framework to send a low-tech video stream.

On Dec 14, 2021, at 12:36 PM, Foti Dim @.***> wrote:

That’s right, GCVirtualController. If an app implements GCVirtualController but is running in the background, how can the user be providing touches to it if the app is in the background? There would be no UI to touch. Perhaps I’m misunderstanding you. Maybe if you tell me more generally what your goal is I’ll better understand! … <x-msg://5/#> Yep I missed to mention an important detail. I want to have a controller over the network. So the background app would be listening for web requests and translating those into MFI controller actions.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/robreuss/VirtualGameController/issues/37#issuecomment-993817963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPXCXVMMQMBPBYHRE3UNLUQ56A3ANCNFSM5KAY2QZQ.

[fotiDim]

Funny you mention robots 😄. I maintain the SDKs and apps for Romo. I am in fact in the process of switching to WebRTC because the current implementation is quite legacy and I also need to support Android. Not sure if your new lib could serve as a replacement.

But, back to topic... Indeed it's not about a game. The DJI Ronin app can use a game controller to control the PTZ of a camera gimbal but at the same time they offer no API or SDK. Hooking up to the game controller framework would be a work around.

[robreuss]

Wow, what a coincidence! We’re really working on similar engineering problems.

Awesome you brought new life into Romo. I was vaguely aware there was a product out there of that type. My project is very similar insofar as I have onboard iPhones.

I had built a prior generation fixed wheel robot using a Raspberry Pi, wiring in a few IC-based sensors, for example an accelerator and compass, but it was a total pain in the ass dealing with interference from the motors and so forth.

It occurred to me that an iPhone can be thought of as a “bundle” of sensors, and of course other features like the cameras. So with my next vehicle I was all-in on the idea that by putting iPhones onboard, it would open up a huge range of possibilities for control and sensor information. The project is for play - doesn’t have a fixed end-point or product like goal, but rather I’m treating it as a platform for experimentation and learning.

For example, frame a control perspective, I’ve used Apple’s Body Pose Estimation framework (https://developer.apple.com/documentation/vision/detecting_human_body_poses_in_images) so that you can stand in front of the rover and issue commands using your arms, to control motion.

I created a simple framework that rides on top of ElementalController that makes it really easy to stream sensor data from the phone: https://github.com/robreuss/iOSSensorModules

Drive control is done using a Raspberry Pi (ElementalController runs using Swift on Linux) and I’ve written all the drive control software myself. There’s a pair of motor controller boards attached to the RPi, and they control four 300 RPM motors.

The rover has a forward-facing iPhone on a tilt/pan mechanism that I built, and that’s controlled by a separate RPi Zero. Another phone is positioned on the backend using a simple bracket, and it's used for telemetry and orientation, since it’s in a fixed and stable position.

Here’s the kit I used for building the tilt/pan:

https://www.servocity.com/spt200-pan-tilt-kit/

I’ve written a pretty substantial iPad-based control app. All communication between the iPad, the main RPi, the onboard phones, and the tilt/pan mechanism is handled with my framework, over WiFi. It’s ridiculously easy with the framework to do all of the telemetry and control data. I do video streams from both cameras by sending images through the framework - works surprisingly well given it’s a pretty low tech approach to video streaming.

Per your question about WebRTC, Android and my framework. It never occurred to me to try to run my framework under Android. I just did a quick search and there is a project brining Swift to Android, which would be the primarily requirement to get it working:

https://www.scade.io

I’m totally going to look into that! It would hugely increase the usefulness of ElementalController if it could run on Android.

Would love to hear more about your goals with DJI Ronin. If I hear you correctly, the DJI Ronin app running on iOS can interface to a wireless hardware game controller for manipulating the PTZ. Are you looking to hijack that interface in order to control the Ronin in software? If I’m understanding that correctly, the VGC framework wouldn’t be able to solve that problem - it can generally not deal with use cases where you’re trying to provide virtual game controller function to a third-party app.

Picture of my rover attached. Old shoe included for scale. :)

On Dec 15, 2021, at 6:38 AM, Foti Dim @.***> wrote:

Funny you mention robots 😄. I maintain the SDKs and apps for Romo https://github.com/navideck/romo. I am in fact in the process of switching to WebRTC because the current implementation is quite legacy and I also need to support Android. Not sure if your new lib could serve as a replacement.

But, back to topic... Indeed it's not about a game. The DJI Ronin https://apps.apple.com/us/app/dji-ronin/id1282848185 app can use a game controller to control the PTZ of a camera gimbal but at the same time they offer no API or SDK. Hooking up to the game controller framework would be a work around.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/robreuss/VirtualGameController/issues/37#issuecomment-994707660, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPXCXWACSTZFD4FIBUJ2LURB43HANCNFSM5KAY2QZQ.

[fotiDim]

@robreuss great info there! You forgot to attach the image 😁 I am also thinking about the next version of Romo. Apple is likely to drop the lightning port at some point (soonish?) and that would call for new hardware. I also want to include Android support as it will open the project up to a whole new community of makers. For cross platform support I have been investing into Flutter lately and it seems I have a clear path forward. I could suggest making your robot compatible with the Romo app but it is all ObjC and since a rewrite is hanging above me I wouldn't dare so.

There is also openbot from Intel which is Android only. Since, it is a long discussion. I have made a discord server so we can exchange ideas. You can join here.

[fotiDim]

Would love to hear more about your goals with DJI Ronin. If I hear you correctly, the DJI Ronin app running on iOS can interface to a wireless hardware game controller for manipulating the PTZ. Are you looking to hijack that interface in order to control the Ronin in software? If I’m understanding that correctly, the VGC framework wouldn’t be able to solve that problem - it can generally not deal with use cases where you’re trying to provide virtual game controller function to a third-party app.

Yes the DJI Ronin app supports MFI controllers. It seems I need a way to emulate MFI controller presses on OS level and that is not possible without jailbreaking. Tweaks like nControl seem to be doing something similar. I might investigate into making my own tweak, aka web server -> MFI controller commands.

Another very similar use case that comes to mind is having a second player join you over the network. Have you ever used Parsec? They are doing the same thing for PC, emulating a local Xbox 360 controller for the remote player.

[lazlox]

I am looking for the same or similar functionality:

1. use iPhone as racing wheel reading gyroscope data via custom app
2. connect iPhone app to Mac helper app via the MultipeerConnectivity framework
3. Mac helper app running in background would act as virtual controller translating the events from the iPhone to standard HID controller events to be used in a 3rd party app or OS level.

Wondering if this framework could help with the above by using it in the Mac helper app.

[robreuss]

I think the main problem is #3 in your sequence.

I assume you’re talking about delivering a stream of accelerometer data to a 3rd party app running on a Mac or iPhone.

Game developers use Apple’s Game Controller framework (https://developer.apple.com/documentation/gamecontroller/gccontroller)within their app to support a wide range of hardware controllers. The flow of data goes:

1) from the hardware game controller 2) to an Apple device (iOS-based or Mac) 3) into Apple’s game controller framework 4) and finally it is delivered to the game app and the developer has the game respond to it

It sounds like you want to use a second iPhone as a controller and send that data so that it reaches a third-party game developer's app. The problem is that the sequence I described is “closed” - there is no point at which you can start pushing controller data into that sequence and have it be received by the third-party app.

One way to do it, theoretically, would be to have your iPhone controller device (steering wheel) mimic the behavior of a standard hardware MFi game controller by having it appear as a bluetooth device. I investigated this possibility long ago when I created the framework and as I recall there was prtty tight security preventing mimicking a commercial MFi game controller (I think it may be certificate-based). As a result, I gave up on creating a virtual controller framework that could work with ANY third-party game and instead focused on a framework that would help game developers add virtual game controller functionality to their own game.

Note that the Mac helper app doesn’t help this scenario at all, unless I’m missing something.

Also note Apple recently came out with it’s own virtual (screen-based) controller - https://developer.apple.com/documentation/gamecontroller/gcvirtualcontroller - but that’s a different animal because it only present the virtual controller on the device where the game is being played - it does not support using a second device as a game controller.

I hope that’s helpful!

On Dec 29, 2021, at 4:24 PM, lazlox @.***> wrote:

I am looking for the same or similar functionality:

use iPhone as racing wheel reading gyroscope data via custom app connect iPhone to Mac helper app via the MultipeerConnectivity framework Mac helper app running in background would act as virtual controller translating the events from the iPhone to standard HID controller events to be used in a 3rd party app or OS level. Wondering if this framework could help with the above by using it in the Mac helper app.

— Reply to this email directly, view it on GitHub https://github.com/robreuss/VirtualGameController/issues/37#issuecomment-1002781445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEPXCWFXS36RZCBMH3OQLTUTN4BJANCNFSM5KAY2QZQ. You are receiving this because you were mentioned.