Increasing compatibility and functionality of the JavaScript controller (new PR)

[medape]

This is related to issues #2 and #17. I closed my own PR and attempted it a second time because I could not figure out how to merge the changes from the first one.

I own an old low-end mobile, running Android Marshmallow, on which the controller wouldn't display. Its manufacturer has given up on publishing updates.

So while trying to get it to work and adding more functions, I made big changes on @pzmarzly 's code:

• Downgraded it from ECMAScript 6 to ECMAScript 5. I could not get it to display anything in the WebView otherwise. (Oddly, Firefox displayed it correctly, but not Android WebView even after upgrading it.) The code is more verbose now, but it should work the same way and be compatible with more devices.
• base.js now detects if Android WebView is too old to support CSS grids; if so, it displays a short note with a link, prompting the user to upgrade it.
• It also detects if grid-template-areas contains no controls (or is not there at all ― sometimes my mobile refused to download user.css after numerous connections).
• I moved the right joystick around, made the buttons bigger, and coloured them in.
• base.js only binds event handlers for the controls in grid-template-areas, and also doesn't bind ontouch for digital buttons (ontouchstart and ontouchend are enough).
• All controls vibrate just a little when you touch or release them, to have some sort of tactile feedback. This vibration can be changed or disabled changing two variables in base.js.
• Joysticks also vibrate when you try to push them past them limits. I think there is a lot of room for improvement here, but I'm not sure on what direction to take. Maybe vibrating every time you move it between quadrants?
• Tried to add a roll-pitch controller only to realize my mobile doesn't support it. I left the code in in case someone can test and improve it, but it's not active by default ― as far as I know, you need to replace a joystick with it.

I'm happy to keep adding new controls, but I need some direction, and the mapping to virtual controls is weird. yoke.py still has four modes of operation, and the actual order of axes and buttons seems to depend on yoke.py, service.py and app-debug.apk at the same time. It would be great if there was a way to unify them all (and only having to deal with the HTML part), but it's a very complex script and I'm afraid I'll break the engine or something.

Currently, if you only change the contents of the webserver, you can only use the first four axes out of 6, and the last four buttons out of 10 or 11.

[pzmarzly]

There is a hidden Android setting that configures which webview implementation to use system-wide. AFAIK phone manufacturers were forced to ship Google's "Android WebView" binary, but not necessarily use it - this may be why updating it didn't change anything on your device.

It may also be a problem when end user's device WebView is unupgradeable. I guess we will need to see if we run into such problems, but if we do, there is GeckoView HTML engine and Chrome Custom Tabs (which require user to have Chrome installed, which may not be the case, but as a fallback it should be fine)

[rmst]

@medape thank you!! I currently can't test it, but trust you and @pzmarzly with the design decisions.

The phone's orientation is actually already being transmitted (the code for that is in Java). Outsourcing that to html/javascript would be nice but only if it works equally well.

I totally agree that we should be able to control the axis/key mappings from html/javascript. Keys and axis have short names, so it would probably easiest to transmit the names in every UDP message. This would require a change in the python code but not necessarily in the app. You can see in https://github.com/rmst/yoke-android/blob/bc198e5ee298954661008ebe24b05bc4f28cc2ef/app/src/main/java/com/simonramstedt/yoke/YokeActivity.java#L317 that we forward the string that we get from the webapp (concatenated with the phone orientation values).

[medape]

@pzmarzly Wow, I didn't know there could be several webviews.

The weird thing for me is that the update did change something: it allowed me to use the CSS grid-areas. (Thanks a lot for that, by the way! I've learnt lots of stuff from your code.) That's how I thought about detecting support for grid-areas in the first place. But it still stopped working when it found a class definition or an arrow function. It looked like it understood const just fine, but I decided to play it safe, just in case.

It seems like WebView is supposed to be compatible with those features, and supposed to be always upgradable from the Play Store, but there are random problems in random devices. (The question is old, but ECMAScript 6 is supposed to be older.) I downloaded WebView from here as I'm not using the Google Framework. That shouldn't affect the result, but who knows anymore.

Good to know there are alternatives, too.

@rmst I'll keep looking into the sensors issue. I just verified that my mobile has sensors for orientation and Android can read them, so it's probably a browser/WebView problem. I would still rely on the app rather than in the JavaScript for the moment.

About the centralizing of the controls, I think there's a better way, but I never tried programming in Android, or complex Python, so stop me if I'm wrong.

We could transmit the names only once, in a handshake, after the app displays the controls. Using @pzmarzly​'s code as a base, we can load just the controls in the CSS file, instead of sending four axes and sixteen buttons in every case.

It's easy for users to use a letter and a number for controls (j1 for joystick 1, b1 for button, p2 for pedal 2, du for d-pad U). If they use those mnemonics in the CSS file, it's easy to decide which keys to use (ABS_HAT0X & ABS_HAT0Y, BTN_A or something, ABS_BRAKE, BTN_DPAD_UP). Of course, it'll also be easy to add a couple more default CSSs, to restore all the functions of the old Yoke app.

Then, and only then, the computer creates a virtual joystick, registering those exact events. From then on, the mobile can send just numbers as it was doing, with the confidence that the computer will know how to interpret them. The virtual joystick would still be removed when closing the webserver. Maybe you could even connect several controllers to the same server, if the computer can tell them apart by IP or something.

[rmst]

Yes, sending the controller config (buttons, axes, etc.) at the start of the connection would be the right thing to do. Currently there is no handshake, thought. It's just loading the webpage and then starting to send UDP messages. The right way to send the config would probably be to send a json message to the webserver.

As for the event identifiers, I think we should stick the official linux event codes (https://www.kernel.org/doc/html/v4.14/input/gamepad.html) and potentially allow for aliases such as "j1" in the grid definition.

Multiple controllers should already be possible using multiple yoke server instances. For slightly improved user experience we could have the webserver spawn a new udp-receiving process for every client that connects (after the controller config has been received).