Revamped feedback, less Python, preparing kernel events

[medape]

I've done a lot of different changes in ~a single commit~ two commits now. I expect to tweak and maybe undo a couple of them, but I believe this new version is improved in most if not all areas.

Changelog

Support for touch pressure

I had previously written some code to control the pedals by finger pressure (to make them more easy to control and realistic) but in the old version, it was commented and the pedals translated the finger position instead.

This version tries to read the finger pressure as soon as you select a controller, and if it suspects your screen is capable of detecting it (that is ev.targetTouches[0].force is between 0 or 1), shows a screen alliowing you to calibrate finger pressure.

The calibration is pretty simple: you just press the screen softly and as hardly as you feel comfortable with, and Yoke records only the lowest and highest pressure. These bounds are used to normalize the pedal response.

If your screen doesn't support this, or you decide to skip the calibration (yes, you can skip it), Yoke pedals will revert to their old behaviour.

NEW: analog buttons

Analog buttons (similar to car pedals) added in commit c67e17c. They're buttons which report 0 on rest, and can report any state between 0 and 1.

In compatibility mode, the state depends on distance to the centre, measured on the longer axis. E.g. in a wide button, the left and right edges report a 0, and the very centre reports a 1. Intermediate points in the X axis output intermediate values. In a tall button, only the Y axis is checked.

In touch-force-detecting mode, analog buttons behave exactly like pedals, but with a different style and kernel event assignation.

Bumps for the joystick (NEW: improved vibration everywhere)

This is the most experimental change.

Joysticks now have a (rather ugly) dark bump grid. When your finger slides over this grid, it vibrates just a little to help you guide the joystick by touch. This grid now divides the joystick in 8 sectors, to (hopefully) help you find easily the cardinal directions.

Joysticks (every control except analog buttons, as of commit c67e17c) also vibrate continously while saturated now. I used requestAnimationFrame for this, I'd love to know if there is a better way to vibrate in pulses that doesn't depend on async loops.

Tweaked all the vibration times. Saturated controls now vibrate in bursts on 10 ms, which seems to be noticeable but not uncomfortably so. Fine-tuning may still be necessary.

Fixed a bug in Android WebView (and, by extension, in Yoke): vibration on TouchStart was delayed by 200-300 ms even when configuring touch-action. Preventing the default action on the touchstart event corrected this delay and made the WebView fall in line with Firefox' behaviour.

I'd also love your opinion on how to improve this grid and the vibration times, or if I should remove them entirely.

Buttons now visibly click

Buttons (and pedals) now darken when pressed.

A decent artist will probably code a better way to depict pressed buttons at some point in the future.

NEW: Simplified the code for reordering controls, and added support for dummy axes

(Commit c67e17c.) I simplified the rules for deciding the order in which controls are reported: the exact order doesn't really matter as long as every “analog” axis comes before every button.

If your layout contains than 4 analog axes, it creates dummy axes instead of warning you, as it already did with buttons.

Of course, these dummy controls will go when the handshake at #23 is properly implemented, but this will ensure a complete working example for the moment. We could even bump the number of axes to 8 and focus on creating layouts without touching much of the Python.

Programmed a part of kernel events

The JavaScript still doesn't send the kernel events to the Yoke app as the python code doesn't know how to interpret them, but it can assign a kernel event to each and every control.

A string with the kernel event codes for every attached control, split by commas, is displayed in the debug ticker upon loading the gamepad.

I'm aware some of the kernel events are not correct, and the most unusual ones may cause collisions. Can we “create” new events just by editing events.py or will that bring further problems?

Deleted Python obsolete code

I've removed entirely the --type argument from yoke.py, as this job is controlled from the webview now. It also reads the events directly from yoke.GAMEPAD_EVENTS.

This way, vJoy doesn't attach buttons that will never be used, and the order of buttons in axes is much more predictably.

I've also reworked the list of events in service.py, to add buttons and to fix a bug: BTN_GAMEPAD is supposed to be an alias for BTN_SOUTH, but both events were registered on vJoy. This added a delay when pressing this button on the WebView.

[pzmarzly]

I used requestAnimationFrame for this, I'd love to know if there is a better way to vibrate in pulses that doesn't depend on async loops.

Hmm, there are phones with 120 or 144 FPS displays. setTimeout seems to be highly precise in modern Chrome versions, so maybe we could use that (though Yoke uses built-in webview at the moment afaik, and I'm not sure what the accuracy of Android 4.4 webview is)

[pzmarzly]

All in all, awesome job. Huge thanks for your time and dedication!

[medape]

Hmm, there are phones with 120 or 144 FPS displays. setTimeout seems to be highly precise in modern Chrome versions, so maybe we could use that (though Yoke uses built-in webview at the moment afaik, and I'm not sure what the accuracy of Android 4.4 webview is)

The end result is pretty much the same, but setTimeout (or setInterval) require a refresh rate. And they're infamous for being imprecise.

requestAnimationFrame is supposed to throttle itself down if there are speed or battery problems, so I'd rather let that work to the webview than try to find the magic refresh rate and risking irregular vibration patterns.

(They're already very irregular as they are, but I will be tweaking them in my next commits.)

[medape]

Thank you very much for your help with everything. I have a question about, er, “protocol”.

These days I managed to fix an important bug in the HTML side (Android WebView delays the vibration on TouchStart unless you explicitly call Event.preventDefault()) and improve other things. I changed no files outside of the assets folder.

Is there any accepted and clear way to upload and send over my new changes, or would you recommend me to wait until this PR is closed?

[pzmarzly]

I'd be fine if you either pushed these changes to a new branch and put a link here, or added these changes here (into your master, but git pull it first).

If you desperately want to train your git skills, you could try making a new branch with git checkout -b branch-name and using git rebase -i master/origin~4 to clean it up, to make a separate PR. Or you could use git reset master/origin~4 (adjust the number) to remove commits without removing file changes.

But we will be fine with whatever method you use.

[medape]

Thank you for your pointers. I did some tests with throwaway branches, but I finally decided I'd rather push my new changes here for now, because a lot of them build on what's already commited on this PR.

I'll also edit the changelog on this PR for future reference.

[pzmarzly]

Your changes in Python code made me notice that sometimes UDP packets were sometimes getting truncated, resulting in silent data loss when using code on master, or crash in code on this PR branch. Hopefully 4 digits after a dot should be a good balance between precision and size (though maybe we should make a binary protocol one day :smile: ).

[pzmarzly]

Merging this, I will continue making changes on master.

[pzmarzly]

Though, I have a question - is force touch actually useful? My current phone doesn't support it, my previous did, but it worked pretty bad - yet I've never tried using it with a gamepad app.

[medape]

(though maybe we should make a binary protocol one day :smile: ).

Wow, yes, it would be MUCH more efficient.

Now that you mention it: @pzmarzly, @rmst, do you know something about using WebSockets in Android WebView? Or have any experience with XHR once the page is loaded?

I ask because I was thinking it would be useful for the web page to communicate directly with the computer, without using Yoke.update_vals(). That way, the app will be useful to search for connections, but it won't be as necessary over wi-fi. And theoretically it should reduce some latency, too.

Merging this, I will continue making changes on master.

Now you mention it, I just found a bug in base.js. Sorry.

~I'll correct it quickly~, I don't need to change any other files, but it's just to let you know. So we won't have commit conflicts.

EDIT: Bug corrected, see PR #31.

Though, I have a question - is force touch actually useful? My current phone doesn't support it, my previous did, but it worked pretty bad - yet I've never tried using it with a gamepad app.

It should be... I only did quick tests with jstest-gtk, though. I didn't actually try them in a driving game.

When I first started experimenting with force-touch, my phone reported between 0.2 and 0.5 if I used only my fingertip. To approach 0.8, I had to mash my whole fingerprint against the screen, and there was no way to ever reach 0.9 or higher. Sure, It is bad.

But, as long as your screen is not completely broken, and keeps reporting higher values for higher pressure, there should be a way to calibrate that. Even if you have to use gamma curves.

I suppose we won't really know how well it works until users start reporting issues with them. As it's completely optional, it doesn't really do any harm.

[pzmarzly]

update_vals should be faster as its UDP (no ACK messages, stateless connection). AFAIK it's possible to make UDP connection in JS using WebRTC, but as long as we rely on built-in WebView and support Android 4.4, we won't be able to use it.

Speaking of protocol, it turned out that we are sending 3 floats from Android code, then floats from JS code. So there's a lot of room for improvement. In this PR I only reduced the precision for Joysticks, I already have commits that reduce it for all elements (will push it later).

Thanks for another PR, will look at it later.

[medape]

update_vals should be faster as its UDP (no ACK messages, stateless connection). AFAIK it's possible to make UDP connection in JS using WebRTC, but as long as we rely on built-in WebView and support Android 4.4, we won't be able to use it.

Oh, okay. Not worth it, then.

Speaking of protocol, it turned out that we are sending 3 floats from Android code, then floats from JS code. So there's a lot of room for improvement. In this PR I only reduced the precision for Joysticks, I already have commits that reduce it for all elements (will push it later).

I remember, @rmst talked about those; they're supposed to be the orientation of the phone. A leftover from when the layout was fixed and bundled with the app, I think.

There's a lot of potential, but also a lot of cleanup to do. There is still a lot of old code from before the WebView: the APK still bundles rmst's first layout and your first layout, and the Python code is still hardcoded for six axes. It does gravity calculations for the first two floats, it renormalizes the other four floats to the interval (-1, 1), and then maps everything to (0, 255)... it's difficult to follow, but these calculations can be done just as quick from the JS side, and apparently there's a different layout hardcoded for Windows.

Yoke's events.py also seems to be a copy from uinput's ev.py. Is there a reason for this?

[pzmarzly]

Yoke's events.py also seems to be a copy from uinput's ev.py. Is there a reason for this?

Probably Windows - EVENTS are modified here, plus python-uinput` is Linux-only dependency.

I agree there's a lot of room for improvements, though right now I'll try to prioritize creating a stable release (as anything would be better than what's currently in Play Store or F-Droid).