Key chatter

[linduxed]

This behaviour is present on firmware version 8.2.0. It could be that this is fixed by a higher version of the firmware (I'm guessing that c38114648a5c57cd7de2f3135e0fc05c7f82df98 could maybe help, for instance), but I'm currently unable to update, as a result of https://github.com/UltimateHackingKeyboard/agent/issues/691.

I'm experiencing frequent but seemingly irregular occurrences of double input from pressing certain keys. This causes misspellings such as "dotted" becoming "dottetd".

The key that seems to be doing this the most is (on QWERTY) the T key. Other keys that exhibit this behaviour are I, S, F, and Space (right half, next to Fn).

There are probably other keys that I've witnessed producing a double input, but I can't remember them off the top of my head.

If one was to focus on the most frequently misbehaving key: I'm guessing that my habits of how I press keys, combined with where T is positioned triggers some kind of edge case.

With the T key in particular I'm also noticing the occasional issue of the input being skipped (no T written), but this could be unrelated and I have to type more to confirm.

[mondalaci]

@linduxed Thank you for reporting this issue! I've also experienced some chatter lately, and we received about 2 reports so far from our backers. I'll ping the reporters to share their experiences here so that it'll be easier for me to fix it. I have some ideas how to track this down.

[linduxed]

Updated to 8.2.5. Issue still present.

[kour1er]

@linduxed out of curiosity, what OS are you using? I'm wondering if there are difference between OS and UHK with this. macOS here and seeing the issue

[linduxed]

@kour1er Arch Linux.

[kour1er]

Interesting - so not limited to a specific OS. The plot thickens :)

[kareltucek]

@kour1er This clearly happens in case of bad connection of switches (either inside of the switch or somewhere around)(in my case this happens with loosely connected unsoldered switches)*, yet there may probably be some other hardware-or-firmware-design-related causes.

I am quite curious if this is related to specific switch type/brand.

Simple testing procedure - get a key which produces a character as primay action and has some secondary action. Then hold, change pressure, push the key into sides etc.. Every skip will produce a keystroke...

*(I am of course not complaining about that :-) .)

[kour1er]

@kareltucek interesting. So you think that this is a fundamental hardware issue? It's so weird because when I first got my UHK I had zero chatter issues. It's only over the last month and a half that I've been seeing it.

[kareltucek]

Well, from my obervations, I am confident that one possible cause is this hardware one - minor malfunctions of the mechanical switches which are detected by hypersensitive firmware. I do not believe that the problem is between firmware and PC. I believe that it is in the way firmware interacts with switches. I would not call it fundamental hardware issue - I would rather say that firmware's assumptions about hardware's behaviour are a tiny bit off. I believe that it will be possible to fix in firmware.

However, this is my personal opinion based on my observations. Moreover, these obervations are based on having done quite barbarous modification to my hardware and therefore may be totally irrelevant. I am sure Laci will be able to provide much more relevant explanation.

[kour1er]

@kareltucek what a great way of explaining it - thank you

And I am now so curious about your barbarous modifications!

[linduxed]

@kour1er It's so weird because when I first got my UHK I had zero chatter issues.

Strange, I also recall this being a somewhat new issue for me. I think that it's only been going on for the past two weeks or so. IIRC I got my UHK a month and a half ago.

[mondalaci]

The characteristics of the of switches may change over time, but the firmware should deal with such issues. I plan to get to the bottom of this, but I have to deal with a lot nowadays. I'll follow up.

[kour1er]

@mondalaci I can't imagine how many things you're juggling at the moment! Thanks for following up

[mondalaci]

Our intern, @eltang has taken a quick look at the firmware debouncer which was followed by this conversation:

Eric: So I think the issue might be that debouncing is only happening on press, and not release. This seems to be what's happening to the issue reporter, so if you agree I can go ahead and make a fix. Me: Why should debouncing happen on release? Eric: Because bouncing can happen on release as well. Did you not know that? Me: Nope :|

My cluelessness is truly limitless!

Then he went ahead to fix the debouncer accordingly. He also changed the debounce interval from 100ms to the suggested 5ms of MX switches. I was gradually increasing this value from release to release in the hope to combat this issue without realizing the root cause.

Please give a try to the newly released firmware 8.3.2 and let us know how it works for you.

[linduxed]

I just installed 8.3.2 and the issue has worsened significantly. Not only is doubling of characters now more common on the keys where I previously experienced it, on T (my most problematic key) now triple input now happening fairly often.

Additionally, no input from the key (which I was previously unsure of whether it's related) is also more common.

I've been trying to press the problematic keys for a while now to figure out a pattern (slow press, fast, hard, shallow, deep, fast repetitions, single stroke, etc.) and I can't figure something out.

[mondalaci]

@linduxed Sorry to hear, and thanks for the quick feedback! Just uploaded a number of firmware images featuring different debouncing timeouts. Would you please give them a shot?

[linduxed]

After trying 10ms, 50ms and 100ms, my conclusion is that 100ms is the one with the least chatter: out of the available ones, it works the best.

[kour1er]

Well just tried to flash one of the new images and the flash failed and the LED on the left half of the UHK no longer lights up. Bricked? Do I need to take it apart and do the manual reset thing again? (Right side is still functional)

[mondalaci]

@kour1er Just try to reflash a firmware normally and Agent should instruct you to reconnect the left half possibly multiple times to unbrick it.

[kour1er]

@mondalaci Ah - you weren't kidding about multiple times! Back to life now.

So I would agree with @linduxed that the 100ms has the least chatter. The chatter is still there, but the 100ms is much better than the 10ms version.

Out of curiosity, is the same debounce timeout applied to both the press and release keystroke?

[kour1er]

One odd thing that I'm seeing with the 100ms version is some missed keystrokes. I'm seeing it mostly on the f key. And it's not all the time. It seems to happen when I'm typing a relatively long sentence quite quickly (90 wpm typer here). It feels like the keyboard is trying to keep up with me and falling behind and missing a few characters here and there if that makes any sense?

[linduxed]

Another thing about the 100ms version is that while it has the least chatter, it's enough time that it makes fast single key repetition slow down noticeably. I'll still keep this firmware until some other/better solution comes along, but doing something like fast index+middle finger alternation on the Backspace key is definitely slowed down by the 100ms setting.

[mondalaci]

Looks like we don't know what's really going on exactly. I'll expose the state of the key debouncer via USB so that you'll be able to log it, and based on the logs we should be able to fix this. I need some more time.

[linduxed]

I don't know your schedule and priorities (and I assume you have an absolute ton of stuff to do right now), but could you provide me with some instructions for extracing some useful data from the device?

I'd like to be of greater use to you than just sitting here waiting, but I'm not sure how. Or maybe there is research you guys need to conduct first?

[mondalaci]

We've just closed some major firmware issues, and this is the next on my radar. There's nothing you can do until I implement the logging feature. I'll follow up soon, I promise.

[kour1er]

I don't know if you've seen this but I ran across this tool that might be helpful for testing chatter. Admittedly it's windows but that's what dual booting is for :)

SwitchHitter

[TorC8]

I too have been having this issue (and been in contact for some time by email about it).

My experience suggests it is noticeably related to humidity. My office has open windows just about all the time, with substantial open area. My (possibly subjective) sense is that chatter happens more often when it's raining or recently rained than when there is a dry spell.

I live in a high humidity area, and near enough to the ocean that corrosion is a big issue here. Is there anyone experiencing this issue who lives in a dry area?

[mondalaci]

I'm back with a major update! The debouncing algorithm has been rewritten in the newly released firmware 8.4.0, so be sure to flash it.

A word of warning: This is a pre-release for the time being because it might hang your UHK from time to time, although I think it's stable. I'll give it more testing.

There are two relevant variables: debounceTimePress that debounces key presses and debounceTimeRelease that debounces key releases, both equals to 50 ms by default.

You can tweak these variables by building Agent, and then inside of the packages/usb directory of Agent:

$ ./get-variable.js debounceTimePress
50
$ ./set-variable.js debounceTimePress 60
$ ./get-variable.js debounceTimePress
60
$ ./get-variable.js debounceTimeRelease
50
$ ./set-variable.js debounceTimeRelease 40
$ ./get-variable.js debounceTimeRelease
40

You can set these values between 0-255. I think the default values are reasonable, but we should use the minimum values that do not cause bouncing.

Very excited about your findings, folks!

[Jopie01]

Flashed yesterday right away and so far no problems. Everything works like expected! I had a lot of trouble entering passwords :smile: The firmware becomes more and more mature.

[luteijn]

Same here, Nice to have the macro functionality working too.

[kour1er]

Just some quick feedback - not done extensive testing yet but the new firmware is a significant improvement over the old. Great work. Looking into doing the custom build to experiment with the debounce settings.

What turned out to be the solution? Or is it a completely different approach to debouncing?

[mondalaci]

Glad to hear! It's a totally different algorithm.

[linduxed]

For reasons that I have not managed to figure out yet, Agent fails to build for me (something related to the building of node-hid).

Is get-variable.js debounceTimePress and set-variable.js debounceTimePress (and the same for debounceTimeRelease) something that will potentially be included as a configurable in Agent?

[mondalaci]

Feel free to open an Agent issue regarding the build problem and we'll help you.

Yes, debounceTimePress and debounceTimeRelease is the same in the get and set scripts.

I don't plan to make these values configurable via Agent. Please note that they also get defaulted upon power cycling your keyboard. The idea is that we'll test this, come up with the best values (the lowest values that don't cause bouncing) and use them as factory defaults. There's no reason to mess with these values once they're right.

[kour1er]

@linduxed I had to force node 8 rather than node 10 and then Agent built fine.

@mondalaci ok that's super cool to be able to set the two debounce variables in real time 👍

[kour1er]

currently using 70 for the press and 60 for the release. Seems reasonable at those settings. Will continue to test...

[mondalaci]

@kour1er What is your exact methodology for tuning these values?

The original 50-50 values did not produce bouncing on my UHK, and now I'm experimenting with lower values. A binary search algorithm is the most efficient when experimenting with the values and one should branch whether or not bouncing is experienced.

[kour1er]

@mondalaci So I'm using the SwitchHitter application I referenced earlier in this thread and the macOS Debounce utility Debounce to establish a base line.

I started at 10 for both settings then slowly worked my way up on both values until the tools said no bounce. This happened at around the 60/50 mark.

I then typed a lot (scientific I know, right?) and was still seeing some bouncing. Went to 70/50 then finally 70/60 and then typed a bunch more.

The typing consisted of TypeRacer and copy typing at speed some general new articles to get a variety of text (Washington Post / NYTimes type things).

Have you got a more systematic methodology I could try?

[mondalaci]

@kour1er Thanks for explaining! I think you methodology is just fine. :) Did you try to change the release timeout, too? Both values should be minimal.

[kour1er]

@mondalaci phew - I thought I was missing some well know methodology of keyboard testing :)

I was varying the debounceTimePress and debounceTimeRelease variables. Do you have any advice to offer on testing the two? i.e. do you start with both at say 10 and slowly increment only Press and leave Release? Or do you move both up in tandem? Or some variation?

[mondalaci]

@kour1er. This is an excellent question. So excellent I'm not sure about the answer. Maybe it'd be worth do try to decrease the release timeout once the press timeout seems right.

[kour1er]

@mondalaci I will test some more and let you know 👍

Anyone else in this thread have any experimental results?

[linduxed]

@kour1er Indeed, installing nvm and installing Node 8 through it got it all working!

After some experimentation I'm running 25 for Press and 50 for Release. The inverse could maybe also work, although I'm not entirely sure, haven't tested it for long enough. It's also a bit tricky to figure what Press/Release does to the feeling of typing, assuming one just inverts the values.

It seems like this is something that demands a fair amount of testing.

[mondalaci]

Previously, I used 10-10 (press-release) which produced occasional chatter on my side. I've been running 15-15 for several days and I can't experience any chatter. I'm thinking about reducing release but according to your numbers guys mine may already be too low. The numbers should be high enough, so that no UHK exhibits any chatter.

@kour1er Your UHK seems to be the most problematic when it comes to chatter, and I'd be glad if you could reduce these numbers, be it press timeout, release timeout, or both.

Interested about your findings, guys.

[kour1er]

@mondalaci just tried it with your 15-15 settings and seeing a lot of chatter. Out of curiosity, what's your typing speed? I'm wondering if that has an impact?

[kour1er]

From my non-keyboard software writing/manufacturing perspective, I think it would be helpful to establish some sort of shared testing framework, rather than a more ad-hoc 'typing for a couple of days' and seeing how it feels.

In order to have some sort of methodology, I think it would be helpful to have a more objective measuring tool and an agreed upon set of things we are measuring. Thus, here is a list of variables that I can think of to test:

Key characteristics:

• A key may be pressed soft, or hard
• A key may be pressed quickly, or slowly
• A key has a debounceTimePress variable
• A key has a debounceTimeRelease variable

When typing a key may be pressed:

• in isolation
• in conjunction with an adjacent key on the same row
• in conjunction with an adjacent key on a different row
• in conjunction with a far key on the same row
• in conjunction with a far key on a different row
• in conjunction with an adjacent key on the same row on the opposite keyboard half
• in conjunction with an adjacent key on a different row on the opposite keyboard half
• in conjunction with a far key on the same row on the opposite keyboard half
• in conjunction with a far key on a different row on the opposite keyboard half

Even though it's a windows tool, SwitchHitter seems to be the best tool available (very happy if someone else is aware of a better tool?).

Suggested testing procedure:

• Set SwitchHitter threshold to 10ms
• Iterate through the way a key may be pressed list
• Change one of the key characteristics and then iterate through the way a key may bee pressed list again
• Observe difference and report findings.

Does this seem like a reasonable approach? Is there a better more established methodology?

[laxu]

I haven't experienced any chatter on my UHK with Cherry MX Clears. It's possible it's an issue with the switches themselves as on another keyboard I had some chatter issues that were fixed by simply replacing the switch with another one. Like you I tried adjusting debounce times but that only helped to a degree and going too high will result in the occasional dropped input.

[kour1er]

@laxu Interesting. Red switches here. I would agree with you if it were the same switches that were producing chatter all the time but it's inconsistent. Leading me, from my uninformed position, to think that it might still be a firmware issue?

[linduxed]

With me it's sufficiently inconsistent that I don't know how to test it properly anymore. While I discarded the possibility previously, I'm starting to consider whether @TorC8's explanation actually might be related in some way (humidity and so on), because just tweaking the Press and Release values seems to not give consistent results.

I had some time right after the latest update that I ran with 25ms press, 40ms release, and the keyboard felt like how it felt right after I unboxed it: no chatter, great responsiveness and feel. Some days later, the same values resulted in a fair amount of chatter and I had to bump it up to values above the 50-50 default. The trouble is that on those high values, I get instances where keys like `/~ and Left Mod don't register properly occasionally, so I have to choose between occasional extra input and occasional missing input.

Brown switches here, by the way. Also, as stated in passing above, none of this was a problem during the first week or two, I'm fairly sure.

[kour1er]

Inconsistent really is the word isn't it? It almost feels like each key press is hitting a different part of the 'key press / key released' code every time.

Even with the suggestion I made above about a more systematic testing procedure, I'm getting just different results all the time.

[kbranch]

I'm wondering if there might still be some kind of firmware issue involved. If ~5ms is the usual value for cherry switches, it seems suspicious that values an order of magnitude higher are required to (mostly) eliminate issues.

I think I'll try to take a logic analyzer to some switches tonight and see what they're actually doing compared to what shows up on screen.