Serious Bluetooth regression in version 3

[yanshay]

I've been using version 2 for several months now, using bluetooth on both Mac and Windows and bluetooth worked pretty much ok, some issues from time to time but nothing major.

I just upgraded to version 3 here, and have serious bluetooth issues, managed to get it working so that it barely work on the Mac but not at all on windows. Currently in order to connect to the Mac, every time I turn on the keyboard I have to remove all devices on the Mac and then repair. It appears that every time the keyboard starts up it has a new id, so a new keyboard appears on the bluetooth preferences list, so I have to force removing all devices (just the adv360 is not enough) and reconnect. On windows it just won't connect, the operation fails immediately. This makes this version totally unusable.

BTW - in order to get it working in the first place I had to reset the keyboard both with the settings that clear the flash and the reset-settings for version 3 (from the bottom of the page at https://kinesis-ergo.com/support/kb360pro/)

I'm not very familiar with zmk code base, but I think I've seen there were changes made to the ble module that haven't been merged to the code base used here, maybe that's the cause for these issues.

[ReFil]

All changes from ZMK were merged in when the V3 update was performed. https://github.com/KinesisCorporation/Adv360-Pro-ZMK/blob/2d369451ae2a1326f64c37221802e1f4079b590a/config/boards/arm/adv360/adv360_left_defconfig#L52-L57 These lines alter the bluetooth configuration with the goal of improving the bluetooth experience for some mac users who were experiencing issues. Looking at your config repo you have some very odd configuration things going on, have you tried using the factory V3 files to see if there's any change in behaviour?

[ReFil]

https://github.com/yanshay/Adv360-Pro-ZMK/blob/main/config/west.yml Your west.yml references a version of V2 firmware, not a version of v3 firmware. V3 firmware is derived from this branch https://github.com/ReFil/zmk/tree/adv360-z3.2

[yanshay]

https://github.com/yanshay/Adv360-Pro-ZMK/blob/main/config/west.yml Your west.yml references a version of V2 firmware, not a version of v3 firmware. V3 firmware is derived from this branch https://github.com/ReFil/zmk/tree/adv360-z3.2

I'm using a local repo clone, not the one on Github before I switch to it, so what you see on Github is not the one I'm using. I use podman locally to build and it points to your z3.2 branch.

[yanshay]

All changes from ZMK were merged in when the V3 update was performed.

https://github.com/KinesisCorporation/Adv360-Pro-ZMK/blob/2d369451ae2a1326f64c37221802e1f4079b590a/config/boards/arm/adv360/adv360_left_defconfig#L52-L57

These lines alter the bluetooth configuration with the goal of improving the bluetooth experience for some mac users who were experiencing issues. Looking at your config repo you have some very odd configuration things going on, have you tried using the factory V3 files to see if there's any change in behaviour?

I'm referring to changes in the file /app/src/ble.c - current file on zmk main differs from the one in your branch. As I said, I don't know the codebase and only suspect the ble.c has to do with bluetooth.

[ReFil]

There have been changes to ZMK main ble.c since the V3 update, at the time of the update they were the same. Please try the factory V3 firmware available here: https://kinesis-ergo.com/download/advantage360-professional-factory-default-7-6-23/ and the V3 settings reset available here: https://kinesis-ergo.com/download/advantage360-professional-factory-default-7-6-23/

[yanshay]

There have been changes to ZMK main ble.c since the V3 update, at the time of the update they were the same. Please try the factory V3 firmware available here: https://kinesis-ergo.com/download/advantage360-professional-factory-default-7-6-23/ and the V3 settings reset available here: https://kinesis-ergo.com/download/advantage360-professional-factory-default-7-6-23/

You provided the same link twice, can you provide the correct setting reset link to make sure I try the proper one?

[ReFil]

My apologies, here is the link https://kinesis-ergo.com/download/adv360-pro-settings-reset-version-3-0/

[yanshay]

I tried with the above files, I reset both sides, then installed the firmware. In short - exact same behavior. After I remove all devices on the Mac I can connect the keyboard and it works. Then I turn keyboard off and turn on, it blinks slowly and doesn't connect. It appears on the bluetooth settings of the Mac as a new device to connect to but fails to connect when pressing the button to connect. To get it connected I have to remove all devices on the mac (just this one is not enough, need to use the advanced remove all devices) and need to press the BL reset keyboard sequence to get it blinking fast. Only then it connect again. I can see the hex 12 digits code on the Mac of the devices, every time keyboard turns off and on (both if previously it was successfully connected or not) there is a new code. My other bluetooth keyboard keeps the same code all the time no matter what I do so I guess that's causing the issue, it just doesn't persist the code across sessions.

And on my windows machine it just doesn't connect, windows sees it and when I press to connect it quickly respond with 'try connecting your device again'

All these issues don't happen with the the v2 version (well, at least didn't until I tried to upgrade), will need to switch back soon and will know for sure.

[ReFil]

The code changing behaviour you're describing is Bluetooth LE privacy. This is a feature apple explicitly requires in their accessory device specifications. It can be disabled by removing the lines I previously mentioned from adv360_left_defconfig. During testing of v3.0 i was unable to replicate the behaviour you describe on windows or Mac, what computers and Macos version are you using?

[yanshay]

Is this a new requirement from apple? The other keyboard I mentioned that doesn't change its code is an apple magic keyboard. I also don't see these lines in the 2.0 configuration file, so maybe that's indeed the issue. Do I need to remove all lines? Or just the privacy one change to 'n'?

I'm using MacBook Pro 2018, still with BigSur, and Windows 10 on my PC.

[ReFil]

It's been a requirement recently, although i do not know if it tracks back to big sur, remove only the privacy line and try again

major updates will often change the encryption keys on the keyboard, for the windows issue are you trying to pair to the keyboard or connect? you need to remove the pairing in windows and re-pair

[yanshay]

I removed the CONFIG_BT_PRIVACY=y line and now bluetooth works properly. So probably this setting doesn't work well with BigSur.

As for the windows machine, I don't see there the advantage keyboard, what I do is 'add' device, it scans and shows me devices and the pairing doesn't work. I restarted the machine to see there are no leftovers in memory. It's an external bluetooth dongle that worked with version 2 but don't know what vendor exactly.

Could it be one of the other lines?

[ReFil]

unlikely but you can try. the other 5 lines work as one block, you have to remov them all.

[yanshay]

I had some unstable connectivity issues with the Mac, some key-presses were delayed, some just didn't arrive. I removed those 5 other lines and things improved considerably. I still see cases where there is a miss but rare enough to think it might be me and not the bluetooth connectivity. Is there anywhere that documents those settings for non BT experts? couldn't find a clear explanation anywhere.

Also, it doesn't work with my windows machine. Any way on windows to verify it doesn't have any memory of old bluetooth pairing somewhere?

[ReFil]

What Bluetooth adapter are you using on your windows pc? Make sure it is running the latest drivers

[yanshay]

The bluetooth adapter on my Windows is called 'soomfon', it's using Realtek drivers. I updated the drivers and it still doesn't connect. Any ideas what I can check?

As for the Mac, from time to time the bluetooth connectivity becomes very poor. Not as regular as with those 5 lines above, but it still happens and then I have to connect to USB to be able to work. I think (though not sure) it's happening after some time of not using the keyboard, maybe due to it going to sleep?

Would it help to play with those BT settings in those 5 lines I removed? I found various references on the web to those parameters in other contexts with very different values, but don't understand what they mean.

[ReFil]

The top line simply enables overriding of the settings, the next 4 are the actual parameters. We used the apple values available on page 193 of this document https://developer.apple.com/accessories/Accessory-Design-Guidelines.pdf I dont know where you'd start with changing those values as they have a huge range but could severely degrade the experience if set incorrectly. What sort of connectivity issues are you encountering? is it lockops?

[yanshay]

I experience mostly:

1. Delays - I press a key and it takes time until it takes effect, sometime up to what seems like 1/2 a second. And not a single key, when that happens it happens quite frequently to the point I can't work with the keyboard.
2. Events don't arrive, sometimes the key, sometimes the release of the key (and then it autorepeats forever). Also, when it happens it happens many times, not just once.
3. Bluetooth connects/disconnects - I see that when I keep the bluetooth preferences window open and I see the keyboard flips between connected/disconnected. If I wait (could be 10-15 seconds) it usually ends up back to be connected.

When it happens I connect to USB (I can do that quickly) and that behavior disappears. So it doesn't like like keyboard SW or laptop hickups, feels like connectivity. And on USB it never happens.

And other bluetooth devices at the same time are completely stable and function perfectly.

[yanshay]

The top line simply enables overriding of the settings, the next 4 are the actual parameters. We used the apple values available on page 193 of this document https://developer.apple.com/accessories/Accessory-Design-Guidelines.pdf

This is dated 2022, is there such a document from earlier years? My laptop is 2018, maybe they changed chipsets/sw and there were other values for older HW? Like we've noticed that that privacy parameter completely failed the connection probably because my OS/Laptop are older?

[yanshay]

CONFIG_BT_PERIPHERAL_PREF_TIMEOUT=600 
 CONFIG_BT_PERIPHERAL_PREF_MIN_INT=12 
 CONFIG_BT_PERIPHERAL_PREF_MAX_INT=24 
 CONFIG_BT_PERIPHERAL_PREF_LATENCY=30 

From the document:

Peripheral Latency of up to 30 connection intervals

Seems like above setting is at the max latency allowed

Supervision Timeout from 2 seconds to 6 seconds.

Is that the 600 above? what units are those?

Interval Min of at least 15 ms.

Isn't the relevant number above at 12, and it's less than the minimum?

Interval Min is a multiple of 15 ms.

And it isn't a multiple of 15?

Interval Max (Peripheral Latency + 1) of 2 seconds or less. Supervision Timeout greater than Interval Max (Peripheral Latency + 1) * 3.

?

[cvcore]

I experience mostly:

1. Delays - I press a key and it takes time until it takes effect, sometime up to what seems like 1/2 a second. And not a single key, when that happens it happens quite frequently to the point I can't work with the keyboard.
2. Events don't arrive, sometimes the key, sometimes the release of the key (and then it autorepeats forever). Also, when it happens it happens many times, not just once.

Delay and repeating keys - I also experience exactly these two issues from time to time with the ZMK 3.2 firmware. What I’ve identified as a pattern is it only happens for my MacBook Air with M2 chip but not for an older intel MacBook Pro from year 2021.

For me I suspect the 2.4 GHz band used by Bluetooth is somehow jammed. Because when using the keyboard with 2.4G wifi connection, it happens very frequently, almost once per minute. Switching the wifi connection to 5Ghz band significantly reduces the issue frequency, but still happens from time to time. Also causing the issue could be my Logitech mouse connected through unifying protocol, but I can’t identify a positive correlation here.

[ReFil]

The delay and lockup issue could also be connected to this fix https://github.com/zmkfirmware/zmk/commit/9a963abfc8b27c6fdfb5b6b4b38e5905d6a360a7 which has yet to be merged into the adv360 branch but is on the list of things to do

[jasselin]

I have the same behavior here with repeating keys, even when I use my Macbook keyboard while the 360 is connected. No repeat or delay when I shut it off. Any idea when this could be pulled from the zmk repo?

[ReFil]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

[cvcore]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

@ReFil Do you have a link to your branch with that commit merged?

[cvcore]

I have the same behavior here with repeating keys, even when I use my Macbook keyboard while the 360 is connected. No repeat or delay when I shut it off. Any idea when this could be pulled from the zmk repo?

@jasselin my solution atm is just to implement a feature that allows you disconnect the keyboard from macbook's bluetooth menu when that happens. Not perfect, but works for me in most cases. Ref: https://github.com/ReFil/zmk/pull/7

[ReFil]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

@ReFil Do you have a link to your branch with that commit merged?

Sorry I dont as of yet, it's just a test branch on my laptop. I can push it if needs be

[cvcore]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

@ReFil Do you have a link to your branch with that commit merged?

Sorry I dont as of yet, it's just a test branch on my laptop. I can push it if needs be

That would be great, so I can also test it with my own configurations. Thanks!

[ReFil]

https://github.com/zmkfirmware/zmk/issues/1900 This could possibly be connected

[yanshay]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

@ReFil Do you have a link to your branch with that commit merged?

Sorry I dont as of yet, it's just a test branch on my laptop. I can push it if needs be

That would be great, so I can also test it with my own configurations. Thanks!

Same here. BTW - I witness more and more BT related issues making the keyboard unusable with BT for long time.

[KinesisCorporation]

Hi @yanshay, Can you elaborate?

• When you say "more and more issues" are you comparing two firmware versions or are you saying performance is degrading on the same version? Note there are now 3 main "versions" available at the moment: 2.0, 3.0, or 3.0 beta?
• Are you having problems connecting Keyboard-to-PC, or left-to-right module, or both?
• What O/S?
• What is the specific failure mode/sequence you are experiencing?

[cvcore]

There's now a beta firmware with that change merged. It would be greatly appreciated if yyou could try it and give feedback https://kinesis-ergo.com/support/kb360pro/#firmware-updates

@ReFil Do you have a link to your branch with that commit merged?

Sorry I dont as of yet, it's just a test branch on my laptop. I can push it if needs be

@ReFil any updates?

[ReFil]

https://github.com/ReFil/zmk/tree/adv360-z3.2-beta

apologies for the delay

[cvcore]

https://github.com/ReFil/zmk/tree/adv360-z3.2-beta

apologies for the delay

I switched to this branch with my own keymap modifications and also merged additional logics I made to handle the BLE disconnection from host computer. After around 1h of usage I don't observe the repeating keys issue as before. Also typing on the MacBook internal keyboard no longer gives me a laggy feeling anymore.

I will follow up when I observe any new issues from this version. But so far it's a great improvement for me. Many thanks for the work :)

[KinesisCorporation]

Still working better?

[cvcore]

Still working better?

Yes :) Tested on my both MacBooks (2019, Intel and 2023, M2).

No repeating keys observed when typing on both the Adv360Pro as well as the internal keyboard, even when keyboard is away from the laptop (putting them in different rooms and connected).

[lyallcooper]

I experience this issue as well, although the main manifestation is when I take my laptop (2021 M1 MacBook Pro, macOS 13.5) away from my desk (and Adv360) and into another room. My laptop keyboard will become unusable due to dropped, repeated, laggy keys (as described above) unless I turn off bluetooth or the Adv360 itself.

@cvcore's patch helps in that it lets you disconnect the keyboard from the bluetooth menu without turning off bluetooth entirely, but neither it nor the updated version of zmk the branch is based on fix the underlying issue. If anything the performance seems a little worse on the ReFil:adv360-z3.2-beta branch that it does the stable adv360-z3.2 one? But thank you to both @cvcore and @ReFil for working on this!

In general, the bluetooth performance for my Adv360 has been significantly worse that the other keyboard I have that also runs zmk, and I (poorly) soldered that one together myself so if anything it should be the other way around. The bluetooth performance is not bad enough to make the Adv360 unusable for me, but it's quite annoying sometimes. And the difference between using the Adv360 over bluetooth vs over USB with bluetooth disabled is like night and day.

[KinesisCorporation]

I experience this issue as well, although the main manifestation is when I take my laptop (2021 M1 MacBook Pro, macOS 13.5) away from my desk (and Adv360) and into another room. My laptop keyboard will become unusable due to dropped, repeated, laggy keys (as described above) unless I turn off bluetooth or the Adv360 itself.

@cvcore's patch helps in that it lets you disconnect the keyboard from the bluetooth menu without turning off bluetooth entirely, but neither it nor the updated version of zmk the branch is based on fix the underlying issue. If anything the performance seems a little worse on the ReFil:adv360-z3.2-beta branch that it does the stable adv360-z3.2 one? But thank you to both @cvcore and @ReFil for working on this!

In general, the bluetooth performance for my Adv360 has been significantly worse that the other keyboard I have that also runs zmk, and I (poorly) soldered that one together myself so if anything it should be the other way around. The bluetooth performance is not bad enough to make the Adv360 unusable for me, but it's quite annoying sometimes. And the difference between using the Adv360 over bluetooth vs over USB with bluetooth disabled is like night and day.

Please re-flash with today's release, test again and report back. We have tested physically separating the Adv360 from a 2023 MacBook Pro running 13.5.1 by walking it far enough away that the Mac and 360 become disconnected. The laptop keyboard works fine while the separated and re-uniting the keyboard Mac has the 360 keyboard working fine as well.

[eugene8080]

I'm also having trouble with repeating keys after upgrading to v3.0. I did not have the issue on v2.0. I changed my west.yml file to "revision: adv360-z3.2-beta", but I get a "Process completed with exit code 1." error after I commit changes and attempt a Build action. What did I do wrong? https://github.com/eugene8080/Adv360-Pro-ZMK/blob/V3.0/config/west.yml

Laptop: Surface Laptop Studio OS: Win11

[ReFil]

Hi, theree was a bug in the online build process that has since been fixed, you should be able to build your firmwares now, apologies for the inconvenience

[cvcore]

Please re-flash with today's release, test again and report back. We have tested physically separating the Adv360 from a 2023 MacBook Pro running 13.5.1 by walking it far enough away that the Mac and 360 become disconnected. The laptop keyboard works fine while the separated and re-uniting the keyboard Mac has the 360 keyboard working fine as well.

Hi @KinesisCorporation , today I got some time to test the v10-23-23 firmware downloaded from https://kinesis-ergo.com/support/kb360pro/#firmware-updates. I followed exactly the instructions when flashing the firmware. Unfortunately, the test result is not satisfying.

I tested by typing two paragraphs of text from https://official-typing-test.com/test/10test.html and made this screen recording. The repeating keys issue happened for me 3 times within a time frame of 5 minutes (as can be seen in 0:54, 1:59, 3:04).

For a fair comparison, I've also used a HHKB-BT keyboard paired to the same macbook via Bluetooth and typed for 5 minutes on the same website. There were no repeating keys observed. So the Bluetooth connectivity shouldn't be an issue.

Some more hardware info to help debug:

• Laptop: MacBook Air M2, 2022 Model Mac14,2
• OS: macOS Ventura 13.0
• Bluetooth: Chip BCM_4387
• ADV360 has RSSI of -64 dBm when connected (in comparison my HHKB-BT has -45 dBm)

Since this issue is impacting many users already, I'm hoping for a systematic approach to identify the root cause of the issue before releasing a newer version. It's a great piece of hardware and I hope the software as well :)

[eugene8080]

I tested ReFil:adv360-z3.2-beta for a 10min typing test and had no repeat keys errors. Repeating keys issue appears to have been fixed for me.

Keyboard: Kinesis Advantage360 Pro Firmware: adv360-z3.2-beta Computer: Surface Laptop Studio, 11th Gen Intel(R) Core(TM) i7-11370H @ 3.30GHz OS: Windows 11 Home 22H2

[KinesisCorporation]

@cvcore

We should probably open new "issues" that are laser focused on one specific "bug". The latest update was targeting improvements in syncing between the left and right module, and stability of connection with the PC.

Please start a new issue pertaining to your repeating keys and we'll look to get to the root cause of that. Thanks!

[yanshay]

Please start a new issue pertaining to your repeating keys and we'll look to get to the root cause of that. Thanks!

When I had repeating keys issues those definitely were driven by BT issues. I've also had other symptoms. I'm not sure you want to fill this repo here with all BT symptoms one by one when they definitely feel like all driven by BT issues.

Anyway, the interference BT issues on my end seem to be much better from the time I've opened this issue. The main thing I've done was to move Logitech wireless dongle further from my laptop and keyboard using a USB extension coord. I think the Advantage Pro is prone to BT interference from Logitech, maybe low transmittion or reception power. I saw someone else on this issue also wrote he had such issues. Other BT devices I have didn't have those issues, only the Advantage Pro had.

I also updated to the latest branch a few days ago, I think that improved on the disconnect issues which took place once in a while prior to doing that upgrade, I did not have that since I made the update. It may be too early to tell for sure but I think it made a difference.

BTW - I used the 3.2-2 branch, seems to me that's the latest one.

[ReFil]

Yes, the 3.2-2 branch uses the current latest updates. It implements the repeating key fix here: https://github.com/zmkfirmware/zmk/commit/9a963abfc8b27c6fdfb5b6b4b38e5905d6a360a7 as well as a host of other improvements. The 3.2-beta branch is currently equal with 3.2-2 however is subject to frequent, often breaking changes so unless you're willing to be on the bleeding edge it's recommended you use 3.2-2

[ReFil]

@yanshay just following up, have you encountered further issues since updating and making those setup changes?

[yanshay]

@yanshay just following up, have you encountered further issues since updating and making those setup changes?

It's much better and completely usable so far on BT. I think I did notice a few cases where when turning on the keyboards (left then right) it doesn't work and the right side blinks error, but turning off and on again both gets it to work, so not a big deal for me. It then stays connected w/o noticeable issues.

[ReFil]

For the left right connection try leaving the left on for a few seconds before turning the right on, since the 10/20/23 update having the left on for a bit (Once the leds come online it should be good) before turning the right on should help. If you're happy with the performance as is are you ok to close this issue out?

[ReFil]

@cvcore please try this change that another macos user reported helped their situation significantly https://github.com/KinesisCorporation/Adv360-Pro-ZMK/issues/262#issuecomment-1793689158

[lodi]

Sorry to take this off topic but I found this thread while trying to troubleshoot why I couldn't connect via Bluetooth to an Android tablet. The tablet would pair, but then wouldn't connect in any way. I tried commenting out the CONFIG_BT_PRIVACY=y line as suggested above and it completely resolved the issue for me.