search

atar-axis / xpadneo

Advanced Linux Driver for Xbox One Wireless Controller (shipped with Xbox One S)
https://atar-axis.github.io/xpadneo/
GNU General Public License v3.0
2.01k stars 113 forks source link

[NOT FOR QUESTIONS] Bluetooth connection stability broken #198

Open kakra opened 4 years ago

kakra commented 4 years ago

NOTE: Do not report here if you cannot reproduce a working state with a previous kernel version.

This is a information collection issue to pinpoint the kernel change that broke the Bluetooth stability with the controller. I'll ask anyone who wants to help to provide the following information:

Stay updated

Subscribe to this thread to stay updated about Bluetooth issues with Xbox controllers. Use the button on the right to subscribe, no need to leave a comment.

Observations

  1. If you're seeing the problem mostly while rumble effects are playing, I strongly suggest to update the controller firmware first. You can do this by connecting the controller to a Windows box, running the Xbox app, and applying the update from within the app. Then go back to Linux. After re-connecting the controller to Linux, it may start in the wrong mapping mode. My latest code re-design should fix it but if it doesn't, maybe open a new bug report with all the details, then remove the controller from your Bluetooth settings and re-pair it (or if you want to help out, don't remove and re-pair it just yet and wait for my reply). If you don't have a Windows box, you can download an evaluation version of Windows 10, install it in a VM and use USB pass-thru to make the controller visible in the VM.
  2. There's also a known problem that causes instabilities, input lag, or pairing/connection issues due to Bluetooth discovery continuously running in the background. To observe this yourself, run btmgmt to find that discovery is continuously active or switches off and back on almost immediately to run for around 10s each cycle. Then log out of your desktop environment to observe that this is no longer an issue, and now your controller may connect just fine and the connection is stable. The kdeconnectd daemon may be (directly or indirectly) involved here. Running pkill kdeconnectd may resolve the issue for you until the clients spawns it again (e.g., kdeconnectd/Plasma browser integration).
  3. Bluetooth LE is relatively new and uses a different approach to support HID devices. The current implementation of bluez may not set latency properties properly, but as a work-around, you can set them manually. See below.
  4. Bluetooth LE pairing may be problematic, depending on privacy settings and kernel version. Thus it's recommended to use the latest versions. If the controller pairs correctly, you should see a very reliable connection, in contrast to older Xbox controller models.

Prerequisites

  1. You've not experienced the issue before updating the kernel
  2. You can reproduce the issue with your currently running kernel
  3. You are using the same USB dongle before and after the update and tried all the dongles you own
  4. You are not using different distributions for each kernel
  5. If possible, you've updated the controller to its latest firmware (see above).

Information collection

Please provide kernel versions as shown by uname -a if possible because distributions often implement their own patchlevel versioning.

  1. Which distribution are you running?
  2. Last known working kernel version
  3. First known broken kernel version
  4. Kernel timing setting: zgrep '^CONFIG_HZ' /proc/config.gz
  5. Run lsusb and take note of the device number of your USB Bluetooth dongle
  6. Run lsusb -v -s## | tee xpadneo-lsusb.txt where ## is the device number picked in the previous step, post the resulting file.
  7. Disconnect your controller, run sudo btmon | tee xpadneo-btmon.txt and connect the controller, post the resulting file.

Thanks!

Kernel bug reports

  1. https://bugzilla.kernel.org/show_bug.cgi?id=199843 - Xbox One S and other bluetooth devices fail to connect
  2. https://bugzilla.kernel.org/show_bug.cgi?id=198919 - Xbox (One) Wireless Controller won't connect
  3. https://bugzilla.kernel.org/show_bug.cgi?id=209745 - BLE devices fail to connect in privacy mode
  4. https://lore.kernel.org/linux-bluetooth/CAC2ZOYuG_BBG0uisXkoTtroxwgvv+JO-yMFC2+bRQZjsvwK1qw@mail.gmail.com/T/#u - Bug report to linux-bluetooth with ERTM enabled and L2CAP patch applied

Bluez bug reports (maybe also kernel bug reports)

  1. https://github.com/bluez/bluez/issues/123 - Continuous discovery disrupts Xbox controller stability
  2. https://github.com/bluez/bluez/issues/155 - Pairing issues with Bluetooth LE Xbox controllers
  3. https://github.com/bluez/bluez/issues/156 - High input latency / event loss with Bluetooth LE Xbox controllers

KDE Plasma bug reports

  1. https://bugs.kde.org/show_bug.cgi?id=435444 - Continuous discovery running despite no Bluetooth wizard running
  2. https://bugs.kde.org/show_bug.cgi?id=401983 - Unstable wireless connectivity while discovery is running (also affects wifi)

Changelog

This documents some of the changes found with some kernel updates.

Kernels 5.4.72, 5.8.16, 5.9.1, 5.10...

Contains some Bluetooth pairing fixes. Doesn't seem to affect XB1S. Some XBE2 users report the 20s connection delay gone but I cannot confirm that. Bluetooth commits:

Luiz Augusto von Dentz (5):
      Bluetooth: A2MP: Fix not initializing all members
      Bluetooth: L2CAP: Fix calling sk_filter on non-socket based channel
      Bluetooth: MGMT: Fix not checking if BT_HS is enabled
      Bluetooth: Consolidate encryption handling in hci_encrypt_cfm
      Bluetooth: Disconnect if E0 is used for Level 4

Some users reported a broken Bluetooth connection with these latest patches except for 5.9.1 or higher.

Kernels 5.9, 5.10

The kernel Bluetooth stack solved a bug for BLE devices in privacy mode: https://bugzilla.kernel.org/show_bug.cgi?id=209745, seen since 5.9, solved since 5.10.4, not sure if it has been backported yet but it usually gets fixed, too, in all kernel versions chronologically newer than 5.10.4 so 5.9.17 should be fine, too.

Update: The Privacy=device setting is actually not needed and should not be used - leave it at its default setting.

Continuous discovery caused by desktop environments

At least KDE Plasma seems to exhibit a behavior that causes it to continuously trying to discover nearby devices which in turn makes Bluetooth connections from Xbox controllers unstable and hardly possible to connect. The problem was discovered while investigating this report: https://github.com/bluez/bluez/issues/123

The bug is not in KDE Bluedevil itself but rather another component which probably triggers it by listing the contents of the bluetooth:/ kio virtual directory: https://bugs.kde.org/show_bug.cgi?id=435444

If your controllers connect without problems and the connection is stable after logging out of your desktop environment, you're probably affected by this behavior. Different Xbox controller models act slightly different when affected by this behavior: Sometimes it pairs and connects just fine but input events get lost or laggy, sometimes the connecting or pairing may just take 30-60s but the input event transport itself is unaffected.

We currently know of only one way to work around this if KDE or kdeconnect is used: Kill kdeconnectd (pkill kdeconnectd) or install kdeconnectd without Bluetooth support.

HCI event ordering

There may be an issue with the ordering of HCI events exchanged between the Bluetooth stack and the gamepad:

  1. https://github.com/bluez/bluez/issues/127 - diff of btmon log between a failed and a successful connect

Bluetooth LE issues

Latest Xbox controllers use Bluetooth LE. While the pairing and connecting is much more reliable IF it works, the Bluetooth stack of Linux still seems to see a lot of changes and fixes for it. If you're experiencing issues, ensure updating to the latest controller firmware, and also kernel and bluez version. Some manual tuning may still be needed to fix latency issues, see https://github.com/bluez/bluez/issues/156#issuecomment-864547205.

Upstreaming efforts

Reported by @hadess in https://github.com/atar-axis/xpadneo/issues/44#issuecomment-767550295:

The pre-requisite Bluetooth patch has landed in bluetooth-next, and work on upstream can proceed.

envyniv commented 4 years ago
  1. Ubuntu 20.04
  2. I'm not sure, but i think any 5.3 up to 5.4.0-31-generic?
  3. current, ubuntu-patched?; 5.4.0-33-generic
  4. xpadneo-lsusb.txt
  5. xpadneo-btmon.txt hope i managed to help in some way. :) if i manage to find something else, i'll edit this/post another comment
envyniv commented 4 years ago

i know i shouldn't chat here, but am i the only one that seems to correlate this issue with the kernel update?

kakra commented 4 years ago

I think this will get some attention with the next version bump.

wangyu6 commented 4 years ago

So ,will next version support xbox-one-s? ( it just couldn't connected pc(ubuntu-1804) through bluetooth)

kakra commented 4 years ago

So ,will next version support xbox-one-s? ( it just couldn't connected pc(ubuntu-1804) through bluetooth)

I don't want to be rude but this issue is solely about collecting information with regressions. Please look at the other issues, the one's linked above are a good starting point. Otherwise feel free to open a new issue report. I'll mark off-topic comments as off-topic now. Thanks in advance for understanding and your cooperation.

kakra commented 4 years ago

HEADS UP

Anyone affected may try #204 and report in #189 or #180. After the latest discoveries, this may prominently affect people with high-HZ kernel configurations (a 1000 Hz kernel seems to show the problem much more likely than a 100 Hz kernel, I'm using 300 Hz and didn't have many issues but many game-optimized kernels often opt-in for 1000 Hz, tho I think that's overkill, 300 Hz may look odd but fits best here as it cleanly divides by most common FPS rates (25 Hz, 30 Hz, 50 Hz, 60 Hz, 100 Hz) and should thus reduce jitter. If you're not an Ultra-High-FPS gamer (120+ FPS), I recommend using 300 Hz.

For checking your settings, follow the instructions at the top of this page.

envyniv commented 4 years ago

[mark this as off-topic] after a month, I decided to retry connecting my controller to my computer. it now works. I have no idea what has changed. i didn't update the driver, I, pretty much, did not do anything. my guess is that the latest kernel patched the issue? I don't know anything about anything. So sorry i couldn't help.

kakra commented 4 years ago

There's also an issue with establishing an encrypted connection between the controller and the Linux Bluetooth stack. The first problem is (from my tests) often a missing link key. This can be fixed by retrying to remove the controller in bluetoothctl and then pairing it again, immediately followed by trust, repeat 1-3 times until the link key shows up in /var/lib/bluetooth/HOST-MAC/PEER-MAC/info. During pairing you may need to press buttons on the controller (or click the connect button again) to confirm, but that may also be just placebo - at least clicking the connect button sends data to the Bluetooth stack for some reason. Once the link key is there, it now longer constantly disconnects and reconnects due to a missing or non-matching link key. But still, the controller may take up to 60s for establishing the encryption. I don't know whats wrong but often the handshaking fails until it eventually works after 3-5 times. I wonder if the firmware tries multiple encryption methods until it finds one that works with Linux. It may well be related to the kernel or the bluez version after the initial link key issue was resolved.

Will mark as off-topic later.

envyniv commented 4 years ago

ight that's good to know, if this problem arises again and the fix you've proposed doesn't work, i'll be sure to hit this up again.

richardtatum commented 4 years ago

I have been meaning to post this for a while, which means my data is somewhat out of date, so I can only apologise.

  1. Arch Linux 5.8.14-zen1
  2. 5.8.10-ish?
  3. 5.8.10-ish onwards
  4. CONFIG_HZ_1000=y CONFIG_HZ=1000
  5. xpadneo-lsusb.txt
  6. xpadneo-btmon.txt
  7. Whether or not this is helpful I do not know, but here is the bluetoothctl logs for trying to connect whilst piping the above btmon output.
    • ERTM is disabled.

After updating the kernel, it started connecting very slowly (30s+) then stopped connecting altogether. I switched to windows and updated the firmware on the controller, then switched back to linux and I was no longer able to connect via bluetooth. Connecting via bluetoothctl usually end in the same way, I remove the controller from paired devices, I scan for it (sometimes it takes a very long time to show up, or does not show at all and I have to turn the controller off/on and put back into pairing mode), trust, pair and attempt to connect (which always fails).

If you need anything further, please let me know.

Edit: After reading various issues, I was able to connect via bluetoothctl by trusting the device, then connecting immediately (skipping the pair step). Obviously this does not persist through power cycles though.

kakra commented 4 years ago

I'm seeing a similar problem after upgrading to latest 5.4.y: firmware was already up to date for me but I had a lot of connection dropouts and establishing connection is very slow (30+ seconds), so I removed the controller from Bluetooth and paired it again, and now connection never happens automatically, I always need to use "connect" command manually (or via GUI). But at least it seems stable once connected. The XBE2 controller acts very differently: It absolutely has no problem establishing the Bluetooth connection (so there seems to be no general kernel problem), but it stalls in the HID layer for around 20s before xpadneo finally initializes it (probably due to its huge HID descriptor), then it drops the connection as soon as the first few real rumble packets are sent by games (the notify rumble works, tho).

There are some unofficial logs about XB1S firmware upgrades, some of them contain notes about improving Bluetooth connection quality and fixing issues with connecting to some Android devices. So there are definitively changes in the Bluetooth part of the controller.

Initially found here: https://answers.microsoft.com/en-us/xbox/forum/all/new-release-of-controller-update-23-october-2019/4ce5abe6-86f8-4490-bda6-c3943c31713e

Then followed the link to "ASA Insider": https://news.xbox.com/en-us/2019/10/23/xbox-insider-release-notes-alpha-skip-ahead-ring-2004-191021-2100/

From there it gets quite messy to follow the update logs but it lists a lot of Xbox firmware updates which also include the controller firmwares.

I think the kernel is missing some bits and pieces that the controller actually expects (and which Android may do differently), as the controller seems to have stable Bluetooth connections with both Windows and Android. OTOH, I believe the Bluetooth implementation in the controller firmware is not very well done (and not very tolerant for deviating from implementation expectations).

kakra commented 4 years ago

@richardtatum 

[CHG] Device 5C:BA:37:D6:DE:E9 ServicesResolved: yes
[CHG] Device 5C:BA:37:D6:DE:E9 Paired: yes
Pairing successful
[CHG] Device 5C:BA:37:D6:DE:E9 ServicesResolved: no
[CHG] Device 5C:BA:37:D6:DE:E9 Connected: no

Yeah there's something happening that immediately kicks the controller again (or it kicks itself). This may end in an endless cycle of auto-connecting and then immediately disconnecting again. Only host-initiated connection attempts do seem to work currently, at least most of the times, sometimes you'll have to retry and hit the right timing by luck.

kakra commented 4 years ago

HEADS UP

Kernel 5.4.72 brought some patches regarding Bluetooth pairing. All chronology later kernels (5.8.16, 5.9.1) should have the same patches. For my controller, this doesn't make any difference but you may want to try.

richardtatum commented 4 years ago

Thanks @kakra. I thought I'd try with the latest update (5.9.1-zen2 for me) but no luck. After my initial edit of being able to connect to the controller by omitting the 'pair' step, I have not been able to do it since. Now I am at a stage where the controller does not actually show up in the discovered devices in bluetoothctl when the scan is on. For now I am just using the controller in wired mode until there are some further patches to the kernel or a reasonable work-around.

kakra commented 4 years ago

Scan usually doesn't show your device if it is already known. Use devices in bluetoothctl to find the MAC address to remove, then use remove <MAC>. Now, with scan on the controller should show up, you may need to hold the connect button so it doesn't try to reconnect to host it is no longer associated with. Worked for me when I ran into that situation.

Status: Will mark off-topic later as this is not for discussions.

schmitmd commented 4 years ago

Confirming. My XBES2 controller was suffering from the 20s delay on initial connect. Since updating to kernel v5.9.1, connection is now basically instantaneous.

kakra commented 4 years ago

@schmitmd I cannot confirm this for 5.4.72. But the actual commits tell that this pairing problem only came apparent after some Bluetooth changes introduced with 5.9. So older kernels may miss some other changes. I'm currently not updating to 5.9 to do any confirmations because I'm waiting for it going LTS which will be around new year. Since kernel development is around 2 months early now compared to the previous years, even 5.10 may become LTS.

I think the instantaneous connect may actually be a different thing. Have you been running 5.8 before and now switched to 5.9?

kakra commented 3 years ago

The kernel Bluetooth stack solved a bug for BLE devices in privacy mode: https://bugzilla.kernel.org/show_bug.cgi?id=209745, seen since 5.9, solved since 5.10.4, not sure if it has been backported yet. Updated topmost post accordingly.

kakra commented 3 years ago

Confirming. My XBES2 controller was suffering from the 20s delay on initial connect. Since updating to kernel v5.9.1, connection is now basically instantaneous.

I can confirm the same for kernel 5.10.4 (upgraded from 5.4.85).

kakra commented 3 years ago

To all participants of this thread: Do you still see issues with kernel 5.10.4 or later?

richardtatum commented 3 years ago

Sorry for the incredibly late response on this, but yes. I am still having the exact same issues on 5.10.16.

Edit: Obviously as soon as I submitted this it appears to have started working correctly...

I did a wipe and re-pair and my controller now seems to connect correctly and consistently with no change to the kernel.

Thanks for all of your assistance with this @kakra, it was really appreciated.

kakra commented 3 years ago

Yes, it's important to pair the controller to a different machine and remove it from the Bluetooth stack. Otherwise one partner may fall back to a partially broken and old state when re-pairing.

kakra commented 3 years ago

I've added bug reports to the initial report above to cover instabilities due to continuous discovery.

kakra commented 3 years ago

Identified two possible problems for connection stability and initial connection: kdeconnectd is somehow involved as we see continuous Bluetooth discoveries going on, overwhelming the RF environment, resulting in the controller losing connection. Also, the initial connect may depend on proper ordering of some HCI events but this needs further investigation. I've updated the initial post on this issue.

kakra commented 3 years ago

Updated initial post regarding Bluetooth LE issues.

1985a commented 3 years ago

In my case, this is the solution Xbox controller model 1914 In the main.conf Bluetooth file add the following

[General]
ControllerMode = Dual
JustWorkingRepairing = confirm

[LE]
MinConnectionInterval = 7
MaxConnectionInterval = 9
ConnectionLatency = 0
kakra commented 3 years ago

Some general updates on the topic since I was finally able to test the issue on a Windows system which had multiple Bluetooth dongles connected (that system is extensively used for VR, and the user connected all the Bluetooth dongles that came with the various accessories):

The tests included an XB1S controller which we updated to the latest firmware first. Connection with the internal AX200 Bluetooth was almost always impossible, the connection was unreliable, and most of the time it connected in some limbo state where Windows would say "connected" but the controller keeps flashing. This reminds me a lot of the behavior we can see in Linux. The other Bluetooth devices had similar issues: While they usually were able to connect, pairing and connected took a long time, some devices needed several tries before they worked. This results in the conclusion that the AX200 chipset is just borked and incompatible. Don't use it.

With only one CSR 4.0 dongle connected, all devices were able to pair and connect almost instantly. Even the XB1S controller connected instantly. This works a lot better than I'm used to it in Linux: In Linux, it CAN connect instantly, but often it takes several seconds. This results in the conclusion that there are still issues with the bluez daemon.

The kernel drivers for the chipsets may have an impact on that but I think most of the drivers are using btusb anyways, the driver should be in good shape, and only some new chipset revisions or cheap China clones may need btusb driver updates or quirk fixes. This leads to the conclusion that you want to use CSR chipsets most likely because those are supported very well and work with the gamepad. There's only the problem with a lot of CSR China clones on the market which you should try to avoid (but may not be possible because who knows).

1985a commented 2 years ago

I don't know if someone can see help with this but I felt the need to share it. I'd like to say at the moment the kernel 5.18.15 and these devices, are working together so nice.

Before, I made several attempts in order to get this controller to work on BT mode, but it never any success.

This is my Bluetooth main conf file, that I'm using, and as I say, it's working like some kind of magic after updated the firmware.

PD: excuse me my bad English, it is not my mother language.

[General]
ControllerMode = dual
JustWorksRepairing = confirm
[BR]
[LE]
MinConnectionInterval=7
MaxConnectionInterval=9
ConnectionLatency=0
MinConnectionInterval=9
MaxConnectionInterval=12
[GATT]
[AVDTP]
[Policy]
[AdvMon]
dnet890 commented 2 years ago

same issue, it does not work on Ubuntu 22.04