search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
4.56k stars 940 forks source link

SwitchBot Bluetooth integration not working well since HAOS 9.5 #2961

Open rhpijnacker opened 7 months ago

rhpijnacker commented 7 months ago

Describe the issue you are experiencing

I have a number of SwitchBot Curtains connected via the Bluetooth and SwitchBot integrations. When I open or close one of them, there seems to be a 10 second delay before it responds. When I open or close multiple simultaneously, they seem very slow to respond, and it appears like the are operated in sequence.

More people seem to be affected by this issue, as reported in https://github.com/home-assistant/core/issues/92839.

What operating system image do you use?

ova (for Virtual Machines)

What version of Home Assistant Operating System is installed?

11.1

Did you upgrade the Operating System.

No

Steps to reproduce the issue

  1. Create a group with multiple SwitchBot Curtains
  2. Open or close the group
  3. With HAOS 9.5 this opens or closes all of the curtains simultaneously, and they can be interrupted
  4. With HAOS 10.0+ it takes a while for the first curtain to start moving and the other curtains seem to be running in sequence. Interrupting does not seem to be working, or very slowly.

Anything in the Supervisor logs that might be useful for us?

Nothing that stands out

Anything in the Host logs that might be useful for us?

Nothing that stands out

System information

System Information

version core-2023.11.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.11.6
os_name Linux
os_version 6.1.59
arch x86_64
timezone Europe/Amsterdam
config_dir /config
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 11.1 -- | -- update_channel | stable supervisor_version | supervisor-2023.11.6 agent_version | 1.6.0 docker_version | 24.0.6 disk_total | 30.8 GB disk_used | 3.9 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons |
Dashboards dashboards | 1 -- | -- resources | 0 views | 1 mode | storage
Recorder oldest_recorder_run | 3 december 2023 om 19:49 -- | -- current_recorder_run | 3 december 2023 om 20:59 estimated_db_size | 0.32 MiB database_engine | sqlite database_version | 3.41.2

Additional information

I was able to trace down this problem to one commit in HAOS:

commit 25c672a27fde665a0f661675a8b7014e1fe796ee
Author: Stefan Agner <stefan@agner.ch>
Date:   Thu Feb 16 14:53:38 2023 +0100
    Linux: Update kernel 6.1.12 for non-Raspberry Pi systems (#2358)

A custom build right before this commit works perfectly:

commit 26757f17e85e1fdb03c489d20fb63e35c2ef1d2a
Author: Stefan Agner <stefan@agner.ch>
Date:   Wed Feb 15 11:01:50 2023 +0100
    Linux: Update kernel 5.15.94 (#2356)

starting with the commit the problems are reproduceable.

I opened and closed a group of curtains in both versions with the debug logging enabled for both Bluetooth and SwitchBot integrations. I added the homeassistant.log for both versions as attachments.

bluetooth-ok.log bluetooth-fail.log

andorardo commented 7 months ago

Confirmed still an issue with HAOS 11.2.

Related issue in HAOS: https://github.com/home-assistant/operating-system/issues/2612

Related issue in HA Core: https://github.com/home-assistant/core/issues/102077

sairon commented 7 months ago

That confirms what @agners said in the discussions thread - the kernel update is indeed the cause. I was able to reproduce Blutetooth issues easily after the upgrade too and ended up bisecting the kernel, which pointed me to this commit in linux-stable tree:

commit 27d54b778ad1fb32c2c108cfe97e861c3909a46f
Author: Brian Gix <brian.gix@intel.com>
Date:   Fri Aug 5 16:42:29 2022 -0700

    Bluetooth: Rework le_scan_restart for hci_sync

    le_scan_restart delayed work queue was running as a deprecated
    hci_request instead of on the newer thread-safe hci_sync mechanism.

    Signed-off-by: Brian Gix <brian.gix@intel.com>
    Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

 net/bluetooth/hci_request.c | 89 ---------------------------------------------
 net/bluetooth/hci_sync.c    | 75 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+), 89 deletions(-)

It took about 13 steps when bisecting between the two tagged versions - maybe if you check changes around the above patch first, it could take a bit less - more Bluetooth refactoring and rewites happened around that patch. I'd be glad to help resolving that but unfortunately I'm unable to reproduce any Bluetooth issues reliably anymore. If you're willing to do some further tests, let me know.

Btw. I'm missing the information what Bluetooth adapter are you using, could you tell us?

rhpijnacker commented 7 months ago

I am certainly willing to do more testing; gladly. However, I would need some help picking the right kernel sources. If you could point me to some documentation on how to pick a kernel version commit, that would be great.

With respect to the Bluetooth adapter used: I'm not exactly sure. Can I find this information somewhere? I tried looking in the output of dmesg, but that did not really tell me much. My test system is an old iMac (13,1) running proxmox. I can reproduce the problem on both production and test system.

sairon commented 7 months ago

You are probably already familiar with how bisecting works but maybe someone else finds the description of the whole process useful, so here it is:

  1. Clone the linux-stable repository.
  2. Start bisecting: git bisect start
  3. Mark known good/bad revisions: git bisect good v5.15.94 and git bisect bad v6.1.12
  4. Git shows you the hash of the current commit - take it and use it as BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE in the board config. AFAIK nothing else needs to be changed at this point, there are no breaking changes that would need more tweaks in the HAOS codebase.
  5. Rebuild the OS, install the update (fastest is to take the .raucb from the images folder, SCP it over to the device and install using rauc install *.raucb) and reboot. Clean build is not necessary, however, before each build, remove the whole target/lib/modules/ in the output folder folder, otherwise the image will bloat with modules from different Linux versions on the way.
  6. Check if Bluetooth works - if it does, run git bisect good, otherwise git bisect bad in your bisected kernel tree.
  7. Git shows another hash - goto 4.

You can try testing individual commits that you find suspicious an mark them as good/bad - e.g. git bisect bad 27d54b778ad1fb32c2c108cfe97e861c3909a46f once you test the commit I've mentioned before. If you aim well, you can reduce the number of iterations needed before converging to the offending commit.

Hope I haven't missed anything, if there's something unclear, let me know.

About the BT adapter - you should be able to see it from dmesg - search for "Bluetooth", there should be the driver and device model printed around.

Deckoz2302 commented 6 months ago

I also have this issue, and am using the Sena UD100-G03 adapter from the high performance Bluetooth list of recommended hardware with external Antenna. On a Home Assistant Blue.

would like to help If I can, but this issue persistent all the way up to OS 11.2

rhpijnacker commented 6 months ago

FYI: The Bluetooth adapters used are:

does this provide the requested information regarding the Bluetooth adapter used?

rhpijnacker commented 6 months ago

@sairon

take it and use it as BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE in the board config

I updated this in buildroot-external/configs/ova_defconfig and did an ova build. That failed with a 404 when it tried to download http://sources.buildroot.net/linux-8bb7eca972ad531c9b149c0a51ab43a417385813.tar.xz.

Is this the right place to change this?

rhpijnacker commented 6 months ago

@sairon

fastest is to take the .raucb from the images folder, SCP it over to the device and install using rauc install *.raucb

I'm trying with actual linux versions now (as a workaround). However, after a successful build I do not have any raucb files in the images folder. There are some in the build folder, but those are probably not the right ones.

sairon commented 6 months ago

That failed with a 404 when it tried to download http://sources.buildroot.net/linux-8bb7eca972ad531c9b149c0a51ab43a417385813.tar.xz.

Is this the right place to change this?

It is indeed, however I forgot to point out you need to make one more adjustment to tell Buildroot to use Git repository to obtain the sources. See the following diff:

diff --git a/buildroot-external/configs/ova_defconfig b/buildroot-external/configs/ova_defconfig
index b49d826d3..bcd2a764e 100644
--- a/buildroot-external/configs/ova_defconfig
+++ b/buildroot-external/configs/ova_defconfig
@@ -16,8 +16,9 @@ BR2_ROOTFS_POST_BUILD_SCRIPT="$(BR2_EXTERNAL_HASSOS_PATH)/scripts/post-build.sh"
 BR2_ROOTFS_POST_IMAGE_SCRIPT="$(BR2_EXTERNAL_HASSOS_PATH)/scripts/post-image.sh"
 BR2_ROOTFS_POST_SCRIPT_ARGS="$(BR2_EXTERNAL_HASSOS_PATH)/board/pc/ova $(BR2_EXTERNAL_HASSOS_PATH)/board/pc/ova/hassos-hook.sh"
 BR2_LINUX_KERNEL=y
-BR2_LINUX_KERNEL_CUSTOM_VERSION=y
-BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.1.69"
+BR2_LINUX_KERNEL_CUSTOM_GIT=y
+BR2_LINUX_KERNEL_CUSTOM_REPO_URL="git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git"
+BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="8bb7eca972ad531c9b149c0a51ab43a417385813"
 BR2_LINUX_KERNEL_DEFCONFIG="x86_64"
 BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="$(BR2_EXTERNAL_HASSOS_PATH)/kernel/hassos.config $(BR2_EXTERNAL_HASSOS_PATH)/kernel/docker.config $(BR2_EXTERNAL_HASSOS_PATH)/kernel/device-support.config $(BR2_EXTERNAL_HASSOS_PATH)/kernel/device-support-pcie.config $(BR2_EXTERNAL_HASSOS_PATH)/kernel/device-support-wireless.config $(BR2_EXTERNAL_HASSOS_PATH)/kernel/device-support-wireless-pci.config $(BR2_EXTERNAL_HASSOS_PATH)/board/pc/ova/kernel.config"
 BR2_LINUX_KERNEL_LZ4=y

However, after a successful build I do not have any raucb files in the images folder. There are some in the build folder, but those are probably not the right ones.

Every build should end with bunch of messages from RAUC, along with a message like this one:

(rauc:163451): rauc-DEBUG: 09:03:12.785: output bundle: /build/output_ova/images/haos_ova-11.4.dev0.raucb

If it's not there, then something else might be off.

rhpijnacker commented 6 months ago

It is indeed, however I forgot to point out you need to make one more adjustment to tell Buildroot to use Git repository to obtain the sources.

With the additional changes, I now get a finished build. Thanks for the additional pointers. It did take a lot longer to finish building it, though (which I guess can be expected, as it has to do a full kernel build).

However, I still do not get any raucb files. The command I'm using to build all this is:

scripts/enter.sh make O=output/ova ova

which gives me a bunch of outputs:

haos_ova-10.0.dev20240102.ova
haos_ova-10.0.dev20240102.qcow2.xz
haos_ova-10.0.dev20240102.vdi.zip
haos_ova-10.0.dev20240102.vhdx.zip
haos_ova-10.0.dev20240102.vmdk.zip

leading me to believe the build was ok. Just no raucb file.

sairon commented 6 months ago

Judging from the version identifier, you've got checked out some older version of HAOS. To get the RAUC update file, you need signing certs, which had to be supplied manually just until recent change (c3b9912e2e7acacf0c52bcc69976d84c3337a566). I assume the build also ends with some errors and non-zero return code, which would be another indicator something's wrong. I suggest to use the latest dev and start from there - or at least run the command to generate the self-signed certificate - after that you should have cert.pem and key.pem in the root of the repository. If nothing of that works, please share the last lines of the build output.

rhpijnacker commented 6 months ago

You are correct, I checked out a version near to where the bluetooth problems were first introducted. As far as I can tell, the build ends without any errors. I will try to build on the latest dev as suggested instead.

github-actions[bot] commented 3 months ago

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

andorardo commented 3 months ago

Oh well, it's a shame to have to give up on this device :-/

On Wed, Apr 3, 2024 at 1:40 AM github-actions[bot] @.***> wrote:

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/operating-system/issues/2961#issuecomment-2033573588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFGJJBWABMNO7P7SZ2IWTY3OI6PAVCNFSM6AAAAABAFAPP7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZTGU3TGNJYHA . You are receiving this because you commented.Message ID: @.***>

github-actions[bot] commented 1 week ago

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

andorardo commented 1 week ago

I'm finding this device still doesn't work (not reporting power usage) in 12.4.

On Wed, Jul 3, 2024, 1:40 AM github-actions[bot] @.***> wrote:

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/operating-system/issues/2961#issuecomment-2205139691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFGJOCCC7LOPYCIN7MXDLZKOFGDAVCNFSM6AAAAABAFAPP7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVGEZTSNRZGE . You are receiving this because you commented.Message ID: @.***>