Conflicting bleak library version with HomeAssistant 2022.8

[carsonb]

It looks like 2022.8 has updated to bleak 0.15.0 https://github.com/home-assistant/core/pull/75941

When trying to use 4.0.0 of sensor.airthings_wave I get the following error in the logs:

Unable to install package bleak==0.14.3: ERROR: Cannot install bleak==0.14.3 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[ismarslomic]

Just upgraded HA to v2022.8.0 with already installed sensor.airthings_wave v4.0.0 and got same error statement in HA logs.

[lymanepp]

HA 2022.8 added native bluetooth support using the bleak library - https://www.home-assistant.io/blog/2022/08/03/release-20228/#first-class-bluetooth-support

I modified the manifest.json file to reference bleak 0.15.0, but it's still not working. Here's what the log shows.

2022-08-03 14:33:47.636 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.airthings_wave
2022-08-03 14:33:57.661 INFO (SyncWorker_5) [custom_components.airthings_wave.sensor] Found 1 airthings device(s)
2022-08-03 14:33:57.730 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform airthings_wave is taking over 10 seconds.
2022-08-03 14:33:57.749 INFO (SyncWorker_5) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13/
2022-08-03 14:33:57.749 ERROR (SyncWorker_5) [custom_components.airthings_wave.airthings] Not getting device info because failed to connect to device.
2022-08-03 14:33:57.793 INFO (SyncWorker_5) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13

[kongjudas]

Just a question: As the this integration is unsupported for the time being, will devs focus on the new BT integration, if even possible?

[lone-cloud]

bleak v0.15 is considered a potentially breaking release based on semserv versioning of a 0.x pre-release (aka unstable). I'm not a python dev, but it looks like poor debugging output as we don't actually know what the exception is here: https://github.com/custom-components/sensor.airthings_wave/blob/master/custom_components/airthings_wave/airthings.py#L245 I believe that anyone with the faintest knowledge of python should be able to easily solve this. If not, I'll work on this tomorrow, as I'd like to be on the latest release of HA with this integration working.

[lymanepp]

I added logging of the exception details:

This might be the root cause of the issue: got Future <Future pending> attached to a different loop

2022-08-03 21:44:38.041 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.airthings_wave
2022-08-03 21:44:48.062 INFO (SyncWorker_0) [custom_components.airthings_wave.sensor] Found 1 airthings device(s)
2022-08-03 21:44:48.157 INFO (SyncWorker_0) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13
Traceback (most recent call last):
  File "/config/custom_components/airthings_wave/airthings.py", line 239, in connect
    ret = await self._dev.connect()
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/client.py", line 107, in connect
    device = await BleakScannerBlueZDBus.find_device_by_address(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 221, in find_device_by_address
    return await cls.find_device_by_filter(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 250, in find_device_by_filter
    async with cls(detection_callback=apply_filter, **kwargs):
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 96, in __aenter__
    await self.start()
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/scanner.py", line 137, in start
    self._stop = await manager.active_scan(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/manager.py", line 376, in active_scan
    reply = await self._bus.call(
  File "/usr/local/lib/python3.10/site-packages/dbus_next/aio/message_bus.py", line 305, in call
    await future
RuntimeError: Task <Task pending name='Task-3555' coro=<AirthingsWaveDetect.get_info() running at /config/custom_components/airthings_wave/airthings.py:259> cb=[_run_until_complete_cb() at /usr/local/lib/python3.10/asyncio/base_events.py:184]> got Future <Future pending> attached to a different loop
2022-08-03 21:44:48.170 ERROR (SyncWorker_0) [custom_components.airthings_wave.airthings] Not getting device info because failed to connect to device.

[lone-cloud]

I've been debugging on a fork (https://github.com/lone-cloud/sensor.airthings_wave) and I'm getting the exact same error. I would definitely agree that it's the root cause of the issue here. The eception implies that there are multiple threads running and python doesn't seem to like when variables set in one thread get re-assigned in another. I have not been able to figure it out yet.

[B1ob]

In my opinion we should adapt this component to the new bluetooth integration. However i'm quite busy at the moment and also lacking the experience here.

[MartyTremblay]

Tons of hints on how to update to bleak 0.15.0 here https://github.com/home-assistant/core/commit/80a9659524a47868070a16cdfa7f654fba7f6bb7

[mjmccans]

While I have not done much digging yet, I wonder if this is potentially related to the issue addressed by flecmart's PR. I wonder that because I have tried the dev branches of my own related scripts (here and here) against Bleak 0.15.1 and they work without issue, but I did implement flecmart's changes to ensure that disconnect is always called. It may be worth looking into that angle.

[lone-cloud]

@mjmccans I don't understand how anything works for you because using bleak 0.15.1 will result in the same bleak version mismatch that we started out with. edit: I noticed that a new HA patch just came out (for me) which is on bleak 0.15.1. This custom component really needs to migrate to using the new bluetooth integration instead of calling bleak directly, as was suggested, otherwise we'll keep running into this issue.

flecmart's PR didn't solve the issue, but I noticed that it broke exception handling because he was raising exceptions that didn't originate from the base Exception class. You'll see a very unhelpful "exceptions must derive from BaseException" errors in the log without fixing the new raises.

I did look over https://github.com/home-assistant/core/commit/80a9659524a47868070a16cdfa7f654fba7f6bb7 but I couldn't find anything relevant to the issue here. Saying that, I'm a (mostly) typescript dev and this is my first time playing with python and with HA custom integration dev in general. I'm now at my wit's end about this got Future <Future pending> attached to a different loop error and someone more familiar with python+asyncio should probably take a look at this point.

[mjmccans]

@lone-cloud I will start with a disclaimer that I am not expert in this area, so please keep that in mind when reviewing anything I say here. I did the initial conversion from Bluepy to Bleak, but I definitely made mistakes along the way (some of which may not have been found yet). In terms of my scripts, one is a script that is meant to run on a machine separate from Home Assistant and it would have its own version of Bleak. The use case for me was to have the script running on a Raspberry Pi closer to the Airthings device which would in turn send the sensor values to Home Assistant using MQTT. The other is that same script packaged as a Home Assistant add-on, which I made when I transitioned to Home Assistant OS. The add-on runs in its own container and also has its own version of Bleak, and Home Assistant exposes the DBUS socket to the container for communications with the bluetooth adapter. What I find is interesting is that these scripts appear to have no issue running with their own copies of Bleak 0.15.1. The reason I thought flecmart's PR could be in play was the remark about this looking like a potential threading issue, and that was also the only difference between my version of airthings.py and the one in this repository.

I wonder if the issue is that the new bluetooth integration is not playing nice with this custom integration and that is causing the issue. Ultimately @B1ob is likely correct that the right answer is to move to the new bluetooth integration, but it would be interesting to see if the issue goes away if you uninstall or disable the new Home Assistant bluetooth integration. That could help debug this issue, or show that I am completely out to lunch.

[lone-cloud]

I just re-tested on the latest HA with bleak 0.15.1 and the new bluetooth integration removed. It appears that the issue is indeed because of the new bluetooth integration. Unfortunately, when it's disabled this component still fails to connect due to Event loop is closed... new error message though, yay. I've been trying to create and assign a new event loop via asyncio, but to no avail. HA somehow has control of it and I haven't been able to figure out how to get it working...

[mjmccans]

Well, I guess that is progress. Probably obvious, but this link has some details and best practices for using the new bluetooth integration. It is not urgent for me to dig into at the moment because my scripts are still working, but I am interested in the topic and will try to dive in in the future. From the documentation it actually looks like it would be pretty easy to get auto-discovery working, which would be pretty useful.

[flecmart]

I was trying to find some documentation about how to use the new bluetooh integration as a developer. Do we even need a custom component after integrating the airthing devices with the new integration?

[B1ob]

Do we even need a custom component after integrating the airthing devices with the new integration?

Yes, as i understand it the integrations rely on the new bluetooth integration as a kind of middleware.

As stated on the bluetooth integration page under "Integrations that require exclusive use of the Bluetooth Adapter" this does not work in parallel with legacy integrations. I also thought it could be possible to introduce "async" to the sensor.py. However i still think adapting to the new integration is the best approach.

[vincegio]

I've played around a with this I think we need to wait for https://github.com/home-assistant/core/pull/76342 to be merged as it adds polling possibilities to the new system.

Anyways, here's some of my progress :tada:

first time working on home assistant integrations :baby:

[lymanepp]

API changes coming in 2022.9 - https://developers.home-assistant.io/blog/2022/08/08/bluetooth-api-changes/

[hubertron]

Like the others I have the same issue. Actually updated from Feb update.

[DavidS]

I downgraded back to 20.22.7 for now. Thank $deity for backups :-D

[blackest]

Back to 2022.7.7 for me also (HA 2022.8.3 is using bleak 0.15.1) HA 2022.8.3 breaks more than just this component and it probably doesnt help i'm using python 3.9x as well.

Funny coincidence bleak 0.15.1 development was sponsered by nabucasa.com

[BladeBear]

https://github.com/mjmccans/hassio-addon-airthings, this is working with Airthings wave with current Home Assistant release.

[MartyTremblay]

Thanks for the info @BladeBear,

However, the project link you provided is an add-on and not an HA component which means that it doesn't depend on the same bleak instance as the one built into HA.

None the less, this MQTT approach clearly works and could be a viable solution until we get our component working again.

[lone-cloud]

I've gotten hard stuck on the new way to connect to BLE devices: BleakClient(bluetooth.async_ble_device_from_address(hass, mac)) The new docs say that the BleakClient must now be passed in a BLEDevice and the bluetooth.async_ble_device_from_address(hass, mac) resolves as expected, but when passing it into the BleakClient it just hangs forever for me with no debugable trace. Not sure if anyone's run into this issue. I searched through the hassio core integrations, but nowhere else do they try to use the BleakClient, other than the bluetooth integration.

[Jc2k]

HA integrations in core have to push some of their logic for talking to a device to a library on pypi. So what you'll probably find is stuff like this where the bledevice is passed to an external library and that is where the BleakClient lives.

There will also be logic in there to handle if the BLEDevice changes (via bluetooth.async_register_callback maybe). For example, in a future release you can have remote proxies (remote bluetooth dongles over usbip or EPSHome etc). AIUI, the BLEDevice might change as it self optimises the best proxy to use for each device.

What does the output from async_ble_device_from_address look like for you?

I don't know if it will help but make sure that your custom component depends on the bluetooth integration in its manifest.json. What does your logging config look like? And what bluetooth adapter are you using (we are having a haaard time with realtek atm).

[vincegio]

I started working on a draft to try and get this into core. https://github.com/vincegio/airthings-ble

That's inspired from the Bluetooth devices repos, but since there are no advertisements, it's closer to PySwitchbot. Got some minor help from bdraco where to find the docs and how to think around this.

It works and the core component is kind of done. But haven't been able to commit it to my fork due to pre-commit errors for an unrelated file. Anyways, that's irrelevant for this issue though :)

[lymanepp]

I started working on a draft to try and get this into core. https://github.com/vincegio/airthings-ble

Very clean!

[vincegio]

Thanks! Here's the commit in core. Still got some stuff to do but it works https://github.com/vincegio/core/commit/a07d07b5f268c73e6d7962b1088fca6edcbba3a1

[B1ob]

Hi @vincegio nice work :) i had a first look at your integration. My feedback is the following:

• There is a bug in _get_service_characteristics which is missing a "return device" statement.
• I would add the address (mac) to the enity_id as with multiple devices the id would not be unique anymore (Or ask the user during config flow for a unique name).

[vincegio]

Hi @vincegio nice work :) i had a first look at your integration. My feedback is the following:

• There is a bug in _get_service_characteristics which is missing a "return device" statement.
• I would add the address (mac) to the enity_id as with multiple devices the id would not be unique anymore (Or ask the user during config flow for a unique name).

Thank you :+1: The entity part makes sense, thought it'd just add _2 at the end or something.

[lone-cloud]

Does anyone know if there's a simple way to use vincegio's component before it gets merged into core? I'm guessing someone would need to create a new repo, copy and paste his code in and add that repo in HACS? Is that correct? Also @vincegio will your component support the battery % for the airthings mini? I'm using one if there's something I could help to test.

[vincegio]

Does anyone know if there's a simple way to use vincegio's component before it gets merged into core? I'm guessing someone would need to create a new repo, copy and paste his code in and add that repo in HACS? Is that correct? Also @vincegio will your component support the battery % for the airthings mini? I'm using one if there's something I could help to test.

I haven't posted the external library on pypi yet, so you would need to clone that too and install it locally. Otherwise put my files from core into custom_components i think. But planning on finalizing the pypi part at least tomorrow.

I'll add some config like battery voltage min-max options like this one has. I guess it will support mini as well?

[B1ob]

Does anyone know if there's a simple way to use vincegio's component before it gets merged into core? I'm guessing someone would need to create a new repo, copy and paste his code in and add that repo in HACS? Is that correct? Also @vincegio will your component support the battery % for the airthings mini? I'm using one if there's something I could help to test.

I did it as follows: ~~1. Add the bluetooth folder from the dev branch to the custom_components folder (https://github.com/home-assistant/core/tree/dev/homeassistant/components/bluetooth).

2. Add a version to the bluetooth manifest.json file (value is irrelevant - e.g.: "version": "0.1.0")~~
3. Add the airthings_ble to the custom_components folder (https://github.com/vincegio/core/tree/vincegio/airthings-ble/homeassistant/components/airthings_ble)
4. Add a version to the airthings_ble manifest.json file (value is irrelevant - e.g.: "version": "0.1.0")
5. Change the relative imports in airthings_ble/init.py from ...exceptions and ...helpers.update_coordinator to homeassistant.exceptions and homeassistant.helpers.update_coordinator. Do it similar in the sensor.py
6. When HA 2022.9 is released remove the bluetooth folder again.

[vincegio]

Actually pushed airthings-ble to pypi yesterday evening @B1ob, so just dump stuff into custom_components. Package should be installed automatically now. Good that you mentioned no.5, forgot to fix that yesterday :)

I wonder if I missed the deadline of getting stuff into 2022.9, but will try to make some tests tonight and see if I can create a MR asap.

[vincegio]

If it doesn't get added to 2022.9, I will just toss out a quick HACS repo as a temp. workaround I think.

[B1ob]

I have seen the package :) As i understand it if you add a custom_component (airthings_ble and bluetooth) it needs to have a version number. And i also thought the dev bluetooth integration is needed. So at least for now, these changes are needed.

By the way a feature request would be to have some explaination in the config flow for the configuration of the timeout.

[B1ob]

And i also thing the entity_id is currently not that favorably. For example "sensor.humidity" is quite generic. I would vote for something like sensor.airthings_wavehumidity or sensor.airthings_humidity. As with the mac and key it will be distinct.

[vincegio]

Did it revert to just sensor.humidity? It's been prefixed with the device name for me. But am looking into just adding mac to it, might as well.

[Jc2k]

If it's going into core the rules can be quite strict, but i'm happy to help. I've gotten a couple of integrations into core so far. If you link to the PR here I will review it. Will need someone else from HA to review it as well, but hopefully can get the ball rolling.

I already mentioned to @vincegio but there are rules around naming things. See https://developers.home-assistant.io/docs/core/entity/#entity-naming and https://developers.home-assistant.io/blog/2022/07/10/entity_naming/. If thats done right HA will automatically generate entity names and ids that are composed of the device name and the sensor name. I think all that is missing is a _attr_has_entity_name = True on the sensor class definition.

If i understand https://github.com/home-assistant/core/compare/dev...vincegio:core:vincegio/airthings-ble#diff-6f57e49b322caa61d2dbeb4f371bb47971d811b89b6c82997911e4ae859ba910R45 correctly - normally users can rename a config entry, but if you do this, they won't be able to any more, as it will keep resetting? Or did i miss something?

I think basing your unique_id and device identifiers off the device.name field will trigger a query from a reviewer. Both of these fields are meant to be immutable, and names can often be changed. MAC address is probably a safer choice here as the least likely to change. Or in homekit we use the serial number.

I think putting the scan interval in the config flow might be frowned upon. There are probably other ways to do this now, e.g. integrations are being pushed towards: https://www.home-assistant.io/blog/2021/06/02/release-20216/#disable-polling-updates-on-any-integration.

Theres some new commits landed you should be aware of:

• https://github.com/home-assistant/core/commit/3938015c93e4e9d3d7b3978117c35175643ec7f7 - we now have unfortunately connectable and unconnectable BLEDevices. Some users might use an EPS32 as a gateway. We can't use ESP32's for polling (yet) so that would be an unconnectable BLEDevice. async_ble_device_from_address should default to getting a connectable address so i don't think this is likely to be a problem for you.
• What is more of a problem is that you hold a BLEDevice forever. Because we now support multiple bluetooth dongles (included "mounting" remote ones with usbip) you might end up using a poor "route" if you do this. It's probably better to call async_ble_device_from_address before every poll.

[Jc2k]

Oh - and a couple of the existing BLE integrations did put the MAC address in the config flow title and device title, but they chopped off the first half (as its the vendor prefix and likely the same for every device for a given vendor - and it also makes the names lightly less ugly). So we'd have something like "Plant Sensor ab01cd" for a Xiaomi device.

[ismarslomic]

Never published integration to the core, but have been considering that! Are there any expectations from the core team in regards to maintenance of the new integrations? Like you as contributor is committed to actively maintain (and test) the integration for upcoming HA releases, otherwise integration will be excluded from the core? Or will the core team be responsible for maintenance?

[Jc2k]

The reality is somewhere in between. Let me give some examples.

You will get added to CODEOWNERS for your integration. So tickets will get assigned to you, and PR's will get assigned to you. You will be able review other peoples PR's to your integration. If you are given access you'll be able to merge other peoples PR's. You'll need someone else to review your own PR's though. CODEOWNER PR's get priority I should add, though I don't know if thats formalised or anything.

There are volunteers working to improve the quality of the codebase constantly. They are doing fairly thankless work updating the codebase to follow currenty best practices. Right now there is a lot of work on retrospectively increasing the mypy quality for example. They will send PR's to any integration they need to, and core maintainers will often approve them.

When something big like bluetooth multi dongle happens, the expectation is the work doesn't break existing integrations. So you might find a new feature is added to your code, or you might find you are running in some sort of compat mode.

But you aren't going to (on the whole) get free bug fixes out of core team members, unless they have a device that your integration unlocks and they are really excited about supporting something new.

Annnd... broken integrations will get removed. Sometimes it might take a few releases, but if tickets are piling up and no one is answering it will get kicked out. Theres no strict procedure for this, so if its popular and works for most people it might not happen right away.

Have to say that the code review process is kind, but hard work and sometimes slow (because it is so thorough). I wouldn't have learned HA as well as I did without the excellent code review that HA does. And whilst it was hard sometimes, I would never not aim to get integrations in core from the start.

[B1ob]

I have no problem with the current naming of the entity. I think these can have duplicates. But i have a problem with the id, as it should be unique. (Name: Humidity current entity_id: sensor.humidity) If you would add two Airthings devices they would end up both with the same entity_id, which would probably run into an issue.

[vincegio]

Thanks for the feedback @Jc2k

I already mentioned to @vincegio but there are rules around naming things. See https://developers.home-assistant.io/docs/core/entity/#entity-naming and https://developers.home-assistant.io/blog/2022/07/10/entity_naming/. If thats done right HA will automatically generate entity names and ids that are composed of the device name and the sensor name. I think all that is missing is a _attr_has_entity_name = True on the sensor class definition.

I knew I missed something :+1:

home-assistant/core@dev...vincegio:core:vincegio/airthings-blediff-6f57e49b322caa61d2dbeb4f371bb47971d811b89b6c82997911e4ae859ba910R45

That link doesn't work for me, getting "There isn’t anything to compare."

Thanks for the information regarding polling and the BLEDevice. Will also certainly look into manipulating the mac address and get it into the name. How much unit testing is needed? I see that some components just test the flow, and that's it.

Would be nice to get this in a PR as soon as I fix the most "critical" things you guys have mentioned :)

[kongjudas]

I have tried the solutions mentioned here, but only end up with Setup failed for custom integration airthings_ble: No setup or config entry setup function defined.

A temporary HACS integration would be much appreciated for now, just to get the HA thermostats running again. And: you guys are awesome!

[Jc2k]

@B1ob HA won't assign duplicate entity ids, so don't worry. And the documentation i linked to talks about the name, but its important to understand that the entity id is based on the name. So if you comply with the entity name rules, you are complying with the entity id rules, and you wont have the problems you are worried about.

The most important thing to HA is that the "unique id" field is unique, and that the device identifiers are unique. And that they are stable/immutable. Otherwise you'll start getting duplicate entries in the device and entity registries.

[Jc2k]

@vincegio i think it was meant to be to here: https://github.com/vincegio/core/blob/d17a66dcc4ead18533e02161b897f568f707f5e8/homeassistant/components/airthings_ble/__init__.py#L46-L49. Does that work any better?

Unittesting - there is a mandate that config flows have 100% coverage. The rules around everything else are less strict, but i'd probably want to see at least a test that a sensor could be created and its value could be updated.

[B1ob]

I have tried the solutions mentioned here, but only end up with Setup failed for custom integration airthings_ble: No setup or config entry setup function defined.

A temporary HACS integration would be much appreciated for now, just to get the HA thermostats running again. And: you guys are awesome!

maybe you increase the debug output. Then the error message will be more clear.

Add this to your configuration.yaml:

logger:
  default: error
  logs:
    custom_components.airthings_ble: debug

[vincegio]

@vincegio i think it was meant to be to here: https://github.com/vincegio/core/blob/d17a66dcc4ead18533e02161b897f568f707f5e8/homeassistant/components/airthings_ble/__init__.py#L46-L49. Does that work any better?

Unittesting - there is a mandate that config flows have 100% coverage. The rules around everything else are less strict, but i'd probably want to see at least a test that a sensor could be created and its value could be updated.

That works better. The reason for this logic is that the device doesn't broadcast it's name (defaults to mac) until it has been connected. So if it is not the same - update. Maybe there's a better way to connect and get it's name prior to this logic? Otherwise the entry name will only be it's address.. Maybe that's OK tho?

[Jc2k]

So some integrations actually query the device as part of the config flow, right before they create the entry. So your config flow is allowed to connect to the device, query its name characteristic and disconnect, then create the config entry with the correct title.

[ismarslomic]

Does anyone know if there's a simple way to use vincegio's component before it gets merged into core? I'm guessing someone would need to create a new repo, copy and paste his code in and add that repo in HACS? Is that correct? Also @vincegio will your component support the battery % for the airthings mini? I'm using one if there's something I could help to test.

I did it as follows:

1. Add the bluetooth folder from the dev branch to the custom_components folder (https://github.com/home-assistant/core/tree/dev/homeassistant/components/bluetooth).

2. Add a version to the bluetooth manifest.json file (value is irrelevant - e.g.: "version": "0.1.0")

3. Add the airthings_ble to the custom_components folder (https://github.com/vincegio/core/tree/vincegio/airthings-ble/homeassistant/components/airthings_ble)

4. Add a version to the airthings_ble manifest.json file (value is irrelevant - e.g.: "version": "0.1.0")

5. Change the relative imports in airthings_ble/**init**.py from ...exceptions and ...helpers.update_coordinator  to homeassistant.exceptions and homeassistant.helpers.update_coordinator. Do it similar in the sensor.py

6. When HA 2022.9 is released remove the bluetooth folder again.

I followed these steps and managed to successfully install the airthings_ble component and detect my Airthings Wave+ device. The only exception I needed to do was to skip step 1 and 2. HA complained about conflicting version of bluetooth-adapter and that it couldn't find the requirement bluetooth-adapter 0.2.0 in the bluetooth component (from dev branch).

This looks really promising, thanks alot @vincegio !