Setting up Aranet devices broken with Bluetooth proxies that don't provide device names fails

[Jabe]

The problem

After updating to 2023.11.0 my Aranet 4 CO2 sensor stopped working. It is connected using an ESPHome Bluetooth Proxy. The problem still exists on latest 2023.11.2.

Trying to set up the integration again produces this error in the UI:

Config flow could not be loaded: 500 Internal Server Error Server got itself in trouble

Full log entry is attached.

What version of Home Assistant Core has the issue?

core-2023.11.2

What was the last working version of Home Assistant Core?

core-2023.10.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Aranet

Link to integration documentation on our website

https://www.home-assistant.io/integrations/aranet/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-11-16 10:56:32.786 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 94, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 31, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 63, in with_admin
    return await func(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 147, in post
    return await super().post(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
    result = await method(view, request, data, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 71, in post
    result = await self._flow_mgr.async_init(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 880, in async_init
    flow, result = await task
                   ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 908, in _async_init
    result = await self._async_handle_step(flow, flow.init_step, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 389, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/aranet/config_flow.py", line 99, in async_step_user
    adv = Aranet4Advertisement(
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aranet4/client.py", line 250, in __init__
    if device.name.startswith("Aranet2"):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

Additional information

No response

[home-assistant[bot]]

Hey there @aschmitz, @thecode, mind taking a look at this issue as it has been labeled with an integration (aranet) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `aranet` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign aranet` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

aranet documentation aranet source _{^{(message by IssueLinks)}}

[aschmitz]

That's interesting and certainly disappointing: do you know if you have other Bluetooth devices connected via an ESPHome proxy that are working?

Also, I've updated to HA 2023.11.0 and the Aranet4 is working for me, but I haven't actually updated my proxy to ESPHome 2023.11.x yet: do you know what version your ESPHome device is on? I'll try to update mine today and see if anything exciting happens.

Finally, do you know what errors you might have been getting when the device stopped working, and were they the same? Was it just that the CO2 sensor stopped and the others were working? (If so, recent changes to support the Aranet2 may have broken that, though it would be surprising: my Aranet4 is still fine on HA 2023.11.0.)

(@thecode may also have ideas.)

[Jabe]

ESPHome is now on 2023.11.1. If I remember correctly, I've kept ESPHome on 2023.10.x while upgrading HA. The sensor has been broken for at least 10 days and I didn't take notes unfortunately. I think that I only started updating ESPHome after I noticed the sensor was missing. I remember looking at the following ESPHome releases very carefully (and applying them) but they contained no obvious fixes. So I'm pretty sure I've had the issue with a combination of HA 2023.11.0 and ESPHome 2023.10.5~ish (I remember the single revert in .6 and not feeling very optimistic about my issue :smile:).

Home assistant simply reported the entities of the device as missing and then I deleted the device hoping to recreate it and it's entities.

Unfortunately I don't have another Bluetooth device to test.

@aschmitz What's your ESPHome version just out of interest? Maybe we can pin it down like that.

[thecode]

  File "/usr/local/lib/python3.11/site-packages/aranet4/client.py", line 250, in __init__
    if device.name.startswith("Aranet2"):
       ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

This may be caused if you are using a passive scanner, can you show your BT proxy scanner config?

[Jabe]

@thecode you are right, I was using

esp32_ble_trackers:
  scan_parameters:
    active: false

I switched it to active and it came back immediately. Sorry for the confusion. And thank you @thecode @aschmitz

Should we keep it active for the aranet devices? Maybe we can add a note to the docs? I would assume I switched it to passive after the device was connected a long time ago and now something has happened causing it to drop and forgetting about passive.

[aschmitz]

For what it's worth, I updated my Bluetooth proxy to ESPHome 2023.11.1 and Home Assistant 2023.11.0 and disabled active scanning in ESPHome:

esp32_ble_tracker:
  scan_parameters:
    active: False

bluetooth_proxy:
  active: False

This is still showing my Aranet4 values, including CO2 and temperature, successfully, so I'm really not sure what might have happened in your case. I notice you didn't actually specify a bluetooth_proxy configuration, so I'm not sure what options you have set there (I don't think it'll work without one? Maybe?), but either way I'm not really sure why it wasn't working for you. (I did not try removing and re-adding the device in production, as I kind of want the historical data. :confused:)

I do see an open PR in the Aranet4-Python library to fix the error you were running into when adding the device - I think that code got added to support the Aranet2. I'm not sure if that's really the right fix (it assumes any devices without a name being broadcast are an Aranet4, but hopefully there's some other way to identify them based on the data - any ideas, @thecode?), but if it does mean we broke adding a device from a passive scan, it would be nice to have a solution there.

(The ESPHome docs do recommend setting active: True for the Bluetooth proxy, so I expect most users will have that, but I wrote the integration before active Bluetooth proxying was available, and I would kind of like it to still work in that mode if possible.)

[thecode]

For what it's worth, I updated my Bluetooth proxy to ESPHome 2023.11.1 and Home Assistant 2023.11.0 and disabled active > This is still showing my Aranet4 values, including CO2 and temperature, successfully, so I'm really not sure what might have

Did you try to reload the integration after that?

I do see an open PR in the Aranet4-Python library to fix the error you were running into when adding the device - I think that code got added to support the Aranet2. I'm not sure if that's really the right fix (it assumes any devices without a name being broadcast are an Aranet4, but hopefully there's some other way to

That fix is wrong. Although the .startswith() was added to support Aranet2, without a device name the integration will fail anyhow and now it will make it assume the device is Aranet4. I know that because when trying to add a device before updating the library to see what it fails on, I used it on a dev machine without an active proxy and it failed in few places where we expect a device name.

The upstream library should be fixed to check the packet length instead of device name and after that we can change HA integration to generate a default name if it doesn't get a name from the BT scanner (or try another way to generate the name).

[aschmitz]

Did you try to reload the integration after that?

I didn't delete and re-add the device, but I did restart the VM running Home Assistant: it's still working. (This does seem slightly surprising to me, given that the code appears to be checking for the device name in any advertisement. I haven't dug into what is actually being passed back by ESPHome, but I did at least verify that none of my other ESPHome devices have Bluetooth proxies - or Bluetooth of any kind - enabled.)

That fix is wrong.

Yep, agreed. I'll continue over there, but I suspect this is a relatively easy thing to fix there. You're right that handling a name in HA may be tricky if we don't get one from the advertisement - I'm not sure what HA will do if we don't give it a name, but that may be a better behavior than trying to make one out of, say, the last bytes of the Bluetooth address.

[Jabe]

@aschmitz For completeness, my old config looked like this:

esp32_ble_tracker:
  scan_parameters:
    active: false

bluetooth_proxy:

It worked again after just setting scan_parameters.active to true.

The ESPHome docs make it sound like a good idea go to passive, provided it works:

active (Optional, boolean): Whether to actively send scan requests to request more data after having received an advertising packet. With some devices this is necessary to receive all data, but also drains those devices’ power a bit more. Some devices don’t need this, in that case you can save power and RF pollution by setting it to false. Defaults to true.

I believe I initially paired the devices in active mode, then set it to passive. A year later, obviously, I forgot about that and by happenstance the device got unpaired somehow, which I mistakenly correlated to the upgrade of HA/ESPHome.

[andrzej-r]

The issue is closed but it is not clear to me what is the solution it. Wait for a HA update? I am getting the same error 500.

My Home Assistant host does not have a Bluetooth adapter. I am using Shelly devices as Bluetooth proxies and the solution worked quite well until a week ago when I upgraded my HA docker image.

[Jabe]

Make sure your Bluetooth proxy scans in active BLE mode. Unfortunately, I don't know how to do that with Shelly.

For ESPHome keeping the config default is sufficient:

esp32_ble_tracker:

bluetooth_proxy:

That fixed it for me immediately.

[andrzej-r]

Thanks @Jabe. I don't use ESPHome and when I tried to add these directives to Home Assistant's configuration.yaml it returned these errors:

Integration error: bluetooth_proxy - Integration 'bluetooth_proxy' not found.
Integration error: esp32_ble_tracker - Integration 'esp32_ble_tracker' not found.

[Jabe]

@andrzej-r These options are only for ESPHome, which is a different technology. Have a look at your Shelly integration and click "Configure". You can configure the bluetooth proxy there. Maybe you can change it from passive to active and maybe that might work:

Otherwise I think you should open a new issue with logs attached.

[andrzej-r]

@Jabe, thanks. I already had my Shelly devices in the active scanner mode (this is how I was connecting my aranet4 device to Home Assistant server before).

The error message and the traceback in the logs are identical to yours, so I still think this is the same underlying issue with perhaps different solutions.

Logs:

``` 2023-11-29 16:07:20.056 ERROR (MainThread) [aiohttp.server] Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 227, in forwarded_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 31, in headers_middleware response = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle result = await handler(request, **request.match_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 63, in with_admin return await func(self, request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 147, in post return await super().post(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 71, in post result = await self._flow_mgr.async_init( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/config_entries.py", line 880, in async_init flow, result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/config_entries.py", line 908, in _async_init result = await self._async_handle_step(flow, flow.init_step, data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 389, in _async_handle_step result: FlowResult = await getattr(flow, method)(user_input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/aranet/config_flow.py", line 99, in async_step_user adv = Aranet4Advertisement( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aranet4/client.py", line 250, in __init__ if device.name.startswith("Aranet2"): ^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'startswith' ```

[aschmitz]

I'm going to reopen this because it's pretty clear there has been a regression in practice, and there are multiple ways to run into this issue. (Not to mention that we shouldn't throw 500 errors even if there's a workaround.) I don't think it's necessarily any particular person's fault, it just slipped into the gaps between the Aranet4-Python library's expectations and what Home Assistant provides, but we ought to be able to fix it.

There's a bit of discussion in https://github.com/Anrijs/Aranet4-Python/pull/35 about how we might be able to do that, and if I get some time this evening I'll try to submit a different PR there to update that code and make it not dependent on having a device name. Hopefully that'll be a quick fix and we can bump Aranet4-Python and everything will Just Work. If not, I guess we can explore alternative solutions.

(Of course, figuring out how to get device names back from Shelly Bluetooth proxies might be handy for other reasons, but that's well outside the scope of this issue.)

@home-assistant reopen @home-assistant rename Setting up Aranet devices broken with Bluetooth proxies that don't provide device names

[thecode]

Created an issue to track the problem upstream https://github.com/Anrijs/Aranet4-Python/issues/36