Bluetooth proxy doesn't work (well) with 2022.12

[Ulrar]

The problem

I've updated this morning to 2022.12 and noticed that all of my bluetooth devices failed to show up in home assistant. I only use bluetooth_proxy over 4 esp32 relay boards.

I've tried a few things like changing from arduino to esp-idf and back, I also flashed each through serial (using edge) to make sure the partition table would be done but no change.

I don't really have any more info unfortunately, I didn't see any errors anywhere. Home Assistant just does not see the announces of my switchbot curtains or airthings devices, and keep retrying their setup over and over.

I do have a switchbot contact sensor that did not fail to setup, but it also never sent any new values. That may be normal. That's the only device I have that does not depend on active: true, so it's possible that the issue is only with active devices. I don't have other passive devices here to make sure, sadly.

I downgraded to 2022.11 and everything is back to normal, so it must be related to the bluetooth_proxy change in 2022.12 I imagine

Which version of ESPHome has the issue?

2022.12

What type of installation are you using?

Docker

Which version of Home Assistant has the issue?

No response

What platform are you using?

ESP32

Board

doit-devkit-v1 and olimex's esp32-poe, seem to affect both

Component causing the issue

bluetooth_proxy

Example YAML snippet

bluetooth_proxy:
  active: true

### Anything in the logs that might be useful for us?

```txt
Nothing in the logs

Additional information

I have a lot of Switchbot Curtains, and one Airthings Wave+. The curtains and airthings on the side of the house I updated on all failed to appear in HA after the update.

I kept the other side of the house on 2022.11, at the same time, and the curtains on that side kept working, so it's not a HA issue. Downgrading all the boards back to 2022.11 fixed everything.

[PetrMa]

I have similar issue, Rollback back to 2022.11.5 solved the issue.

[txitxo0]

Same here. After updating to the latest esphome version in a esp32 board my ATC thermometers started to don't report any activity

[cpuks]

Same here with LYWSDCGQ and LYWSD02 and ATC - although esp32 is literally a meter away there's no signal, this has been working stable for months now. Reverted back to 2022.11.5 solved issues.

[bdraco]

Which version of Home Assistant are you using?

Also is there anything useful in the log?

[bdraco]

If there isn't anything useful in the log you may need to recompile with logging to to VERBOSE

[bdraco]

Also please post your complete yaml (not partial)

[bdraco]

None of the devices mentioned should need an active connection to setup, only be controlled so it seems like a problem with advertisement parsing it delivery

There was a fix for parsing zero padding advertisements in 2022.12.0 but I don't see how that would have caused this issue as all my SwitchBot devices are working as expected with proxies.

Please post diagnostics for esphome from HA 2022.12.6 or later as well as it will tell us if advertising data is being received by core

[txitxo0]

Hi again, I am running the latest HAOSS with Home Assistant 2022.12.6, sadly there is nothing useful in the log.

The yaml I have in my bt proxy is:

substitutions:
  name: "bluetooth-proxy-1"
esp32:
  board: esp32dev
  framework:
    type: arduino
packages:
  esphome.bluetooth-proxy: github://esphome/bluetooth-proxies/esp32-generic.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

api:
logger:
  level: VERBOSE
ota:
# improv_serial:
captive_portal:

web_server:
  port: 80

binary_sensor:
  - platform: gpio
    pin: GPIO26
    name: "Desktop motion"
    device_class: motion

sensor:
  - platform: adc
    pin: GPIO32
    name: "Desktop illuminance"
    device_class: illuminance
    unit_of_measurement: lx
    update_interval: 60s
    attenuation: auto
    filters:
      - lambda: |-
          return (x / 10000.0) * 2000000.0 * 1.67;   

For the verbose log, I have just updated to the last version esphome ask me for in the proxy and set logs in Verbose mode:

INFO Starting log output from bluetooth-proxy-1.local using esphome API
INFO Successfully connected to bluetooth-proxy-1.local
[08:51:55][I][app:102]: ESPHome version 2022.12.1 compiled on Dec 16 2022, 08:49:03
[08:51:55][I][app:104]: Project esphome.bluetooth-proxy version 1.0
[08:51:55][C][wifi:504]: WiFi:
[08:51:55][C][wifi:362]:   Local MAC: E0:5A:1B:75:8A:90
[08:51:55][C][wifi:363]:   SSID: 'disconnected'[redacted]
[08:51:55][C][wifi:364]:   IP Address: 192.168.3.50
[08:51:55][C][wifi:366]:   BSSID: 18:D9:8F:1F:06:80[redacted]
[08:51:55][C][wifi:367]:   Hostname: 'bluetooth-proxy-1'
[08:51:55][C][wifi:369]:   Signal strength: -60 dB ▂▄▆█
[08:51:55][V][wifi:371]:   Priority: 0.0
[08:51:55][C][wifi:373]:   Channel: 6
[08:51:55][C][wifi:374]:   Subnet: 255.255.255.0
[08:51:55][C][wifi:375]:   Gateway: 192.168.3.1
[08:51:55][C][wifi:376]:   DNS1: 192.168.3.1
[08:51:55][C][wifi:377]:   DNS2: 0.0.0.0
[08:51:55][V][api.connection:900]: Hello from client: 'Home Assistant 2022.12.6 (::FFFF:192.168.3.81)' | API Version 1.7
[08:51:55][C][logger:293]: Logger:
[08:51:55][C][gpio.binary_sensor:015]: GPIO Binary Sensor 'Desktop motion'
[08:51:55][C][gpio.binary_sensor:015]:   Device Class: 'motion'
[08:51:55][C][gpio.binary_sensor:016]:   Pin: GPIO26
[08:51:55][D][api.connection:918]: Home Assistant 2022.12.6 (::FFFF:192.168.3.81): Connected successfully
[08:51:55][C][bluetooth_proxy:065]: Bluetooth Proxy:
[08:51:55][C][bluetooth_proxy:066]:   Active: YES
[08:51:55][C][safe_mode.button:022]: Safe Mode Button 'Safe Mode Boot'
[08:51:55][C][safe_mode.button:022]:   Icon: 'mdi:restart-alert'
[08:51:55][C][adc:087]: ADC Sensor 'Desktop illuminance'
[08:51:55][C][adc:087]:   Device Class: 'illuminance'
[08:51:55][C][adc:087]:   State Class: 'measurement'
[08:51:55][C][adc:087]:   Unit of Measurement: 'lx'
[08:51:55][C][adc:087]:   Accuracy Decimals: 2
[08:51:55][C][adc:097]:   Pin: GPIO32
[08:51:55][C][adc:099]:  Attenuation: auto
[08:51:55][C][adc:126]:   Update Interval: 60.0s
[08:51:55][C][esp32_ble_tracker:874]: BLE Tracker:
[08:51:55][C][esp32_ble_tracker:875]:   Scan Duration: 300 s
[08:51:55][C][esp32_ble_tracker:876]:   Scan Interval: 1100.0 ms
[08:51:55][C][esp32_ble_tracker:877]:   Scan Window: 1100.0 ms
[08:51:55][C][esp32_ble_tracker:878]:   Scan Type: ACTIVE
[08:51:55][C][esp32_ble_tracker:879]:   Continuous Scanning: True
[08:51:56][C][captive_portal:088]: Captive Portal:
[08:51:56][C][web_server:125]: Web Server:
[08:51:56][C][web_server:126]:   Address: bluetooth-proxy-1.local:80
[08:51:56][C][mdns:103]: mDNS:
[08:51:56][C][mdns:104]:   Hostname: bluetooth-proxy-1
[08:51:56][V][mdns:105]:   Services:
[08:51:56][V][mdns:107]:   - _esphomelib, _tcp, 6053
[08:51:56][V][mdns:109]:     TXT: version = 2022.12.1
[08:51:56][V][mdns:109]:     TXT: mac = e05a1b758a90
[08:51:56][V][mdns:109]:     TXT: platform = ESP32
[08:51:56][V][mdns:109]:     TXT: board = esp32dev
[08:51:56][V][mdns:109]:     TXT: network = wifi
[08:51:56][V][mdns:109]:     TXT: project_name = esphome.bluetooth-proxy
[08:51:56][V][mdns:109]:     TXT: project_version = 1.0
[08:51:56][V][mdns:109]:     TXT: package_import_url = github://esphome/bluetooth-proxies/esp32-generic.yaml@main
[08:51:56][V][mdns:107]:   - _http, _tcp, 80
[08:51:56][C][ota:093]: Over-The-Air Updates:
[08:51:56][C][ota:094]:   Address: bluetooth-proxy-1.local:3232
[08:51:56][C][api:138]: API Server:
[08:51:56][C][api:139]:   Address: bluetooth-proxy-1.local:6053
[08:51:56][C][api:143]:   Using noise encryption: NO
[08:51:56][C][improv_serial:032]: Improv Serial:
[08:51:56][V][bluetooth_proxy:074]: [0] Free connection
[08:51:56][V][bluetooth_proxy:074]: [1] Free connection
[08:51:56][V][bluetooth_proxy:074]: [2] Free connection
[08:51:56][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:51:56][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:51:57][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:51:57][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:51:57][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:51:57][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -82 dB
[08:51:58][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:51:58][V][bluetooth_proxy:033]: Proxying packet from  - 62:16:D6:A1:66:95. RSSI: -83 dB
[08:51:58][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -76 dB
[08:51:59][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:51:59][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:00][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:00][V][bluetooth_proxy:033]: Proxying packet from  - 7A:75:27:DA:04:88. RSSI: -84 dB
[08:52:00][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:01][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:52:01][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -82 dB
[08:52:02][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:52:02][V][bluetooth_proxy:033]: Proxying packet from ATC_BECE9B - A4:C1:38:BE:CE:9B. RSSI: -70 dB
[08:52:02][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -74 dB
[08:52:02][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:03][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:03][V][bluetooth_proxy:033]: Proxying packet from  - 62:16:D6:A1:66:95. RSSI: -85 dB
[08:52:03][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:03][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -78 dB
[08:52:03][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:05][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -71 dB
[08:52:05][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:05][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -74 dB
[08:52:06][V][bluetooth_proxy:033]: Proxying packet from ATC_BECE9B - A4:C1:38:BE:CE:9B. RSSI: -76 dB
[08:52:06][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:52:06][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -91 dB
[08:52:07][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:07][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -78 dB
[08:52:07][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:08][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:08][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:08][V][bluetooth_proxy:033]: Proxying packet from  - 7A:75:27:DA:04:88. RSSI: -87 dB
[08:52:09][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:09][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:09][V][bluetooth_proxy:033]: Proxying packet from ATC_162894 - A4:C1:38:16:28:94. RSSI: -37 dB
[08:52:10][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -75 dB
[08:52:11][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -73 dB
[08:52:11][D][binary_sensor:036]: 'Desktop motion': Sending state ON
[08:52:11][V][json:031]: Attempting to allocate 512 bytes for JSON serialization
[08:52:11][V][json:051]: Size after shrink 80 bytes
[08:52:11][V][bluetooth_proxy:033]: Proxying packet from ATC_162894 - A4:C1:38:16:28:94. RSSI: -35 dB
[08:52:12][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -72 dB
[08:52:12][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -79 dB
[08:52:12][D][binary_sensor:036]: 'Desktop motion': Sending state OFF
[08:52:12][V][json:031]: Attempting to allocate 512 bytes for JSON serialization
[08:52:12][V][json:051]: Size after shrink 80 bytes
[08:52:13][V][bluetooth_proxy:033]: Proxying packet from  - 59:9B:D2:E9:80:B5. RSSI: -77 dB
[08:52:13][V][bluetooth_proxy:033]: Proxying packet from Mi Smart Band 5 - D5:67:0B:FB:5E:CB. RSSI: -83 dB
[08:52:13][V][bluetooth_proxy:033]: Proxying packet from  - 44:D1:81:D0:E6:66. RSSI: -81 dB
[08:52:13][V][bluetooth_proxy:033]: Proxying packet from ATC_162894 - A4:C1:38:16:28:94. RSSI: -36 dB

One of the atc thermometers is A4:C1:38:BE:CE:9B who is already proxying apparently from the log, but home assistant didn't receive any new packet from 19:10, stuck on the "last_packet_id" property 190

[bdraco]

Are you still getting "Proxying packet from" and HA stops updating or does the "Proxying packet from" logs stop when the device stops updating?

[cpuks]

The difference for me is I switched to esp-idf framework to get those goodies that 2022.12 brings - previously (up till 2022.11.5) was using arduino framework. Now I reverted to 2022.11.5 but kept esp-idf and it's all good. Need a bit of time to get one of spare esp32 up and running on 2022.12 to get logs. Also my is HA 2022.12.6

[txitxo0]

Are you still getting "Proxying packet from" and HA stops updating or does the "Proxying packet from" logs stop when the device stops updating?

It is still proxying: [09:39:13][V][bluetooth_proxy:033]: Proxying packet from ATC_BECE9B - A4:C1:38:BE:CE:9B. RSSI: -68 dB and HA has stopped updates

[bdraco]

If you turn on debug logs for aioesphomeapi on HA do you see them it making over the connection?

Also the diagnostics for esphome in HA should show all the advertisements

[txitxo0]

If you turn on debug logs for aioesphomeapi on HA do you see them it making over the connection?

Also the diagnostics for esphome in HA should show all the advertisements

Apparently at 8:51 HA says saomthing about a problem connecting with this esp device:

2022-12-16 08:51:55.393 WARNING (MainThread) [aioesphomeapi.reconnect_logic] Can't connect to ESPHome API for bluetooth-proxy-1 @ 192.168.3.50: Hello timed out

But I guess this was just temporary, I got this same message for all the esphome devices but now ther are all working via Home Assistant

[PetrMa]

Hello,

in my case it seems that BT Proxy works fine if I changed arduino to esp-idf after upgrade to 2022.12.x but I have a problem with ESP32 devices which contains configuration for BT Proxy and Camera together. Camera does not support esp-idf so I must use arduino and there is still this issue https://github.com/esphome/issues/issues/3907

installation of 2022.12.x succeed but the device stopped responding on WiFi IP addreess few seconds after boot. If I remove proxy settings from configuration, esp32 camera works fine.

[PetrMa]

I found some issue in verbose log.

When the device stopped working on network, this error was presented in verbose log:

[V][api.connection:900]: Hello from client: 'Home Assistant 2022.12.6 (::FFFF:192.168.1.2)' | API Version 1.7 [D][api.connection:918]: Home Assistant 2022.12.6 (::FFFF:192.168.1.2): Connected successfully [D][esp32_camera:172]: Got Image: len=47492 [D][esp32_camera:172]: Got Image: len=89102 [D][esp32_camera:172]: Got Image: len=88054 [D][esp32_camera:172]: Got Image: len=88552 [D][esp32_camera:172]: Got Image: len=87671 [V][sensor:076]: 'Kamera předsíň Uptime': Received new state 45.730000 [D][sensor:127]: 'Kamera předsíň Uptime': Sending state 45.73000 s with 0 decimals of accuracy [V][api.connection:992]: Cannot send message because of TCP buffer space [V][api.connection:992]: Cannot send message because of TCP buffer space [V][api:114]: Removing connection to Home Assistant 2022.12.6 (::FFFF:192.168.1.2)

If I remove these lines from config, camera and wifi works fine:

bluetooth_proxy: active: true

esp32_ble_tracker: scan_parameters: interval: 1100ms window: 1100ms active: true

It seems there is some issue with TCP buffer when BT is used with arduino

[txitxo0]

for me it is happening the same being arduino or esp-idf framework

[bdraco]

[V][api.connection:992]: Cannot send message because of TCP buffer space
[V][api.connection:992]: Cannot send message because of TCP buffer space

This usually indicates you are running out of ram

You can add an uptime and ram sensor to get some more visibility:

# Sensors with general information.
sensor:
  # Uptime sensor.
  - platform: uptime
    name: upstairsdesk89proxy Uptime
  - platform: template
    name: upstairsdesk89proxy Free Memory
    lambda: return heap_caps_get_free_size(MALLOC_CAP_INTERNAL);
    unit_of_measurement: 'B'
    state_class: measurement

[cpuks]

Sorry I din't set up my test env yet but I'm only using my esp32 for one purpose - bluetooth proxy and my config below:

substitutions:
  node_name: ble-proxy-wiata
  device_name: "bluetooth LE proxy HA wiata"

esphome:
  name: $node_name
  comment: $device_name

esp32:
  board: esp-wrover-kit
  framework:
    type: esp-idf
    #type: arduino

logger:

api:

ota:
  password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: !secret wifi_fallback_ssid
    password: !secret wifi_fallback_password

# bluetooth proxies
esp32_ble_tracker:
  scan_parameters:
    interval: 1100ms
    window: 1100ms
bluetooth_proxy:
  active: true

[bdraco]

It looks like you are using this board

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/hw-reference/esp32/get-started-wrover-kit.html#get-started-esp-wrover-kit-v4-1-board-front

I'll try flashing one of those with your config later today

[bdraco]

I thought I had one of those boards but it looks like I need to order one. Should hopefully make it in before the new year

[cpuks]

Yes, that's photo of board I'm using three of them in different locations, up till 2022.11.5 I had arduino framework, now esp-idf

Should be fine on esp32dev too?

[bdraco]

In the mean time, would please setup the memory sensor?

https://github.com/esphome/issues/issues/3913#issuecomment-1355258621

[bdraco]

It seems there is some issue with TCP buffer when BT is used with arduino

arduino uses about 50000 more bytes of ram so its not surprising

[Ulrar]

I did try both arduino and esp-idf, and like above I also have relay boards that don't do anything but bluetooth proxy. I mostly use doit devkits v1, I'll have to try upgrading them again today and add that memory sensor

[txitxo0]

I have already add these sensors (uptime and free memory) I just changed the unit of memory mesurement to kB to be easier to read (just addin /1024 to the lambda) and here are my results:

substitutions:
  name: "bluetooth-proxy-1"
esp32:
  board: esp32dev
  framework:
    type: arduino #esp-idf
packages:
  esphome.bluetooth-proxy: github://esphome/bluetooth-proxies/esp32-generic.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

api:
logger:
  # level: VERBOSE
ota:
captive_portal:

web_server:
  port: 80

binary_sensor:
  - platform: gpio
    pin: GPIO26
    name: "Desktop motion"
    device_class: motion

sensor:
  - platform: adc
    pin: GPIO32
    name: "Desktop illuminance"
    device_class: illuminance
    unit_of_measurement: lx
    update_interval: 60s
    attenuation: auto
    filters:
      - lambda: |-
          return (x / 10000.0) * 2000000.0 * 1.67;

    # Uptime sensor
  - platform: uptime
    name: btproxy Uptime
  - platform: template
    name: btproxy Free Memory
    lambda: return heap_caps_get_free_size(MALLOC_CAP_INTERNAL) / 1024;
    unit_of_measurement: 'kB'
    state_class: measurement

And having in consideration with the previous ESPhome version it didn't happen

[cpuks]

On 2022.12 I've also turned on active mode - in past I only had passive as I didn't have use case scenario up till snooz integration. And now even on 2022.11.5 there are two devices that regularly out despite being 1 m away from proxy nodes (two different rooms two separate esp32 nodes)

[Ulrar]

So I've just tried it again with the latest version and VERBOSE log mode, and I don't see anything from the ESP's log. Usually on startup there's a connection made to the AirThings, but with the latest esphome release it just doesn't do that and HA fails to setup the integration saying it's not advertising. Again that's only with 2022.12, works fine with 2022.11.

I setup the memory sensor and it's still above 100k of free memory so I don't think that's the issue, I suspect we're not all having the same problem here. What else can I do to help diagnose this ? It looks like whatever it is doesn't show up on the ESP side, do I need to enable some logs on the HA side somehow ?

[bdraco]

Thats definitely at risk of running out of ram. I would use esp-idf instead if possible

[bdraco]

So I've just tried it again with the latest version and VERBOSE log mode, and I don't see anything from the ESP's log.

Usually on startup there's a connection made to the AirThings, but with the latest esphome release it just doesn't do that and HA fails to setup the integration saying it's not advertising.

Again that's only with 2022.12, works fine with 2022.11.

Can you post diagnostics from from Home Assistant with it flashed to both versions so we can compare the advertisement data.

I setup the memory sensor and it's still above 100k of free memory so I don't think that's the issue, I suspect we're not all having the same problem here.

Looks like there are multiple different issues in this thread 😬

What else can I do to help diagnose this ? It looks like whatever it is doesn't show up on the ESP side, do I need to enable some logs on the HA side somehow ?

[txitxo0]

Thats definitely at risk of running out of ram. I would use esp-idf instead if possible

I will use esp-idf, but it didn't fix the problem and.... Why was it running successfully before the new version?

[bdraco]

Thats definitely at risk of running out of ram. I would use esp-idf instead if possible

I will use esp-idf, but it didn't fix the problem and.... Why was it running successfully before the new version?

Nobody knows why yet.

[Ulrar]

Yeah there's definitely something non ram related going on. I'll post diagnostics when I get home later

[txitxo0]

I am using now esp-idf and no using webserver, I win ~117kB

I will track the behaviour in this way then ;)

[Ulrar]

Okay there we go, there is a difference in the diagnostics. This is a Switchbot Curtain device that is failing to setup in HA when the board is running 2022.12, but succeeding as soon as I use 2022.11. That's from two different boards right now, I have one running each to make this easier, but I've already confirmed that the behavior is the same if using only one board.

2022.11 :

          {
            "name": "",
            "address": "FA:62:0F:CF:2D:F2",
            "rssi": -64,
            "advertisement_data": [
              null,
              {
                "2409": {
                  "__type": "<class 'bytes'>",
                  "repr": "b'\\xfab\\x0f\\xcf-\\xf2-\\x1fd\\x12\\x04'"
                }
              },
              {},
              [],
              -127,
              -64,
              []
            ],
            "details": {
              "source": "esp32-relay-livingroom",
              "connector": {
                "__type": "<class 'homeassistant.components.bluetooth.models.HaBluetoothConnector'>",
                "repr": "HaBluetoothConnector(client=<class 'homeassistant.components.esphome.bluetooth.client.ESPHomeClient'>, source='esp32-relay-livingroom', can_connect=<function _async_can_connect_factory.<locals>._async_can_connect at 0x7fec353c8310>)"
              },
              "address_type": 1
            }
          },

2022.12 :

          {
            "name": "",
            "address": "FA:62:0F:CF:2D:F2",
            "rssi": -58,
            "advertisement_data": [
              null,
              {
                "2409": {
                  "__type": "<class 'bytes'>",
                  "repr": "b'\\xfab\\x0f\\xcf-\\xf20\\x0fd\\x12\\x04'"
                }
              },
              {
                "0000fd3d-0000-1000-8000-00805f9b34fb": {
                  "__type": "<class 'bytes'>",
                  "repr": "b'c\\xc0Vd\\x12\\x04'"
                }
              },
              [],
              -127,
              -58,
              []
            ],
            "details": {
              "source": "esp32-hallway-sensor",
              "connector": {
                "__type": "<class 'homeassistant.components.bluetooth.models.HaBluetoothConnector'>",
                "repr": "HaBluetoothConnector(client=<class 'homeassistant.components.esphome.bluetooth.client.ESPHomeClient'>, source='esp32-hallway-sensor', can_connect=<function _async_can_connect_factory.<locals>._async_can_connect at 0x7fec2d0fb6d0>)"
              },
              "address_type": 0
            }
          },

That second object (looks like the service used to send the Switchbot orders) present in .11 is missing in .12. So presumably the problem is coming from that new parsing code, it's now failing to parse the service ?

[bdraco]

The ServiceData being missing will definitely cause the issue you are reporting. Thats good as we are getting close to the cause here.

Can you try installing from dev with https://github.com/esphome/esphome/pull/4162 reverted to see if it fixes it?

[txitxo0]

Is it possible to choose the version you want to run as ha addon? I guess I can't choose the version to use

[Ulrar]

@bdraco I imagine that's not as simple as using the :dev docker image ? If there's no existing docker image for that I can try to build it from git later tonight

[cpuks]

Is it possible to choose the version you want to run as ha addon? I guess I can't choose the version to use

Nope, if you want to downgrade or change versions either portainer or console.

[txitxo0]

Is it possible to choose the version you want to run as ha addon? I guess I can't choose the version to use

Nope, if you want to downgrade or change versions either portioner or console.

Thank you,

I am using now HAOS instead of a Raspbian with docker. So, I would try with portairner instead. I'll try it

[bdraco]

@bdraco I imagine that's not as simple as using the :dev docker image ? If there's no existing docker image for that I can try to build it from git later tonight

git clone https://github.com/esphome/esphome
cd esphome
git revert 0c24d951ff07bccd982443733ecce0675e93833d
pip3 install --upgrade . -vvvv

cd ~/YOUR_YAML_FILES
esphome compile your_yaml.yaml
esphome upload your_yaml.yaml

[bdraco]

Also missing ServiceData can also be caused by the scanner being set to passive instead of active.

[PetrMa]

[V][api.connection:992]: Cannot send message because of TCP buffer space
[V][api.connection:992]: Cannot send message because of TCP buffer space

This usually indicates you are running out of ram

You can add an uptime and ram sensor to get some more visibility:

# Sensors with general information.
sensor:
  # Uptime sensor.
  - platform: uptime
    name: upstairsdesk89proxy Uptime
  - platform: template
    name: upstairsdesk89proxy Free Memory
    lambda: return heap_caps_get_free_size(MALLOC_CAP_INTERNAL);
    unit_of_measurement: 'B'
    state_class: measurement

Hi,

I made some test with my ESP32 cam with Arduino. With only Camera settings I have about 170-190 kB of memory free. With Camera and BT Proxy settings together I have only 27 kB of memory free and got the error:

[V][api.connection:992]: Cannot send message because of TCP buffer space [V][bluetooth_proxy:033]: Proxying packet from - 27:CA:24:07:00:67. RSSI: -41 dB [V][api.connection:992]: Cannot send message because of TCP buffer space [V][api.connection:992]: Cannot send message because of TCP buffer space [V][api:114]: Removing connection to Home Assistant 2022.12.7 (::FFFF:192.168.1.2) [D][binary_sensor:036]: 'Kamera předsíň Status': Sending state OFF [D][esp32_camera:172]: Got Image: len=97423 [D][esp32_camera:172]: Got Image: len=97475 [V][sensor:076]: 'Kamera předsíň Uptime': Received new state 48.987000 [D][sensor:127]: 'Kamera předsíň Uptime': Sending state 48.98700 s with 0 decimals of accuracy [D][esp32_camera:172]: Got Image: len=97450 [V][sensor:076]: 'Kamera předsíň Free Memory': Received new state 27.000000 [D][sensor:127]: 'Kamera předsíň Free Memory': Sending state 27.00000 kB with 1 decimals of accuracy [V][sensor:076]: 'Kamera předsíň WiFi Signal': Received new state -51.000000 [D][sensor:127]: 'Kamera předsíň WiFi Signal': Sending state -51.00000 dBm with 0 decimals of accuracy [D][esp32_camera:172]: Got Image: len=97376 [D][esp32_camera:172]: Got Image: len=98061

[Ulrar]

Okay, so I think I've done it ? I ran the commands in my ESPHome container, but I made sure to then use /usr/local/bin/esphome which seems to be where pip installed it (or at least the last modified time matches).

Unfortunately it didn't solve the issue, I'm still not getting the ServiceData in HA. I also checked the git status to confirm that esphome/components/esp32_ble_tracker/esp32_ble_tracker.cpp is the file that got modified, so I reverted the correct commit.

Guess it's something else ! I've also added esp32_ble_tracker: to my config but that didn't help either.

[cpuks]

Okay, so I think I've done it ? I ran the commands in my ESPHome container, but I made sure to then use /usr/local/bin/esphome which seems to be where pip installed it (or at least the last modified time matches).

Unfortunately it didn't solve the issue, I'm still not getting the ServiceData in HA. I also checked the git status to confirm that esphome/components/esp32_ble_tracker/esp32_ble_tracker.cpp is the file that got modified, so I reverted the correct commit.

Guess it's something else ! I've also added esp32_ble_tracker: to my config but that didn't help either.

You need two entries:

esp32_ble_tracker:
bluetooth_proxy:

And if you want to have active BLE there's also under bluetooth_proxy: active: true

check docs: https://esphome.io/components/bluetooth_proxy.html?highlight=bluetooth+proxy

[Ulrar]

Yeah I had the bluetooth proxy active only, and it was working fine. But regardless even with the tracker defined it doesn't work on 2022.12

[bdraco]

I think there are two possible causes:

1. The scanner is somehow doing passive scan when active has been requested.
2. There is another bug in the advertisement parser

We can check 2 with the following:

• Enable VERY_VERBOSE logging
• Capture all the advertisements from your switchbot devices
• Be sure to label the address of the device and type of device in the log
• The addresses are logged as uint64 in the log so you'll need to convert the BLE MAC address you see in the switchbot app to figure out the uint64 for the switchbot. You can do that with this snippet

  Python 3.10.8 (main, Oct 21 2022, 22:22:30) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> int("AA:BB:CC:DD:EE:FF".replace(":", ""), 16)
  187723572702975
  >>> 

[Ulrar]

Okay there we go, this should be the data from the same switchbot curtain I was looking at before :

[09:18:46][VV][esp32_ble_tracker:648]: Parse Result:
[09:18:46][VV][esp32_ble_tracker:665]:   Address: FA:62:0F:CF:2D:F2 (RANDOM)
[09:18:46][VV][esp32_ble_tracker:667]:   RSSI: -56
[09:18:46][VV][esp32_ble_tracker:668]:   Name: ''
[09:18:46][VV][esp32_ble_tracker:676]:   Ad Flag: 6
[09:18:46][VV][esp32_ble_tracker:682]:   Manufacturer data: FA.62.0F.CF.2D.F2.DA.0F.00.22.04 (11)
[09:18:46][VV][esp32_ble_tracker:698]: Adv data: 02.01.06.0E.FF.69.09.FA.62.0F.CF.2D.F2.DA.0F.00.22.04.00.09.16.3D.FD.63.C0.56.00.22.04 (29)
[09:18:46][V][bluetooth_proxy:033]: Proxying packet from  - FA:62:0F:CF:2D:F2. RSSI: -56 dB
[09:18:46][VV][api.service:334]: send_bluetooth_le_advertisement_response: BluetoothLEAdvertisementResponse {
  address: 275299078974962
  name: ''
  rssi: -56
  manufacturer_data: BluetoothServiceData {
  uuid: '0x0969'
  data: '\xfab\xcf-\xf2\xda

From HA's perspective no change, it's still not getting the service data. I've also looked through a few of these packets and they all look like that, as far as I can tell.

[bdraco]

02.01.06.0E.FF.69.09.FA.62.0F.CF.2D.F2.DA.0F.00.22.04.00.09.16.3D.FD.63.C0.56.00.22.04

02.01.06 = length 2, BLEGAPType.TYPE_FLAGS: 1, value 06 0E.FF.69.09.FA.62.0F.CF.2D.F2.DA.0F.00.22 = length 14, TYPE_MANUFACTURER_SPECIFIC_DATA: 255, value 69.09.FA.62.0F.CF.2D.F2.DA.0F.00.22.04 00 padding 09.16.3D.FD.63.C0.56.00.22.04 = length 9, TYPE_SERVICE_DATA: 22, value 3D.FD.63.C0.56.00.22.04

So the service data is in that string.

It has single byte of padding

[bdraco]

So the whole field is 29 bytes which means it might fit in the adv data without a scan response

[Ulrar]

Do you want me to give it a try with either (or both ?) of these PRs ?