Use the nimble stack instead of current bluetooth stack

[bluelu]

Describe the problem you have/What new integration you would like

The current bluetooth implementation uses the majority of available flash and memory space.
Someone has backported the new nimble stack to arduino (https://github.com/h2zero/NimBLE-Arduino), with only minor differences to the API.

It would be great if all the bluetooth classes could be ported to use this library instead of the current implementation.

Please describe your use case for this integration and alternatives you've tried:

Additional context

[OttoWinter]

ESPHome doesn't use the Arduino BLE stack.

We directly access the functions from esp-idf. From a quick look this "NimBLE" stack also uses esp-idf internally.

[bluelu]

@OttoWinter I might be wrong, but I thought that this component would use the nimble stack (backported from idf-4.0) directly without using the old direct access functions from esp-idf 3.x based on the bluedroid stack (https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/bluetooth/esp_gap_ble.html), and thus save memory?

There is also a non Arduino component version at https://github.com/h2zero/esp-nimble-cpp

However, it's only available for esp32.

I did test it by replacing the ble_scanner platform by the sample code from the example, and I saved 70KB of memory usage (showed by the esphome debug module), while still publishing all discovered devices over a text sensor field.

[OttoWinter]

Ok I thought NimBLE was a library like Arduino's BLE library by neil kolban.

But as far as I understand now nimble is actually an esp-idf component. Is that right?

If so, that would mean we only have to replace the function calls to the bluetooth stack in esp32_ble_scanner.cpp from esp_bluedroid_ to esp_nimble_ variants (obviously the APIs won't be 100% the same).

[bluelu]

Yes, nimble is an esp-idf component and has been ported to esp32 by espressif, but only available from esp-idf 4.0+ on. The arduino component is a backport of that version to earlier esp-idfs, and https://github.com/h2zero/esp-nimble-cpp seems to provide the old bluedroid stack API calls through nimble (however with a few changes).

Yes, but esp32 ble scanner depends on esp32_ble_tracker. I think it would be best to do the replacement for all the bluetooth classes (at least for esp32).

If esphome uses idf > 4.0, the best would be to use the esp-idf component directly I assume, otherwise the arduino component (which according to https://github.com/h2zero/NimBLE-Arduino/issues/80 will be published soon in platform.io) which should also contain the base nimble APi calls which could be used instead of the arduino API.

[AieatAssam]

Backport is already available for ESP-IDF 3.2 and 3.3: https://github.com/h2zero/esp-nimble-component

[morganrallen]

The NimBLE BLE stack is fantastic, I use it directly with the IDF in several projects, would be great to see ESPHome move over as I suspect it'll solve a lot of these WiFi disconnect issues.

[hawkeye217]

Encountering wifi disconnect issues pretty routinely (almost exactly 15 minutes apart each time) with an ESP32 controlling some LEDs and using `esp32_ble_tracker: for 4 Xiaomi sensors. It's annoying because the LEDs flash briefly every time the disconnect happens :(

[reaper7]

@hawkeye217 - I have similar, bad experiences :( as I can see, many people have a problem with the current ble implementation even if their projects are not very extensive...

maybe it's worth rethinking switching to NimBLE?

[met67]

Maybe now that EspHome moved under Home Assistant umbrella they will find the reources to try this? I can't have BLE tracker and Display at the same time...

[oetelaar24]

I'd also like to vote this up.. Seems like BLE is quite hefty right now and if there is any way of improving this, it would be very nice. Running into same memory issues on a ESP32_CAM module trying to do BLE and camera at the same time...

[met67]

Seems like the new esp-idf framework solves this issue... I managed do have BLE, Display and Wi-Fi at the same time (still too young to use in production, IMHO, but on the right path) https://esphome.io/changelog/2021.10.0.html#esp-idf

[morganrallen]

Oh cool, I had missed this (despite already updating a couple devices). Couple things to note though..

• ESP Arduinio is built on top of ESP-IDF, sounds like they've stripped this additional layer (great!)
• ESP-IDF still contains two BLE frameworks, Bluedroid and NiBLE.
• The improvements your seeing could just be reduced memory usage from removing the Arduino layer
• If that is true, further improvements could still be made by migrating to NiBLE.

Now that I'm aware of this I'm going to pay more attention to device stability and see if I'm getting improvements as well.

[OttoWinter]

Yeah we're still using bluedroid even with esp-idf, so likely just a side effect of removing a bit of overhead.

Potentially also because for esp-idf we only need to start the BLE mode instead of dual BLE+classic BT that arduino usually does.

[AieatAssam]

I have migrated my BLE monitors to ISP-IDF and am noticing a marked improvement in flash utilisation - from 85%+ down to 60%, allowing me to enable display support. As of right now, I have TTGO T-Display module polling 3 different Xiaomi LYWSD03MMC devices (pvvx firmware reflashed) and displaying the results on screen. Even with all these components (and WiFi) enabled, I am in reasonable utilisation ranges:

Wonder how much further this can be pushed with NimBLE. But even as it stands, ISP-IDF is a great improvement over Arduino-based implementation.

Running stable for 3 days+ as of 18 Nov 2021.

[geiseri]

this is good news! I will see if I can move my m5sticks this weekend.

[rradar]

@AieatAssam (& @met67 ) I wonder if you got a improved reception of ble packets? Typically the Xiaomi LYWSD03MMC publish temp+humidty every 10 seconds. So far with "default" esphome setup more than 90% of this packages got lost and esphome could only capture a fraction of it.

Does your configuration with ISP-IDF improve this situation?

[nagyrobi]

https://github.com/pvvx/ATC_MiThermometer/issues/149#issuecomment-1072981933

[mvgijssel]

I've started to experiment with the NimBLE stack in ESPHome here https://github.com/mvgijssel/setup/pull/127 by creating two external components nimble_tracker and nimble_rssi. This enables me to reliably track Apple watches which is not possible with Bluedroid at the moment.

Also I'm kinda surprised how stable the NimBLE stack is (famous last words 😅 )!

[nagyrobi]

Is it an 1:1 replacement? Does it work with bluetooth_proxy? Is it esp-idf only? How's your memory and heap usage compared to ESPHome's default?

[mvgijssel]

Is it an 1:1 replacement?

I think for the most part it can be! For my use case the rpa/irk is good enough, but the other functionality should be relatively easy to implement now that there is a start.

Does it work with bluetooth_proxy?

Don’t know! If the Bluetooth proxy feature relies on Bluetooth serial (or classic) then no.

Is it esp-idf only?

Currently it’s esp-idf only, but the libraries I’m using also support Arduino.

How's your memory and heap usage compared to ESPHome's default?

Haven’t checked this! Will do once I continue hacking and post back here :)

[nagyrobi]

Problem with esp-idf is that most of the components in ESPHome are not (yet) working on that platform, especially Ethernet, which is a popular feature these days.

[benedikt-bartscher]

@nagyrobi Ethernet with esp-idf works fine now: https://github.com/esphome/feature-requests/issues/1683

[nagyrobi]

Yes, I know and it's much better than in the old days.

[andrewjswan]

I'd also like to vote this up! Seems like BLE is quite hefty right now and if there is any way of improving this, it would be very nice.