Mixing ble_client with ble_gateway

[BrianAker]

Hi!

Is there a way to combine ble_client with ble_gateway?

I have miflora sensors which are being passively read via ble_gateway, I am forwarding all of their BLE data processing to Home Assistant. This is working quite well.

What is missing? The battery information. For miflora devices you need to do a read of the battery data; the firmware did at one point broadcast this information but because of the battery drain, later firmware versions for miflora plant sensors removed this feature.

Your ble_client code I think would be an awesome solution to this since you can specify intervals and you are not limited to the number of client limitations that the builtin esphome ble stack has.

So far, I haven't been able to figure out how to make it work. Do you have an example?

An optimal solution would be for the client and gateway code to use the same device list, or better, have the ble_client only make requests of devices it has seen.

Do you have anything you can share about using ble_client with ble_gateway?

Thanks!

FWIW, I have 3 esp32 tracking ~48 miflora sensors.

[myhomeiot]

You can do it like in this example, ble_client will read battery information and sends it to Passive BLE Monitor as BLE packet like earlier firmware's.

[myhomeiot]

@BrianAker Did the example helps you?

[BrianAker]

The example compiled, but it is not working. I don’t understand enough of your code yet to be sure of what I believe the error is.

All that appears in the log is the following ( repeatedly):

[23:01:51][D][myhomeiot_ble_client:069]: [C4:7C:8D:XX:XX:XX] Found device [23:01:51][I][myhomeiot_ble_client:032]: [C4:7C:8D:XX:XX:XX] Connecting [23:01:52][D][myhomeiot_ble_client:118]: [C4:7C:8D:XX:XX:XX] DISCONNECT_EVT [23:01:52][W][myhomeiot_ble_client:083]: [C4:7C:8D:XX:XX:XX] OPEN_EVT failed, status (133), app_id (1)

I also see the following error from time to time, but I don’t believe related ( and pre-dates me adding the battery example): [E][ble_gateway:090]: set_devices: Devices lengths (4) must be a multiple of 12

Thanks!

乌

On Jul 3, 2022, at 3:10 AM, myhomeiot @.***> wrote:

@BrianAker https://www.google.com/url?q=https://github.com/BrianAker&source=gmail-imap&ust=1657447812000000&usg=AOvVaw0YU-h9oncmBS7fQw7msqnm Did the example helps you?

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/issues/11%23issuecomment-1173052048&source=gmail-imap&ust=1657447812000000&usg=AOvVaw31JTMRmmKxeasumuWyyD16, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AAACDX4PVZRDXJK7ZPOKVODVSFRIBANCNFSM52K5666Q&source=gmail-imap&ust=1657447812000000&usg=AOvVaw1r8ZQ1GooGl5eAf1nfUAOC. You are receiving this because you were mentioned.

[myhomeiot]

OPEN_EVT failed, status (133), app_id (1) means that it's can't connect to MiFlora, sometimes it's happens but if it's can't connect at all probably the distance too far. BT connection requires lower distance than BLE, try to move this MiFlora closer to ESP.

set_devices: Devices lengths (4) must be a multiple of 12 check binary_sensor.ble_gateway devices attribute in HA, it's contains MACs which should be set at ESP and size of this attribute must be a multiple of 12 because each MAC address in hex = 12 chars.

[BrianAker]

Hi!

On Jul 3, 2022, at 11:20 PM, myhomeiot @.***> wrote: OPEN_EVT failed, status (133), app_id (1) means that it's can't connect to MiFlora, sometimes it's happens but if it's can't connect at all probably the distance too far. BT connection requires lower distance than BLE, try to move this MiFlora closer to ESP

The miflora is only ~4 feet away from the esp. If the esphome device is only doing the “myhomeiot_ble_client”, aka battery/firmware read, everything is fine. The OPEN_EVT happens when the esphome device is using both ble_gateway and myhomeiot_ble_client.

Looking at this a bit more closely:

[19:26:59][D][ble_gateway:063]: [C4:7C:8D:6D:4B:EB] Packet 043E2802010000EB4B6D8D7CC41C020106030295FE141695FE71209800D4EB4B6D8D7CC40D041002E300A8 [19:26:59][D][myhomeiot_ble_client:069]: [C4:7C:8D:6D:4B:EB] Found device [19:26:59][I][myhomeiot_ble_client:032]: [C4:7C:8D:6D:4B:EB] Connecting [19:27:02][D][text_sensor:067]: 'Kitchen Relay Uptime': Sending state '9m 44s' [19:27:02][D][sensor:125]: 'Uptime Sensor': Sending state 584.04498 s with 0 decimals of accuracy [19:27:05][D][myhomeiot_ble_client:118]: [C4:7C:8D:6D:4B:EB] DISCONNECT_EVT [19:27:05][W][myhomeiot_ble_client:083]: [C4:7C:8D:6D:4B:EB] OPEN_EVT failed, status (133), app_id (1)

The “status (133)” I believe is a generic error that requires digging a bit deeper in order to find what the actual error was.

set_devices: Devices lengths (4) must be a multiple of 12 check binary_sensor.ble_gateway devices attribute in HA, it's contains MACs which should be set at ESP and size of this attribute must be a multiple of 12 because each MAC address in hex = 12 chars.

Turning up the logging verbosity provides this clue:

[00:56:51][D][homeassistant.text_sensor:015]: 'binary_sensor.ble_gateway::devices': Got attribute state 'None' [00:56:51][V][text_sensor:016]: 'ble_gateway_devices': Received new state None [00:56:51][D][text_sensor:067]: 'ble_gateway_devices': Sending state 'None'

[00:56:51][E][ble_gateway:090]: set_devices: Devices lengths (4) must be a multiple of 12

Home Assistant is sending the state as “None”, ie the text string “None”, which is not a valid MAC :)

If at this point I turn off discovery and then re-enable it, a valid list of MAC addresses is sent to esphome device. In the process of checking to see if the error has been resolved on the esphome, the device would also have its api connection reset, so the state of “None” may be resolving itself via that.

— Brian

[myhomeiot]

Try to disable discovery and specify few required MACs in ESP or Home Assistant. I use both ble_gateway and myhomeiot_ble_client simultaneously and doesn't see problems. Sometimes I seen OPEN_EVT failed, status (133) but after few tryes it's recovers.

[BrianAker]

In MyHomeIOT_BLEClient::connect() I see that you call esp_ble_gattc_open(), but I cannot find where esp_ble_gap_stop_scanning() has been previously called.

What I am finding with OPEN_EVT errors is that the devices that are failing appear to have batteries which report lower percentages; I suspect the error is related to that. I am also finding that the esp32 will eventually hang itself in different places with repeated OPEN_EVT errors while having my debugger open to it ( which I believe is some sort of issue with resources not being released/some other random corruption ).

FWIW On the other issue I found, aka the data validation for discovery I found, I will open up a different bug/pull request for a fix for that. I have the HA side of that fixed, but I want to fix the validation side on the ESP32 side as well.

乌

On Jul 5, 2022, at 1:22 AM, myhomeiot @.***> wrote:

Try to disable discovery and specify few required MACs in ESP or Home Assistant. I use both ble_gateway and myhomeiot_ble_client simultaneously and doesn't see problems. Sometimes I seen OPEN_EVT failed, status (133) but after few tryes it's recovers.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/issues/11%23issuecomment-1174764320&source=gmail-imap&ust=1657614156000000&usg=AOvVaw12nshUHm5keC1M7kWbXyGD, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AAACDX76GEWN6WFYXJRIX33VSPWERANCNFSM52K5666Q&source=gmail-imap&ust=1657614156000000&usg=AOvVaw2ySCzCBZ03PawJNRYw3yEP. You are receiving this because you were mentioned.

[myhomeiot]

esp_ble_gap_stop_scanning calling only at esp32_ble_tracker, MyHomeIOT_BLEClient like other BLE components registered using esp32_ble_tracker.register_client (in our case it's MyHomeIOT_BLEHost) and just called from ESP32BLETracker.

Read my reply about my observations of MiFlora battery level.

[BrianAker]

For the call to esp_ble_gattc_open() in:

https://github.com/myhomeiot/esphome-components/blob/main/components/myhomeiot_ble_client/myhomeiot_ble_client.cpp

void MyHomeIOT_BLEClient::connect() { ESP_LOGI(TAG, "[%s] Connecting", tostring(this->address).cstr()); this->state = MYHOMEIOT_CONNECTING; if (auto status = esp_ble_gattc_open(blehost->gattc_if, this->remotebda, BLE_ADDR_TYPE_PUBLIC, true))

MyHomeIOT_BLEClient::connect() is expecting that ESP32BLETracker has called esp_ble_gap_stop_scanning() before your call to esp_ble_gattc_open()?

The same goes for ESP32BLETracke calling esp_ble_gattc_close()? In MyHomeIOT_BLEClient::gattc_event_handler() if it reaches OPEN_EVT, then MyHomeIOT_BLEClient::disconnect() should still be called? ( Which in turn would call esp_ble_gattc_close() )

Thanks!

乌

On Jul 9, 2022, at 12:01 AM, myhomeiot @.***> wrote:

esp_ble_gap_stop_scanning calling only at esp32_ble_tracker, MyHomeIOT_BLEClient like other BLE components registered using esp32_ble_tracker.register_client (in our case it's MyHomeIOT_BLEHost) and just called from ESP32BLETracker.

Read my reply https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/issues/3%23issuecomment-1178925191&source=gmail-imap&ust=1657954870000000&usg=AOvVaw2NeyXDhur3eMEdoY7LB3UW about my observations of MiFlora battery level.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/issues/11%23issuecomment-1179492661&source=gmail-imap&ust=1657954870000000&usg=AOvVaw27FpgJEnVVL2Ieou8f1L69, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AAACDX7QRXYF3F23UADYELDVTEPTJANCNFSM52K5666Q&source=gmail-imap&ust=1657954870000000&usg=AOvVaw2A2a8qORnqYxdvEpyX83J6. You are receiving this because you were mentioned.

[myhomeiot]

MyHomeIOT_BLEHost + MyHomeIOT_BLEClient = regular BLEClient, so you can compare logic of MyHomeIOT_BLEClient with it. MyHomeIOT_BLEClient esp_ble_gattc_open and BLEClient esp_ble_gattc_open

[BrianAker]

I figured out where the resource leak is occurring ( I believe ).

When ESP_GATTC_DISCONNECT_EVT is called I added the following:

 case ESP_GATTC_DISCONNECT_EVT: {

• ESP_LOGD(TAG, "[%s] ESP_GATTC_DISCONNECT_EVT, conn_id (%d), reason (%#x)",
• tostring(this->address).c_str(), param->disconnect.conn_id,
• param->disconnect.reason); if (memcmp(param->disconnect.remote_bda, this->remotebda, sizeof(this->remotebda)) != 0) break;
• ESP_LOGD(TAG, "[%s] DISCONNECT_EVT", tostring(this->address).cstr()); this->state = MYHOMEIOT_IDLE;
• this->disconnect();

The line you care about is the call to this->disconnect(). Without that line, you will go to ESP_GATTC_OPEN_EVT with a general status error, 0x85 with having never made a call to esp_ble_gattc_close(). The reason for the error is found in the call to ESP_GATTC_DISCONNECT_EVT, which for me seems to be 0x3e most of the time. From my reading of Espressif’s documentation, which is pretty sparse on how to handle errors, esp_ble_gattc_close() or esp_ble_gap_disconnect() need to be called when the even ESP_GATTC_DISCONNECT_EVT occurs. ESP_GATTC_OPEN_EVT will always be called after ESP_GATTC_DISCONNECT_EVT as well as ESP_GATTC_CONNECT_EVT. While ESP_GATTC_OPEN_EVT knows an error has occurred via status, the actual error has been lost by that point. So I believe, and from what I can see in Espressif’s examples, ESP_GATTC_DISCONNECT_EVT is where the disconnect should be called.

The second change I made was the following:

ESP_LOGI(TAG, "[%s] Disconnecting", tostring(this->address).cstr()); this->state = MYHOMEIOT_IDLE;

• if (auto status = esp_ble_gattc_close(blehost->gattc_if, this->connid))
• if (auto status = esp_ble_gap_disconnect(this->remotebda)) ESP_LOGW(TAG, "[%s] close error, status (%d)", tostring(this->address).c_str(), status); }

At least in my case, I am looking at requesting the battery value once a day for 30+ devices via one esp32, and my plan is to make this more generic. In the first fix I made sure that the resources would be properly cleaned up during an error. My change disconnect makes it more inline with the nature of passive devices. In order to not hit the compiled in limit to the number of devices you can connect to, you need to disconnect from devices and not just close the virtual handle. By calling esp_ble_gap_disconnect() I achieve that. The documentation on esp_ble_gap_disconnect() is sparse, but coming from other bluetooth stacks, I think the logic of what I am doing makes sense.

Those are the minimum changes I would make.

Putting all of this aside, I think that ble client connections from the esp32 while it is being used for scanning should behave differently. Clearly, a hard limit on the number of devices that it is connecting to needs to exist. I think in the case of scanning, in particular, any query to a device needs to be completed before the device starts scanning again ( I think Espressif’s documentation also states this, and I know this is a limit on other BLE platforms I have worked on ). To achieve this, the client needs to be rewritten to delay the start of scanning until it is done.

I have 30+ devices being updated for their battery usage at the moment. It is has been running for days without any issue.

If you would like, I can give you a merge request which shows you the two changes above ( it would also have some format fixing and clarifies some of the logging calls ).

Thanks for your code, it has been a great help in trying to solve the problem of monitoring my fleet of plants :)

— Brian

乌

On Jul 10, 2022, at 2:57 AM, myhomeiot @.***> wrote:

MyHomeIOT_BLEHost + MyHomeIOT_BLEClient = regular BLEClient, so you can compare logic of MyHomeIOT_BLEClient with it. MyHomeIOT_BLEClient esp_ble_gattc_open https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/blob/cbad9137c3ca3090ac04a81874794c50747bb73b/components/myhomeiot_ble_client/myhomeiot_ble_client.cpp%23L34&source=gmail-imap&ust=1658051867000000&usg=AOvVaw30U91Q8kCD3wPia-7KZihY and BLEClient esp_ble_gattc_open https://www.google.com/url?q=https://github.com/esphome/esphome/blob/e0555e140f8e8b7debe26c498dbf115cfd52bff5/esphome/components/ble_client/ble_client.cpp%23L86&source=gmail-imap&ust=1658051867000000&usg=AOvVaw2SU5kAbuL7g5XdILLuRSJH — Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/myhomeiot/esphome-components/issues/11%23issuecomment-1179695924&source=gmail-imap&ust=1658051867000000&usg=AOvVaw2TadH89w6gzKzj959Bfnl0, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AAACDX5PJRIMU4EVZLOOV6TVTKNBRANCNFSM52K5666Q&source=gmail-imap&ust=1658051867000000&usg=AOvVaw0bo2OOz_9rxjdrwwI6rkWR. You are receiving this because you were mentioned.

[myhomeiot]

@BrianAker Sorry for delay with the answer. I really appreciate the great work which you done and results you achieved. If until today your changes works without any problems please make PR (merge request).

Please take into account that today in version 1.0.9 I fixed compatibility for ESPHome 2022.11.x. Thanks!