search

arendst / Tasmota

Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at
https://tasmota.github.io/docs
GNU General Public License v3.0
22.21k stars 4.81k forks source link

Truncated BLE Beacon Advertisements with "HolyIoT" tags #18659

Closed ajlennon closed 1 year ago

ajlennon commented 1 year ago

PROBLEM DESCRIPTION

I am experimenting with some BLE tags from HolyIoT. These send out a regular unencrypted beacon advertisement with sensor data depending on the type of sensor (e.g.currently pressure, temperature, humidity)

http://www.holyiot.com/eacpzs.asp?dlb_id=31

10:36:45.845 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-22,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}}

I think this is because NimBLE as implemented in Tasmota is not handling "extended BLE advertisements" although I am not certain.

I have looked at the received payload size out of NimBLE and it is correct for the truncated data displayed

I have looked at enabling extended advertising in nimconfig.h but it doesn't seem to make a difference

/** @brief Un-comment to enable extended advertising */
#define CONFIG_BT_NIMBLE_EXT_ADV 1

/** @brief Un-comment to set the max number of extended advertising instances (Range: 0 - 4) */
#define CONFIG_BT_NIMBLE_MAX_EXT_ADV_INSTANCES 4

/** @brief Un-comment to set the max extended advertising data size (Range: 31 - 1650) */
#define CONFIG_BT_NIMBLE_MAX_EXT_ADV_DATA_LEN 1024

/** @brief Un-comment to enable periodic advertising */
#define CONFIG_BT_NIMBLE_ENABLE_PERIODIC_ADV 1

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

10:41:29.982 CMD: Backlog Template; Module; GPIO 255 10:41:30.020 MQT: stat/SmartPlug_FactoryDefault/RESULT = {"NAME":"2NICE UP111","GPIO":[0,320,0,0,2720,2656,0,0,2624,576,224,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1],"FLAG":0,"BASE":18} 10:41:30.038 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-25,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}} 10:41:30.262 MQT: stat/SmartPlug_FactoryDefault/RESULT = {"Module":{"1":"ESP32S3"}} 10:41:30.466 MQT: stat/SmartPlug_FactoryDefault/RESULT = {"GPIO0":{"0":"None"},"GPIO1":{"0":"None"},"GPIO2":{"0":"None"},"GPIO3":{"0":"None"},"GPIO4":{"0":"None"},"GPIO5":{"0":"None"},"GPIO6":{"0":"None"},"GPIO7":{"0":"None"},"GPIO8":{"0":"None"},"GPIO9":{"0":"None"},"GPIO10":{"0":"None"},"GPIO11":{"0":"None"},"GPIO12":{"0":"None"},"GPIO13":{"0":"None"},"GPIO14":{"0":"None"},"GPIO15":{"0":"None"},"GPIO16":{"0":"None"},"GPIO17":{"0":"None"},"GPIO18":{"0":"None"},"GPIO19":{"0":"None"},"GPIO20":{"0":"None"},"GPIO21":{"0":"None"},"GPIO33":{"0":"None"},"GPIO34":{"0":"None"},"GPIO35":{"0":"None"},"GPIO36":{"0":"None"},"GPIO37":{"0":"None"},"GPIO38":{"0":"None"},"GPIO39":{"0":"None"},"GPIO40":{"0":"None"},"GPIO41":{"0":"None"},"GPIO42":{"0":"None"},"GPIO43":{"0":"None"},"GPIO44":{"0":"None"},"GPIO45":{"0":"None"},"GPIO46":{"0":"None"},"GPIO47":{"0":"None"},"GPIO48":{"0":"None"}}

- [X] If using rules, provide the output of this command: `Backlog Rule1; Rule2; Rule3`:
```lua
  Rules output here:

- [X] Set `weblog` to 4 and then, when you experience your issue, provide the output of the Console log:
```lua
  Console output here:

10:42:44.225 CMD: weblog 4
10:42:44.232 MQT: stat/SmartPlug_FactoryDefault/RESULT = {"WebLog":4}
10:42:44.723 CFG: Saved, Count 45, Bytes 4096
10:42:44.951 MQT: tele/SmartPlug_FactoryDefault/BLE = {"Time":"2023-05-15T10:42:44","BLEDevices":{"total":3,"8C79F5A7A9E1":{"i":0,"r":-83},"F131F5C217BA":{"i":1,"r":-22},"B85548ABD42E":{"i":2,"r":-82}}}
10:42:44.968 MQT: tele/SmartPlug_FactoryDefault/BLE = {"Time":"2023-05-15T10:42:44","BLE":{"scans":3231,"adverts":352570,"devices":3,"resets":0}}
10:42:48.696 BLE: payload 30
10:42:48.744 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-21,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}}
10:42:49.655 BLE: payload 30
10:42:49.694 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-25,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}}
10:42:53.502 BLE: payload 30
10:42:53.545 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-25,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}}
10:42:56.379 BLE: payload 30
10:42:56.397 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-21,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC"}}
10:42:56.908 EQ3 poll cycle starting
10:42:59.258 BLE: payload 30

TO REPRODUCE

Use bledetails2 to look at raw advertisement data from a HolyIoT tag (or presumably any tag with extended advertising)

EXPECTED BEHAVIOUR

We should see all the advertising packet data for a tag, which should correspond to the raw data seem on e.g. an Android phone tool

SCREENSHOTS

image

image

ADDITIONAL CONTEXT

N/A

Jason2866 commented 1 year ago

Since BLE 5.0 not supported

ajlennon commented 1 year ago

Since BLE 5.0 not supported

I'm sorry I don't understand - longer advertisements are not supported in BLE5.0 (and later) ?

ajlennon commented 1 year ago

Ah you’re saying it needs BLE 5 for extended advertising. Yes the ESP32S3 should support this

Jason2866 commented 1 year ago

Yes, hardware does support. But it is not implemented in Tasmota software. Nimble probably needs to be configured different too.

ajlennon commented 1 year ago

Yes, hardware does support. But it is not implemented in Tasmota software.

Nimble probably needs to be configured different too.

Yes. So I've been through the Tasmota driver and I've been through the code that handles the adverts in NimBLE.

It looks to me as though changing the definitions I reference above for extended advertising in the .h file should work, but it doesn't.

The payload size is wrong coming out of NimBLE so it seems to me it's some configuration I'm not seeing.

Any thoughts on what might be missing?

Thanks,

Alex

barbudor commented 1 year ago

I think we can ask @h2zero some advice

h2zero commented 1 year ago

Do you have active scanning enabled? The missing data appears to be in the scan response as the advertisement data is at maximum.

ajlennon commented 1 year ago

Do you have active scanning enabled? The missing data appears to be in the scan response as the advertisement data is at maximum.

I don't. I'll give it a go!

ajlennon commented 1 year ago

OK this is fantastic thanks @h2zero, everybody . With active scanning enabled I now see all the advertisement data with a vanilla build with -DFIRMWARE_BLUEOOTH !!!

10:10:00.014 MQT: tele/SmartPlug_FactoryDefault/BLE = {"DetailsBLE":{"mac":"F131F5C217BA","RSSI":-36,"p":"0201061AFFFFFF021500000000000000000000000000000000271B4CB9CC0D09486F6C792D494F5400000000101642524162F131F5C217BA06060258C5"}}

I like these little HolyIoT tags a lot as there's - for example - temperature/pressure/humidity in a really small coin cell powered form factor

I am unclear on whether their choice to advertise with packets that use the BLE 5.x extended advertising format causes potential issues.

Clearly I need to do some more research but can you offer any advice on what impact "active scanning" has to the Tasmota stack and/or to power consumption on the devices themselves? It seems like a better option would be for them to advertise in the legacy format that doesn't use active scanning but they are probably unwilling to change their firmware I would guess

image

ajlennon commented 1 year ago

My HolyIoT beacon now reporting via BLE through Tasmota to an MQTT Mosquitto broker where I'm graphing some numbers :)

image

Jason2866 commented 1 year ago

Enabling Active Scan is a bad idea. It will eat up batteries of all BLE devices around. Imho the HolyIoT tags are a design fail.

ajlennon commented 1 year ago

Enabling Active Scan is a bad idea. It will eat up batteries of all BLE devices around. Imho the HolyIoT tags are a design fail.

This is exactly what I was worried about. Thanks for confirming. I suspect you are right. I will ask them if they are open to modifying the firmware so the advertising beacon data doesn't require active scanning.

ajlennon commented 1 year ago

@Jason2866 I have been investigating further. From what I am seeing it looks like an extended advertisement involves a secondary packet rather than connections. So this wouldn't involve connecting to devices and shouldn't be a problem maybe?

image

https://novelbits.io/bluetooth-low-energy-advertisements-part-1/

Jason2866 commented 1 year ago

If it is working without active scan, you are back at square one. Needs to be implemented. Without being verified working / supported from underlying Nimble library. The train stops here.

ajlennon commented 1 year ago

If it is working without active scan, you are back at square one. Needs to be implemented.

OK so "active scan" is where we are connecting to the device to interrogate it for extended information (if I am understanding) and we want to avoid this

Without being verified working / supported from underlying Nimble library. The train stops here.

So I need to confirm that BLE5.x extended advertising does not require active scan, and then work out why I'm not seeing the full packets through the NimBLE library. I'll try some basic (non-Tasmota) NimBLE examples to see what happens and if that doesn't get me anywhere I'll engage with the NimBLE folks and see if they can advise!

h2zero commented 1 year ago

These devices require active scanning in legacy mode because they provide the sensor data in the scan response. This is an unfortunate design choice for battery life.

If these devices can have extended advertising enabled then NimBLE will need to have it enabled as well to receive the extended advertisements. This can be done with the build config flags in your post above.

ajlennon commented 1 year ago

These devices require active scanning in legacy mode because they provide the sensor data in the scan response. This is an unfortunate design choice for battery life.

If these devices can have extended advertising enabled then NimBLE will need to have it enabled as well to receive the extended advertisements. This can be done with the build config flags in your post above.

HolyIoT say they have both. I will investigate... Starting to think I need some kind of BLE sniffer.

image

Jason2866 commented 1 year ago

@ajlennon h2zero is the maintainer of the used NimBLE lib in Tasmota ;-)

ajlennon commented 1 year ago

A little bit more research....

joopheuvel commented 1 year ago

Hi Alex / Jason. I am experiencing the same issue. Was hoping you could help me. I want to use/read the pushbutton on the HolyIOT beacon (ESp32, Arduino). As you indicate, I am getting the string until the point you also had, and the stuff I need is not part of that as it seems truncated. Tries a call for a specific part verus the whole ble string: sprintf (ble_string,"%s",advertisedDevice->toString().c_str()); and sprintf (manufacturer_data_string,"%s",advertisedDevice->getManufacturerData().c_str()); but the last one is no readable text. Desperate to find a solution as the board I am using for a gateopener system with beacon will make registration a lot easier versus having to push an extra button. Can you let me know how you think I can fix this...? Thanks

Joop van den heuvel

ajlennon commented 1 year ago

Hi @joopheuvel,

Sorry for the delay - I've been ill. As the others say above you can enable active scanning and you'll get the full message.

However I agree with the comments that this isn't what we want and really the HolyIoT firmware isn't fit for purpose.

They did say they'd do a redesign to fit everything into a single legacy advertisement but I haven't heard anything since.

Ultimately if they can't make this work as I think it should work for a battery powered beacon I won't be continuing to look at their devices :'(

joopheuvel commented 1 year ago

Thanks Alex No problem, hope you feel better,I did already have active scanning on:

if defined(NimBLE_libraries)

if (use_ble) {
  SerialPrintLogging ("Initializing NimBLE..\n");
  NimBLEDevice::init("");
  pBLEScan = NimBLEDevice::getScan(); //create new scan
  pBLEScan->setAdvertisedDeviceCallbacks(new

MyAdvertisedDeviceCallbacks(),TRUE); // https://github.com/nkolban/esp32-snippets/issues/464 pBLEScan->setActiveScan(TRUE); //active scan uses more power, but get results faster pBLEScan->setInterval(BLE_SCANINTERVAL_MS); // was 1000 when working pBLEScan->setWindow(BLE_SCANWINDOW_MS); // less or equal setInterval value was 500 when working pBLEScan->setMaxResults(0); // do not store the scan results, use callback only. lastlive_ble = millis(); }

endif // NimBLE_libraries

See code above. Ths would do the trick right? am still not seeing the full string but a truncated one as indicated. Am I doing something wrong. TRUE has value 1, so should work, right?. Thanks! Joop

On Wed, Jun 7, 2023 at 12:48 PM Alex Lennon @.***> wrote:

Hi @joopheuvel https://github.com/joopheuvel,

Sorry for the delay - I've been ill. As the others say above you can enable active scanning and you'll get the full message.

However I agree with the comments that this isn't what we want and really the HolyIoT firmware isn't fit for purpose.

They did say they'd do a redesign to fit everything into a single legacy advertisement but I haven't heard anything since.

Ultimately if they can't make this work as I think it should work for a battery powered beacon I won't be continuing to look at their devices :'(

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1580500827, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23VDCMV2M44U77BNZZTXKBMBVANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

ajlennon commented 1 year ago

Just taking a step back here @joopheuvel - I think you should be able to build Tasmota with BLE or install the BLE OTA firmware, then configure active scan on in the configuration section. then up the DEBUG messages and see the packets received which should be the full ones? I might have had to enable bridging to see this? Is this what you're trying?

joopheuvel commented 1 year ago

Thanks Alex,

This would be way too deep for me and beyond my technical capabilities. I will leave it for the moment, maybe will look into this later. Many thanks for your help and hope you feel better.

Another question is on how to update settings in these Holyiot beacons. do you happen to know how to do this, do they have generic passwords or specific?

Thanks! Joop

On Thu, Jun 8, 2023 at 12:24 PM Alex Lennon @.***> wrote:

Just taking a step back here @joopheuvel https://github.com/joopheuvel

  • I think you should be able to build Tasmota with BLE or install the BLE OTA firmware, then configure active scan on in the configuration section. then up the DEBUG messages and see the packets received which should be the full ones? I might have had to enable bridging to see this? Is this what you're trying?

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1582321496, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23QHFZ6DXYWBZQENYLLXKGR4VANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

ajlennon commented 1 year ago

It's actually not too hard. I'm happy to help. What ESP32 device are you running ?

ajlennon commented 1 year ago

Also this is the default password to connect (you can then change it)

image

joopheuvel commented 1 year ago

Great! Thanks Alex, will try. Joop

On Fri, Jun 9, 2023 at 1:08 PM Alex Lennon @.***> wrote:

Also this is the default password to connect (you can then change it)

[image: image] https://user-images.githubusercontent.com/1537834/244675809-d6d84a7b-9221-49f0-9390-952d0fd608f0.png

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1584401360, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23TX3YEGTVEPYXZ5Y73XKL73FANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

joopheuvel commented 1 year ago

Hi Alex,

I have a multi-station (1-6), event (festival/mobile) beacon tracker and gate opener system (for beacons on cords and on cars). Have a mix of (Olimex ESP32-EVB's) that run from a (Teltonika) Router supplying MQTT tags, and some work autonomously running their BLE with the external antenna. They track the beacons and as form of registration/deregistration the button clock close to an ESP32 would be nice to register/unregister.

Thanks! Joop

On Fri, Jun 9, 2023 at 1:07 PM Alex Lennon @.***> wrote:

It's actually not too hard. I'm happy to help. What ESP32 device are you running ?

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1584399679, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23STEIGIXHE3BY4J2FDXKL7WJANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

ajlennon commented 1 year ago

Have a mix of (Olimex ESP32-EVB's) that run from a (Teltonika) Router

OK! So if I assume this is a bog standard ESP32 then I suggest you program a standard Tasmota Bluetooth image in there

If you are doing an OTA update "Firmware Update" then you can use this image

http://ota.tasmota.com/tasmota32/release/tasmota32-bluetooth.bin

Or if you are programming with esptool.py then you can use this image

http://ota.tasmota.com/tasmota32/release/tasmota32-bluetooth.factory.bin

With

esptool.py write_flash 0x0 tasmota32-bluetooth.factory.bin

This will give you an extra configuration option for Bluetooth where you can enable active scanning

Try that and if that works for you I'll try to dig out the commands to enable Bluetooth beacon -> MQTT bridging

Cheers!

joopheuvel commented 1 year ago

Hey Alex.

You are quick. Thanks, very helpful indeed. Let me wait with flashing until I gave the next batch in, just to be safe. I am sure we will get it to work. Was able to change settings already, lower power and slower interval. Many thanks again. Note we were using teltonika beacons but batteries can not be replaced. Bommer. These were 10 euro a piece, replacement batteries, and seem to do the job.

Joop

On Sat, Jun 10, 2023, 16:06 Alex Lennon @.***> wrote:

Have a mix of (Olimex ESP32-EVB's) that run from a (Teltonika) Router

OK! So if I assume this is a bog standard ESP32 then I suggest you program a standard Tasmota Bluetooth image in there

If you are doing an OTA update "Firmware Update" then you can use this image

http://ota.tasmota.com/tasmota32/release/tasmota32-bluetooth.bin

Or if you are programming with esptool.py then you can use this image

http://ota.tasmota.com/tasmota32/release/tasmota32-bluetooth.factory.bin

With

esptool.py write_flash 0x0 tasmota32-bluetooth.factory.bin

This will give you an extra configuration option for Bluetooth where you can enable active scanning

Try that and if that works for you I'll try to dig out the commands to enable Bluetooth beacon -> MQTT bridging

Cheers!

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1585679692, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23V6CUSTGFSCASMUHWDXKR5QBANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

ajlennon commented 1 year ago

Let me wait with flashing until I gave the next batch in, just to be safe.

Sure thing - just @ me here when you're ready to go :)

ajlennon commented 1 year ago

Update: The HolyIoT people have been back to me and are nearly ready with a new firmware build

"the firmware almost finished , we put the battery level , tempreture, humidity,air pressure in the scan mode"

So that sounds positive! Will keep you all updated on this thread

joopheuvel commented 1 year ago

Ah. Top. Great Alex, keep me in the loop please. Very much appreciated. Joop

On Mon, Jun 12, 2023, 12:40 Alex Lennon @.***> wrote:

Update: The HolyIoT people have been back to me and are nearly ready with a new firmware build

"the firmware almost finished , we put the battery level , tempreture, humidity,air pressure in the scan mode"

So that sounds positive! Will keep you all updated on this thread

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-1587065354, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23VY6IA4Z6HXK4DVN33XK3WZZANCNFSM6AAAAAAYB5XIRQ . You are receiving this because you were mentioned.Message ID: @.***>

rus-ik commented 6 months ago

Update: The HolyIoT people have been back to me and are nearly ready with a new firmware build

"the firmware almost finished , we put the battery level , tempreture, humidity,air pressure in the scan mode"

So that sounds positive! Will keep you all updated on this thread

Do you have any updates regarding the HolyIoT beacon?

joopheuvel commented 6 months ago

Thanks for checking.

In the meantime we have been moving on with Teltonika beacons and are not actively using the HolyIOT beacons.

Thanks and best regards. Joop

Op wo 8 mei 2024 om 13:13 schreef rus-ik @.***>:

Update: The HolyIoT people have been back to me and are nearly ready with a new firmware build

"the firmware almost finished , we put the battery level , tempreture, humidity,air pressure in the scan mode"

So that sounds positive! Will keep you all updated on this thread

Do you have any updates regarding the HolyIoT beacon?

— Reply to this email directly, view it on GitHub https://github.com/arendst/Tasmota/issues/18659#issuecomment-2100337918, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSR23QAUAOLHH5CXO2ZYTDZBICHDAVCNFSM6AAAAAAYB5XIRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGMZTOOJRHA . You are receiving this because you were mentioned.Message ID: @.***>