esp32dev-ble-mqtt-undecoded 1.7.0 'adaptivescan: false' trigger ESP32 to reboot every 24h

1technophile / OpenMQTTGateway

MQTT gateway for ESP8266 or ESP32 with bidirectional 433mhz/315mhz/868mhz, Infrared communications, BLE, Bluetooth, beacons detection, mi flora, mi jia, LYWSD02, LYWSD03MMC, Mi Scale, TPMS, BBQ thermometer compatibility & LoRa.

https://docs.openmqttgateway.com

GNU General Public License v3.0

3.62k stars 795 forks source link

esp32dev-ble-mqtt-undecoded 1.7.0 'adaptivescan: false' trigger ESP32 to reboot every 24h #1920

Closed lux73 closed 7 months ago

lux73 commented 7 months ago

Before submitting a problem please check the troubleshooting section https://docs.openmqttgateway.com/upload/troubleshoot.html

Describe the bug my ESP32 reboot every 24h with adaptivescan: false

To Reproduce Steps to reproduce the behavior: BTtoMQTT config: {"bleconnect":false,"interval":100,"adaptivescan":false,"intervalacts":100,"intervalcnct":300000,"scanduration":1000,"onlysensors":false,"randommacs":false,"hasspresence":false,"prestopic":"presence/","presuseuuid":false,"minrssi":-90,"extDecoderEnable":true,"extDecoderTopic":"undecoded","filterConnectable":false,"pubadvdata":false,"pubuuid4topic":false,"ignoreWBlist":false,"presenceawaytimer":120000,"movingtimer":60000,"forcepscn":false,"tskstck":2596,"crstck":3064,"enabled":true,"scnct":132776}

SYStoMQTT config: {"uptime":148566,"version":"v1.7.0","disc":false,"ohdisc":false,"env":"esp32dev-ble-mqtt-undecoded","freemem":94436,"mqttp":"1883","mqtts":false,"msgprc":459597,"msgblck":0,"maxq":5,"minmem":27660,"tempc":60,"freestck":2092,"eth":false,"rssi":-41,"SSID":"xxx","BSSID":"xxx","ip":"192.168.254.254","mac":"A8:42:E3:56:AC:B0","lowpowermode":-1,"modules":["WebUI","BT"]}

RLStoMQTT config: {"latest_version":"v1.7.0","title":"OpenMQTTGateway","release_url":"https://github.com/1technophile/OpenMQTTGateway/releases/tag/v1.7.0","installed_version":"v1.7.0","entity_picture":"https://github.com/1technophile/OpenMQTTGateway/raw/development/docs/img/Openmqttgateway_logo_mini_margins.png","release_summary":""}

Expected behavior with adaptivescan: true my ESP32 will survive Days without reboot - but my ESP32 is used only for Presence Detection with some Feasycom Beacon's so i don't need adaptivescan: true while there are no addional BT Devices to control

Screenshots no Screenshots

Environment (please complete the following information):

esp32dev-ble-mqtt-undecoded 1.7.0
ESP32 WROOM 32U

Additional context i am new to this Project - i am overseeing anything or are my Config not correct? the presence scan/detection works without any Problem. But i would like to know why my System is constantly rebooting every other 24h? i need my ESP32 only to scan my BT Beacons without disruption - any other Config advised?

DigiH commented 7 months ago

Hi @lux73

Would you mind trying out the current development test build version of OpenMQTTGateway, and also answer a few questions about your set-up?

• Do you use the -undecoded version of OMG specifically to use a separate decoding option like Theengs Gateway? • Which model of a Feasycom Beacon do you have? • If you only use this to monitor your Feasycom Beacon for presence detection, do you really need it to scan every 100ms, or would not a presence detection every 10 seconds or so be enough for your purpose? • Similarly to the above, if your only use is for the Feasycom Beacon presence detection you should set "intervalacts": to a higher value, e.g. 300000

The current OpenMQTTGateway development test version does include some changes which address restart issues, but additionally it also includes a recently added Feasycom Beacon decoder which should correctly identify the beacon model and also include the battery status, if applicable, of the beacon, when using a general ble non-undecoded binary.

The current development test builds are available from the web install site https://docs.openmqttgateway.com/dev/upload/web-install.html

lux73 commented 7 months ago

Hi @DigiH

thanks for your reply.

i'm using the -undecoded Version while it's the only one tagged with 'mqtt' so i assume it's the only one with MQTT inside?
Feasycom DA14531
every reboot some of the value's are resettet, even if i try to store them with '"save":true' - these Value's are interval, intervalacts & some other. i'm tired to adjust these value's every Day

first of all i have planned to get in touch with switchbot - but these switchbot were not reliable with my System, so i decided to use my ESP only as Presence detection. if things go stable than i would like to extend the actual presence scanning

at the moment my OpenHAB monitor the 'xxx-undecoded' mqtt opic and if its in Range than my Presence Detection Rule could fire. there is no other Topic related the BT Beacons in Range.

sorry for my ignorance but it is possible to get a compiled Development Firmware? the Web Updater doesnt't work for me & i can't find any dev-binarys. The Website seems a bit nested ^^

initial Setup i had downloaded all Firmware Files:

boot_app0.bin
esp32dev-ble-mqtt-undecoded-bootloader.bin
esp32dev-ble-mqtt-undecoded-firmware.bin
esp32dev-ble-mqtt-undecoded-partitions.bin

& flash it via 'flash_download_tool_3.9.5.exe'

this work's nice - but the constant reboots every 24h drive me crazy ^^

DigiH commented 7 months ago

Hi @lux73

i'm using the -undecoded Version while it's the only one tagged with 'mqtt' so i assume it's the only one with MQTT inside?

I can see that this might be confusing and look like this, especially when completely new to OpenMQTTGateway, but as the name suggest, all different protocol, like RF, IR, LoRa, GSM and in your case Bluetooth BLE gateways use MQTT as the final publishing results. More described and explained on the general OpenMQTTGateway homepage

https://docs.openmqttgateway.com/#products-powered-by-openmqttgateway

Feasycom DA14531

I think this is a Feasycom protocol, used by several different models of their beacons, like the BP121, BP106 etc.

Would you mind sharing one of the undecoded MQTT messages from your Feasycom beacon here, so I can see if it will actually be included already in the Feasycom decoder?

every reboot some of the value's are resettet, even if i try to store them with '"save":true' - these Value's are interval, intervalacts & some other. i'm tired to adjust these value's every Day

I'd be tired, well annoyed and going crazy about that as well ;) but this shouldn't happen at all, once any changed values are committed to flash memory with '"save":true'. So we will have to see what might be the issue with your actual set-up. I think it is actually part of the, not required by you, -undecoded set-up.

sorry for my ignorance but it is possible to get a compiled Development Firmware? the Web Updater doesnt't work for me & i can't find any dev-binarys. The Website seems a bit nested ^^

This is actually what I referred you to above, with the development test build pre-built binaries, easily installable via web upload. All you need is the Microsoft Edge or Google Chrome web browser, and follow the steps described on the site. Ideally you would install the generic esp32dev-ble binary, which should then also decoder your Feasycom Beacon, depending on which model it is with the DA14531 protocol and also have its battery level included in its MQTT published messages.

https://docs.openmqttgateway.com/dev/upload/web-install.html

With the web install you don't actually need to download any binary, but if you really do want to download a binary and install it via 'flash_download_tool_3.9.5.exe', all you need to download is the esp32dev-ble-firmware.bin as the additional bootloader and partitions files are only required if and when you are using the Arduino IDE for installation.

lux73 commented 7 months ago

Hi @DigiH

is this what you are looking for?

T: Creating BLE buffer
N: Device detected: DC:0D:30:16:CF:1A
T: getDeviceByMac DC:0D:30:16:CF:1A
T: Get services data number: 2
T: Converted service data (14) to 20000bd680000014a1df0c2d1eb7
T: Service data: 20000bd680000014a1df0c2d1eb7
T: Service data UUID: 0xfeaa
T: Processing BLE data DC:0D:30:16:CF:1A
T: Random MAC or iBeacon device filtered
T: Origin: /BTtoMQTT/undecoded
T: Enqueue JSON
T: Queue length: 1
T: Converted service data (11) to 29021992dc0d3016cf1a64
T: Service data: 29021992dc0d3016cf1a64
T: Service data UUID: 0xfff0
T: Processing BLE data DC:0D:30:16:CF:1A
T: Random MAC or iBeacon device filtered
T: Origin: /BTtoMQTT/undecoded
T: Enqueue JSON
T: Queue length: 2
T: Dequeue JSON
N: Send on /BTtoMQTT/undecoded msg {"id":"DC:0D:30:16:CF:1A","name":"FBeaconLX","rssi":-86}
T: jsonPubl - ON
T: [ OMG->MQTT ] topic: esp32/OMG_ESP32_BLE/BTtoMQTT/undecoded msg: {"id":"DC:0D:30:16:CF:1A","name":"FBeaconLX","rssi":-86} 
T: Dequeue JSON
N: Send on /BTtoMQTT/undecoded msg {"id":"DC:0D:30:16:CF:1A","name":"FBeaconLX","rssi":-86}
T: jsonPubl - ON
T: [ OMG->MQTT ] topic: esp32/OMG_ESP32_BLE/BTtoMQTT/undecoded msg: {"id":"DC:0D:30:16:CF:1A","name":"FBeaconLX","rssi":-86}

seems to me there's no hidden Information such Battery State or anything ^^

okay, then i should flash the correct Firmware... could this be done via WebInterface 'Upgrade by Web Server' if i adjust the OTA URL Path?

but i assume this will break my actually OpenHAB config when there is no more /undecoded Topic after (Things, Items, Rules) so it can be done not before the Weekend, i am very busy atm...

btw you mentioned that i only need to flash the esp32dev-ble-firmware.bin - then i don't understood while it's not so declared on the official Website: https://docs.openmqttgateway.com/upload/binaries.html#esp32

there is a detailed Screenshot how to flash an ESP32 - my Device is an ESP32 ^^

sorry for my ignorance - this ESP32 is my very first Device... it's very hard for me to understand all these Things even if i familiar with Linux, FreeBSD, Java, Javascript 😎

DigiH commented 7 months ago

Hi @lux73

is this what you are looking for?

Yes, thanks, this helps a lot, and your Feasycom beacon should already be recognised and be (partially) decoded, including the battery level - which is currently 100% in your above data sample - but partially only, because I am still unsure which Feasycom Bluetooth Beacon model you have. If you have any further details on the model, something like BPXXX, possibly from the Feasycom Beacon app or such, it would be great to include this in the decoder. The current decoding will produce an MQTT message like {"brand":"Feasycom","model":"Beacon","model_id":"FEASY","batt":100,"plugged-in":false} and once a model is identified with your help {"brand":"Feasycom","model":"Beacon","model_id":"FEASY","beaconmodel":"BPXXX","batt":100,"plugged-in":false}

With which name, like FSC-BPXXX, does yours come up in the FeasyBeacon app, similar to this?

sorry for my ignorance - this ESP32 is my very first Device...

No worries, we will get there.

okay, then i should flash the correct Firmware... could this be done via WebInterface 'Upgrade by Web Server' if i adjust the OTA URL Path?

Do you not have access to either the Microsoft Edge or Google Chrome web browser, even if it is only for using the very easy web upload functionality?

All you really have to do then is to connect your ESP32 to your computer with a cable, go to the development upload web site

https://docs.openmqttgateway.com/dev/upload/web-install.html

select esp32dev-ble in the binaries selection popup and hit install/upload.

Alternatively yes, this should also possible through the WebUI, though I am not 100% sure with the diverging binaries being used, and you might still need to reset your current "extDecoderEnable":true to "extDecoderEnable":false if the flash isn't erased when installing/updating to the development esp32dev-ble binary.

Unfortunately I have not got a direct OTA URL Path for the WebUI update at the moment, as there are asset issues with the current build actions.

btw you mentioned that i only need to flash the esp32dev-ble-firmware.bin - then i don't understood while it's not so declared on the official Website

My bad, I was thinking of other general ESP32 binary flashing tools ;)

but i assume this will break my actually OpenHAB config when there is no more /undecoded Topic after (Things, Items, Rules) so it can be done not before the Weekend, i am very busy atm...

Take your time, but with the properly decoding development build you should be able to change any /undecoded reference you have in OpenHAB to /DC0D3016CF1A. This will then also only received your decoded Feasycom beacon and not have any other possible undecoded devices in your area which might dazwischenfunken ;) and it will have the correct battery level for you to easily separate from the received JSON with a "JSONPATH", "$.batt" transform.

The undecoded binary really should only be used when an external decoder like Theengs Gateway is being employed.

https://docs.openmqttgateway.com/use/ble.html#advanced-setting-up-an-external-decoder

lux73 commented 7 months ago

my Feasycom Beacons are

BP103B - that shows the Feasycom App under 'Module'

okay that seems very easy to implement - so every Device get's his own Topic which is the Beacon MAC Address 😎

i will flash & re-setup my ESP & OpenHAB next WE and give report ASAP

btw. very nice Explanation

have a nice Day!

DigiH commented 7 months ago

Thanks for the beacon model, which I have also now included in the decoder, so your Feasycom beacons should be correctly recognised when you install the development test binary on your ESP32 next weekend.

All the best.

lux73 commented 7 months ago

Hi @DigiH

sorry for the delayed answer 🤐

i have succesfully flashed esp32dev-ble via Webflasher - after adjust my things, items & rules (OpenHAB) anything is up'n running again

my Feasycom is completely recognised now - Thanks for the implementation!

now i'm curious about the 24h reboots - but i can confirm all Settings will survive a reboot now 😎

we'll see tomorrow 🕓

have a nice Day!

DigiH commented 7 months ago

Hi @lux73

Good to hear everything is running fine for you with the correct esp32dev-ble binary and the Feasycom decoder integration:)

Let's see tomorrow about any possible reboots.

Have a good one!

lux73 commented 7 months ago

Hi @DigiH

Sorry to say but even with the mentioned correct Firmware my ESP did not survive more than 24h with 'adaptivescan: false'

Which Information/Logfiles do you need?

DigiH commented 7 months ago

Hi @lux73

ESP32s with ble firmware do sometimes restart automatically, this seems to depend on the make/model of the ESP32, the power adapter being used to provide a constant clean 5V, if a white-list is being used or not and which other Bluetooth devices might be in the vicinity … .

I have the same here, with one of my ESP32 restarting every now and then, while another one keeps running happily without any restarts. I put it down to manufacturing quality variances of these cheap ESP32 modules ;) and yes, the WROOM dev kits seem to be particularity finicky with that , you using WROOM 32U, me WROOM 32D.

Such restarts should not really be an issue in any running controller set-up, as OMG is up again running within a couple of seconds - with all saved settings properly restored.

So is yours just restarting every now and then, but immediately back up again with all the settings correctly intact, or do you see any settings issues after such restarts?

Could you also post your current BTtoMQTT settings from MQTT Explorer?

lux73 commented 7 months ago

but if it's a Hardware fault it should be restarts either with adaptivescan: false & adaptivescan: true but it only reboots after exactly 24h runtime with adaptivescan: falase ^^

that makes no sense for me, sorry 😉

seems to me a Register or Timer is overflowed - so why the System is exactly rebooting every 24h

not all Settings are restored: e.g. whitelist is not stored, after change from adaptivescan: false to :true the interval & intervalacts is not restored

BTtoMQTT

{"bleconnect":false,"interval":5000,"adaptivescan":false,"intervalacts":100,"intervalcnct":3600000,"scanduration":10000,"onlysensors":false,"randommacs":false,"hasspresence":false,"prestopic":"presence/","presuseuuid":false,"minrssi":-120,"extDecoderEnable":false,"extDecoderTopic":"undecoded","filterConnectable":false,"pubadvdata":false,"pubuuid4topic":false,"ignoreWBlist":false,"presenceawaytimer":120000,"movingtimer":60000,"forcepscn":false,"tskstck":1648,"crstck":3060,"enabled":true,"scnct":7808}

i think i will revert back to the undecoded Firmware - because the mentioned Firmware from you is additionally flooding my mqtt Broker with homeassistant Topics even i have disabled disc: false & ohdisc: false that i don't want...

homeassistant-topic

there are more than 20 BT Devices found in my neighborhood - and the whitelist is not surviving the 24h reboot. it adds over time so much topics

esp32/OMG_ESP32_BLE/BTtoMQTT/0CC844294A03
esp32/OMG_ESP32_BLE/BTtoMQTT/105DDC9467B9
esp32/OMG_ESP32_BLE/BTtoMQTT/1094971DAA18
esp32/OMG_ESP32_BLE/BTtoMQTT/4142D317EFAC
esp32/OMG_ESP32_BLE/BTtoMQTT/4410BACC70C3
esp32/OMG_ESP32_BLE/BTtoMQTT/44281E5415B4
esp32/OMG_ESP32_BLE/BTtoMQTT/46080F55808A
esp32/OMG_ESP32_BLE/BTtoMQTT/480B812498FB
esp32/OMG_ESP32_BLE/BTtoMQTT/48D0A436360C
esp32/OMG_ESP32_BLE/BTtoMQTT/49D99CEC5473
esp32/OMG_ESP32_BLE/BTtoMQTT/4B0232F9800B
esp32/OMG_ESP32_BLE/BTtoMQTT/4E18F1DF387A
esp32/OMG_ESP32_BLE/BTtoMQTT/552F5A8EE1D1
esp32/OMG_ESP32_BLE/BTtoMQTT/586D6D3C7C43
esp32/OMG_ESP32_BLE/BTtoMQTT/587979303B53
esp32/OMG_ESP32_BLE/BTtoMQTT/5915A1BB4E39
esp32/OMG_ESP32_BLE/BTtoMQTT/59893FE8F633
esp32/OMG_ESP32_BLE/BTtoMQTT/5B5A7AD0D9A0
esp32/OMG_ESP32_BLE/BTtoMQTT/646E74CC71C0
esp32/OMG_ESP32_BLE/BTtoMQTT/648829BEA11F
esp32/OMG_ESP32_BLE/BTtoMQTT/65119054B27E
esp32/OMG_ESP32_BLE/BTtoMQTT/6749CB88B8A9
esp32/OMG_ESP32_BLE/BTtoMQTT/68FCCACEF041
esp32/OMG_ESP32_BLE/BTtoMQTT/69F2F3D758CF
esp32/OMG_ESP32_BLE/BTtoMQTT/6AADA8A34951
esp32/OMG_ESP32_BLE/BTtoMQTT/6E765A6AFF15
esp32/OMG_ESP32_BLE/BTtoMQTT/6FF075422811
esp32/OMG_ESP32_BLE/BTtoMQTT/702218614164
esp32/OMG_ESP32_BLE/BTtoMQTT/70AC4C07112E
esp32/OMG_ESP32_BLE/BTtoMQTT/798C7502701E
esp32/OMG_ESP32_BLE/BTtoMQTT/7C517F952FE3
esp32/OMG_ESP32_BLE/BTtoMQTT/7CE6067CBE59
esp32/OMG_ESP32_BLE/BTtoMQTT/B88E824504E0
esp32/OMG_ESP32_BLE/BTtoMQTT/C0288D44AB7F
esp32/OMG_ESP32_BLE/BTtoMQTT/C670455807B6
esp32/OMG_ESP32_BLE/BTtoMQTT/D2520D37AFFB
esp32/OMG_ESP32_BLE/BTtoMQTT/D8B673ED2A4F
esp32/OMG_ESP32_BLE/BTtoMQTT/DC0D3016CF1A
esp32/OMG_ESP32_BLE/BTtoMQTT/E4C10A8FC993
esp32/OMG_ESP32_BLE/BTtoMQTT/E6B694880959
esp32/OMG_ESP32_BLE/BTtoMQTT/F328CF9E280C

with the undecoded Firmware only the undecoded Topic is flooded 😉

DigiH commented 7 months ago

If you feel happier with the undecoded binary I won't stop you, but it is definitely the most awkward way to achieve what I think you are trying to do with tracking the presence of your Feasycom beacons in OpenHAB.

Still let me point out a few misconceptions, likely mistakes and definite possibilities for improvement from what you wrote above.

whitelist is not stored

Luckily the white-list is not actually stored on any OMG gateway device, so it is it is completely separated from any possible ESP32 issues, but needs to be published to the MQTT broker with the retain flag, as documented. Any failure for a white-list to be re-applied after a restart usually is due to a user error of omitting the retain flag when publishing a white-list. 😎

While I don't know of viewing retained messages in MQTT Explorer off hand, an alternative MQTT viewer I also used, called MQTT.fx nicely shows all retained messages at the top of the message list, as soon as subscribing to a topic.

"interval":5000,"adaptivescan":false,"intervalacts":100,"intervalcnct":3600000,"scanduration":10000,

With you having set "intervalacts":100 it means that your gateway does a 10 second ("scanduration":10000) scan with only ever 1/10 second break in between. So you're really giving it a run for its money, which is fine, but this kind of continuous scanning is only required if you want to catch the the broadcasts from contact or motion sensors - in such cases you would also want to set the scanduration lower to about 1 second - but that is a whole different scenario, not really applicable here.

after change from adaptivescan: false to :true

And yes, you want/need to keep adaptivescan:false at all times, and don't toggle it at all, as yes, then OMG will expectedly reset the intervals and scan duration again to account for possible contact or motion sensors, or even existing ones from your neighbours if if a white-list hasn't been applied after a restart due to the missing retain flag.

Assuming that you currently only want to receive your FeasyBeacons, which do not even require active scanning, your best bet would be something like "intervalacts":3600000

Then with the remaining "interval":5000 and "scanduration":10000 it means that you will published messages for your beacons every 15 seconds, generally more than enough for any kind of beacon tracking - personally I even have it set to "interval":20000 and "scanduration":10000, as once every 30 seconds is enough for how I use my beacons.

because the mentioned Firmware from you is additionally flooding my mqtt Broker with homeassistant Topics even i have disabled disc: false & ohdisc: false that i don't want...

These discovery messages are most likely remnants from before you set disc: false, as the default of the pre-built binaries is disc: true, so there would have been some initial discovery messages. all easily and quickly removed by deleting the whole homeassistant topic in MQTT Explorer.

So with a correctly published retained white-list, the suggested changes to BTtoMQTT and SYStoMQTT your BLE OpenMQTTGateway should work correctly, as it does for many others, when always making sure that these changes are also confirmed and saved to flash with "save":true.

The only other thing I can think of, though being very unlikely, is that your flash got somehow corrupted, even when selecting the erase flash option during the web upload. In such a case a separate flash erase through the Espressif site might be in order

https://espressif.github.io/esptool-js/

then correctly setting and saving all parameters as mentioned above for the esp32dev-ble binary.

In the end the decision is yours, go back to the more awkward undecoded version, not at all recommended for this kind of scenario, in which can you may also close this issue, or learn just a little bit more about the various OpenMQTTGateway settings, which admittedly can be a bit overwhelming when starting out with OMG, and have an ideal BLE gateway for your OpenHAB integration 😉

lux73 commented 7 months ago

Hi @DigiH

for the moment i had decided to go back to the esp32dev-ble-mqtt-undecoded-firmware.bin which it seems works best for me

i give it a try next few Month if new Firmware Versions are available and can live with the recurring 24h reboot for now 😉

i really appreciate this Project and would like say Thank You! for your Support 👍

Greetings from Germany