Open BlackEdder0815 opened 8 months ago
Hi @BlackEdder0815,
Could you set Advertisement and Advanced Data to true.
Then you will get the additional raw undecoded advertising data when your SwitchBot Outdoor Meter is not being decoded.
Posting some of these messages here will give us a better understanding as to why it might only be decoded properly occasionally for you.
Also the SwitchBot Outdoor Meter requires to be scanned actively.
What is the BTtoMQTT key setting for "intervalacts"? It is only in this interval (in ms) that the SwitchBot Outdoor Meter will be scanned actively and therefore be able to be decoded correctly.
Very likely it is a high setting of "intervalacts" on your gateway that causes the issue you are seeing.
Thanks for the very quick response. the „intervalacts“ parameter is set to default. So it‘s 55555. I will reduce the setting and will re-try.
Then you should really see properly decoded messages every 55 seconds.
Please try setting the above mentioned advertising data to true and then post some messages including the advertising data which are not decoded.
Because of x-mas it took a moment to get back to this project. I enabled the "advaced data" accordingly and set the intervalacts to 1000. So the data for this now contains the following:
{
"id": "D5:4F:3B:88:E8:1E",
"mac_type": 1,
"adv_type": 0,
"manufacturerdata": "6909d54f3b88e81e6703009a2800",
"rssi": -29
}
and the log contains the following:
N: Send on /BTtoMQTT/D54F3B88E81E msg {"id":"D5:4F:3B:88:E8:1E","mac_type":1,"adv_type":0,"manufacturerdata":"6909d54f3b88e81e6703009a2800","rssi":-29}
...
N: Device detected: D5:4F:3B:88:E8:1E
N: Send on /BTtoMQTT/A4C1383B1EC8 msg {"id":"A4:C1:38:3B:1E:C8","mac_type":0,"adv_type":0,"rssi":-92,"servicedata":"a4c1383b1ec800d137550b97d5","servicedatauuid":"0x181a","brand":"Xiaomi","model":"TH Sensor","model_id":"LYWSD03MMC/MJWSD05MMC_ATC","type":"THB","tempc":20.9,"tempf":69.62,"hum":55,"batt":85,"volt":2.967,"mac":"A4:C1:38:3B:1E:C8"}
N: Send on /BTtoMQTT/D54F3B88E81E msg {"id":"D5:4F:3B:88:E8:1E","mac_type":1,"adv_type":0,"manufacturerdata":"6909d54f3b88e81e6703009a2800","rssi":-32}
Thanks, the manufacturerdata is not taken into account for the decoding, but the servicedata is, and seems to be correctly decoded in your above posting.
Are you getting regular decoded servicedata readings now with your reduced intervalacts? Even though for most uses cases with the previous default interval 55555 is fine with an update every 55 seconds.
unfortunately not. So at least the latest mqtt-message doesn't contain the full data. See the logs. - the relevant device is the D5:4F:3B:88:E8:1E. There is a first mqtt-send send with all the data incl. temp, hum etc. but a second send at the end directly overwrites the entry with an nearly empty post. So I can make a code-snippet on my side to only check the data of "full" mqtt-messages, but I guess this is not intended. If does I have to live with those short messages, which contains only the manufacturer data (undecoded)?
N: Device detected: D5:4F:3B:88:E8:1E
N: Send on /BTtoMQTT/A4C138D926E4 msg {"id":"A4:C1:38:D9:26:E4","mac_type":0,"adv_type":0,"rssi":-75,"servicedata":"a4c138d926e400d33a530b8de0","servicedatauuid":"0x181a","brand":"Xiaomi","model":"TH Sensor","model_id":"LYWSD03MMC/MJWSD05MMC_ATC","type":"THB","tempc":21.1,"tempf":69.98,"hum":58,"batt":83,"volt":2.957,"mac":"A4:C1:38:D9:26:E4"}
N: Send on /BTtoMQTT/D54F3B88E81E msg {"id":"D5:4F:3B:88:E8:1E","mac_type":1,"adv_type":0,"manufacturerdata":"6909d54f3b88e81e7a0f029a2900","rssi":-29}
N: Send on /SYStoMQTT msg {"uptime":370698,"version":"v1.7.0","disc":false,"ohdisc":false,"env":"esp32dev-ble","freemem":89360,"mqttp":"1883","mqtts":false,"msgprc":19390,"msgblck":0,"maxq":6,"minmem":34144,"tempc":55,"freestck":2104,"eth":false,"rssi":-60,"SSID":"Sona_SmartHome","BSSID":"EA:63:DA:B4:C4:29","ip":"192.168.3.30","mac":"C0:49:EF:CD:DB:18","lowpowermode":-1,"modules":["WebUI","BT"]}
N: Send on /BTtoMQTT msg {"bleconnect":true,"interval":55555,"adaptivescan":true,"intervalacts":1000,"intervalcnct":3600000,"scanduration":10000,"onlysensors":false,"randommacs":false,"hasspresence":false,"prestopic":"presence/","presuseuuid":false,"minrssi":-100,"extDecoderEnable":false,"extDecoderTopic":"undecoded","filterConnectable":false,"pubadvdata":true,"pubuuid4topic":false,"ignoreWBlist":false,"presenceawaytimer":120000,"movingtimer":60000,"forcepscn":false,"tskstck":2600,"crstck":3004,"enabled":true,"scnct":5649}
N: Send on /WebUItoMQTT msg {"displayMetric":true,"webUISecure":true,"displayQueue":0}
N: Scan begin
N: Device detected: 6D:80:4C:37:E5:CA
N: Found N2 : Ddevices, scan numeber 5650 end
N: Found N2 : Ddevices, scan numeber 5650 end
N: Send on /BTtoMQTT/D54F3B88E81E msg {"id":"D5:4F:3B:88:E8:1E","mac_type":1,"adv_type":0,"manufacturerdata":"6909d54f3b88e81e7a0f029a2900","rssi":-29}
Ah wait, I was confused by the sys-message, what was following. Also the first one had no decoded values. So I still have the issue, that only in rare situations the data is decoded.
"intervalacts":1000
really is a bit much, trying active scanning every second. This could also be detrimental to the battery life of your devices.
Maybe try with something like 15000 to see what you get every 15 seconds. But somehow it seems that your SwitchBot Outdoor Meter mainly broadcasts advertisement data with manufacturerdata and not so much with servicedatas. As we have not heard of this from other Outdoor Meter users I am wondering if it might be a setting in the SwitchBot App.
Actually my bad! Ignore my previous posts, as the servicedata is only used for the battery level of the Outdoor Meter, but the manufacturerdata should have all the reaming data for temperature and humidity - am traveling at the moment, so not being at my usual desk makes things a bit more awkward ;)
There should however be servicedata and manufacturerdata together in the broadcasts, only then will they be correctly recognised and decoded. This somehow seems to only happen rarely for your Outdoor Meter, again not really sure for what reason.
Can you just make sure that this is the case and post a message with decoded SwitchBot Outdoor Metter data when you get it occasionally, still the the advertising data switched on. This should then show both servicedata and manufacturerdata.
Also please still look in the SwitchBot App, to see if any settings change there might produce more frequent broadcasts with both these advertising data included, so that it will be decoded more frequently.
Otherwise I might also look into the possibility of using just the manufacturerdata to recognise and decode the Outdoor Meter, and then only occasionally also get the battery level in the case of both data being received together. Will have tyo wait until I'm back home though.
Thank you very much for your support. I have three Outdoor Meter devices. Currently two enabled by plugging in the battery. They are currently on factory settings. I only paired one for a time, and "de-paired" them later again. I will check, if the behavior is changing, when the device is paired with the app. Also will look for some settings. Without the service data, you will not get any information about the battery. But the rest should work.
Found no settings in the app to change the behavior of the devices. Also pairing didn’t change the behavior.
interesting: using BLE Hero (iOS) I see the full Bluetooth information.
Maybe there is another issue, as I have some artifacts inside the logs. Or is this currently a bug? See this:
N: NF: oDund evi5ce detected: D5:4F:3B:88:E8:1E
N: NF: oDund evi5ce detected: D5:4F:3B:88:E8:1E
N: Device detected: A4:C1:38:EE:4C:EB
N: Device detected: 75:8E:CB:AB:46:A7
But those artifacts seems only in the text existing, not in the data what is written.
I'm wondering if there have been some changes in 1.7.0 with publishing the manufacturerdata and servicedata together. @1technophile might have more insight on this when he is back for his holidays.
Until then, could you try it with version 1.6.0, but also with the current development version, for which the test binaries are available at
https://docs.openmqttgateway.com/dev/upload/web-install.html
Either way, we would need some sample messages of all three of your Outdoor Meters, with the manufacturerdata, but preferably also messages containing manufacturerdata and servicedata, which would then also be decoded, so please make sure to keep Advertisement and Advanced Data set to true for all OMG versions.
It should make it easier if you use MQTT Explorer, instead of having to watch the serial logging. In MQTT Explorer you can go through the History of messages for each Meter to find the combined manufacturerdata and servicedata messages.
I hope we can find out the issue with the combined messages, as the servicedata actually does contain the Device Type ID which uniquely identifies a devices as the Outdoor Meter.
We would need to know what is the frequency of decoded message and the associated BTtoMQTT settings.
It is the regular behavior to have undecoded messages overwritting others unless the option onlysensors
is set to true
https://docs.openmqttgateway.com/use/ble.html#setting-if-the-gateway-publishes-all-the-ble-devices-scanned-or-only-the-detected-sensors-default-false-available-with-ha-discovery
Ok, continuing the research. I switched to the latest dev right now (setup-from-scratch). Also I fired up mqtt-explorer to get time-stamps. I added the "only sensors" flag, but then the related device/topic is not updated anymore. So it's not discovered as "sensor". After disabling it again, the topic is updated again. I still get no decoded sensor values. I don't know why. I saw them already a few times, but currently there is no decoding happening. I will further monitor it with mqtt-explorer.
Here is a short history log of the topic. All timings are in default, but I will add the current configuration for reference.
So all updates are happening quite regulare after 65 seconds. Sometimes I have some more time between to updates, like one time 3min 28 seconds. But the very most cases are after 65 seconds.
settings
uptime 1047
version "f2d323"
disc true
ohdisc false
env "esp32dev-ble"
freemem 91892
mqttp "1883"
mqtts false
msgprc 54
msgblck 0
maxq 4
minmem 31788
tempc 53.33
freestck 2104
eth false
rssi -75
SSID "########"
BSSID "xx:xx:xx:xx:xx:xx"
ip "192.168.x.x"
mac "C0:49:EF:xx:xx:xx"
lowpowermode -1
modules "'WebUI', 'BT'"
origin "/SYStoMQTT"
BT
bleconnect true
interval 55555
adaptivescan true
intervalacts 55555
intervalcnct 3600000
scanduration 10000
onlysensors false
randommacs false
hasspresence false
prestopic "presence/"
presuseuuid false
minrssi -100
extDecoderEnable false
extDecoderTopic "undecoded"
filterConnectable false
pubadvdata true
pubuuid4topic false
ignoreWBlist false
presenceawaytimer 120000
movingtimer 60000
forcepscn false
tskstck 2912
crstck 3060
enabled true
scnct 15
WebUI
displayMetric true
webUISecure true
displayQueue 0
@BlackEdder0815 could you still do another test with the 1.6.0 release?
And I assume this is the same behaviour for all three of your Outdoor Meters?
Took me some time to get 1.6.0 running on the ESP32. But I managed the installation. I have the same behavior in the 1.6.0. Here is an example:
{
"id":"D5:4F:3B:88:E8:1E",
"mac_type":1,
"adv_type":0,
"manufacturerdata":"6909d54f3b88e81e6e0303962d00",
"rssi":-64
}
The history looks the same like in 1.7.0.
I started my third Outdoor Meter (fresh, completely factory-state, first time battery contact) and here is the behavior a bit different. See the logs. The very first advertisement, and the second one:
{
"id":"E0:50:45:C5:F0:B2",
"mac_type":1,
"adv_type":0,
"manufacturerdata":"6909e05045c5f0b2020300944400",
"rssi":-85,
"servicedata":"7700e4",
"servicedatauuid":"0xfd3d",
"brand":"SwitchBot",
"model":"Outdoor Meter",
"model_id":"W340001X",
"type":"THB",
"acts":true,
"tempc":20,
"tempf":68,
"hum":68,
"batt":100
}
And second:
{
"id":"E0:50:45:C5:F0:B2",
"mac_type":1,
"adv_type":0,
"manufacturerdata":"6909e05045c5f0b20b0309953700",
"rssi":-76
}
And while i'm typing, I got another decoded message.
But this behavior is only for the unpaired, fresh factory-state Meter valid.
After ~1,5h running, I only got those two decoded messages above. The rest were just the manufactorerdata - and those nearly every 65 seconds. So after the initial activation of the Meter, there ar two messages decoded, and the rest are un-decoded. But using BLE-Hero app on the iphone I see the related Manufactorerdata and ServiceData.
Thanks for all the further testing @BlackEdder0815!
As I have not heard this from Outdoor Meter users before, especially also not when initially implementing this decoder with the help of users with the device, I suspect your devices to be pretty new with an already installed newer firmware, behaving slightly different than the versions before - something not unheard of with SwitchBot ;)
from your finding it seems that the servicedata is only being broadcast in the beginning of the activation of a device, and then possibly only afterwards if and when the battery level has changed, which could be quite a while, as the battery level is the only encoded data left in the servicedata.
I suspect that BLE Hero is also only showing the manufacturerdata when you run it while OMG in MQTT Explorer is only showing manufacturerdata messages?
As the Outdoor Meter was also the first/only device which separated its broadcast data into servicedata and manufacturerdata, at least for the ones we have decoders for so far, things did get a bit more convoluted and unclear.
Anyway, as others older Outdoor Meters are likely to get the new firmware as well and to be able to allow for decoding of as many Meters as possible.
Took me some time to get 1.6.0 running on the ESP32. But I managed the installation.
Am I correct in assuming that you managed to create your own 1.6.0 build with PlatformIO/Visual Studio Code?
If so, it will be very easy for you to test out a new Outdoor Meter test branch I have created, by only changing the line
decoder = https://github.com/theengs/decoder.git#v1.5.0
in the platformio.ini
file to
decoder = https://github.com/DigiH/decoder.git#sbot
I you managed to install 1.6.0 some other way let us know and we can kick off a test build for you to install.
Either way, the changed decoder should correctly decode all messages for you now and we're looking forward to your confirmation.
Thanks for the whole support and efforts. Related to BLE Hero: It seems, that BLE Hero is always able to discover the service data. See this post. I can repeat this for every sensor every time. Also: BLE Hero is able to find sensors, which are a bit more away (lower signal strength), bit this is another topic and we shouldn't mix this (might also be IPhone related) If you can trigger a build, this would help. I flashed the ESP32 with the official release-build by using the ESP-Flasher incl. boot- and partition-image.
Related to BLE Hero: It seems, that BLE Hero is always able to discover the service data. See this post.
Yes, I was wondering though if BLE Hero saves the servicedata once it has been received just once, which also seemed to have happened for you with OMG initially, and then just stores the last received servicedata if and until the next braodcasts which includes it comes in. As the only changing value in the servicedata for the Outdoor Meeter is now the battery level it's a bit hard to tell, as it only changes slowly over time.
One thing I did want to ask you to then try with the test decoder build, is to put the Outdoor Meter into your freezer, to hopefully get a bit of strain on the battery, and while monitoring with MQTT Explorer to see if a battery change does turgger another broadcast which included the servicedata as well.
@1technophile @h2zero - any other idea what might be the issue that BLE Hero always shows the servicedata, while OMG only seems to receive manufacturerdata only in these cases. Which definitely wasn't the case when initially implementing the Outdoor Meter in this HA thread
https://community.home-assistant.io/t/switchbot-outdoor-thermometer-hygrometer/565464/5
I'll kick off a test build now, which will take about 90 minutes, and post here once it is available, as it will also be overwritten again shortly after midnight UTC by the default nightly dev builds.
@BlackEdder0815
The test build with SHA:b095c5 is now available at
https://docs.openmqttgateway.com/dev/upload/web-install.html
until shortly after midnight UTC.
Thanks a lot @DigiH . I installed the related build and get the expected data w/o battery information.
{
"id":"E0:50:45:C5:F0:B2",
"mac_type":1,
"adv_type":0,
"manufacturerdata":"6909e05045c5f0b2510308963300",
"rssi":-62,
"brand":"SwitchBot",
"model":"Outdoor Meter",
"model_id":"W340001X",
"type":"THB",
"acts":true,
"tempc":22.8,
"tempf":73.04,
"hum":51,
"mac":"E0:50:45:C5:F0:B2"
}
Related to BLE Hero: I don't think that it stores the data. Also after a reset, the data appears immediately. I don't know why, but I get know some times a also a service data. The one sensor 2 times in the last half hour, the other 1 time. Was there any further change in the build? Maybe some timings during parsing the raw-signal?
Good that you are getting frequently decoded temperature and humidity now, and the battery level should be updated less frequently, whenever the servicedata is also being broadcast/received.
As you have previously also tried the development build there were not other changes since then to the code, apart from the new decoder reference in this current test build.
As some other SwitchBot devices also send out manufacturerdata only broadcasts AFAIK, but we usually only looked at their servicedata for the decoding, I will have to try and find some manufacturerdata only messages for these devices to make sure we don't accidentally recognise and decode any of them with the amended Outdoor Meter decoder now, before merging the changes into the development branch.
You can just keep running this test build on your ESP32 for now, until the amended decoder has been merged into Theengs Decoder and then also referenced in OpenMQTTGateway.
P.S. you don't have any other SwitchBot devices by any chance, for the verification of possible manufacturerdata only broadcasts from them, do you?
Short feedback: The build b095c5 runs smoothe since I installed it. All values are coming in as expected. Also the battery information (but not everytime - but that's OK I guess).
Thanks for the feedback! Unfortunately some bad new regarding this amended decoder.
I have looked more into the adjusted decoder, and I'm afraid I won't be able to merge it into the development branch, as other SwitchBot devices are sending the same manufacturerdata - without any servicedata/uuid - so that the manufacturerdata on its own can be mistakenly be decoded from any other of these devices.
While you only seem to have the Outdoor Meter this is not an issue for you, anyone else with some of the other SwitchBot devices though, or even in their neighbourhood could get lots of misinterpreted data from the other devices.
So the servicedata/uuid is really needed to uniquely identify the Outdoor Meter, or any other SwitchBot device for that matter, as it contains the device ID.
Also the battery information (but not everytime - but that's OK I guess)
How often do you get the battery information? As this is encoded in the servicedata it also means that the SwitchBot Outdoor Meter can be uniquely identified then.
Hm, that's not so good news. I started a session with MQTT-explorer, as my IObroker doesn't record historical time-stamps. Currently I have all three Outdoor-Meters in use. One outdoor, one in the freezer and one in the living room. So let's see what is the update cycle of the bat-value. Still interesting: I didn't get a bat-value with the old stock FW. With the current patched one, there is a bat at least some times. If just the decoder changed with those lines - I do not understand why it's now working....
OK, time between two decoded bat-messages
sensor outdoor: 15 mins, 50 mins, 2 mins,
living-room-sensor: every minute or two minutes
freezer: no bat frame at all.
The living-room-sensor is directly next to the receiver (5 cm). The outdoor-sensor is 6 meters and one wall. The freezer is 5 meters, one wall and of course the isolation and metal-case of the freezer. Maybe there are decoding/timing-issues because of the distance?
@h2zero - could these reception differences cause the difference in receiving manufacturerdata only compared to manufacturerdata and servicedata/uuid?
@BlackEdder0815 - thanks for this details information. Would you mind trying placing the worst case, i.e. the freezer Outdoor Meter, next to the living room one, just to see that it really is the distance/shielding and not due to the actual different devices?
Yes, switching the outdoor-meter, the issue will stay at the position. Means: The meter directly next to the gateway works like a charm and nearly everytime the battery will be extracted (sometimes not). The battery of the meter outdoor is just a few times decoded. The same device next to the gateway is nearly everytime decoded. So seems to be related to the distance....
The only thing I can think about now with the distance making the only difference, is that the Outdoor Meter does require active scanning, which in turn requires the OMG gateway to send an active scan request to the device.
So while the reception might be fine of strong signals of the Outdoor Meter, it could be that your gateway ESP32 might not send out a strong enough active scan signal to the Outdoor Meter to respond with both the manufacturerdata and the servicedata/uuid, but then only picks up the manufacturerdata, which it always sends out, and which can be picked up by passive scanning.
What kind of ESP32 do you use for your esp32dev-ble
gateway?
Do you possibly have other models of ESP32s with which you could see if you might get a better sending/reception, maybe even some with an external antenna?
Alternatively you might require a second esp32 OMG BLE gateway to place in a position so that the other two Outdoor Meters are also being picked up correctly.
This issue is stale because it has been open for 90 days with no activity.
I use a WROOM-32 board (see this offer)
i played already with the Active Scan interval. My mobile phone catches the related packages also from a bigger distance. Currently I have still the custom firmware in use with the small hack. This is working quite good for me. Decoding works and the battery information is sometimes also available, what is OK for my user case.
Thanks @DigiH for this PR https://github.com/theengs/decoder/pull/482, I made those changes to my OMG's TheengsDecoder library code and now I'm getting updates every 65 seconds from my SwitchBot Outdoor Thermometer/Hygrometer (instead of every 20 minutes).
I noticed it misses transmissions a lot, despite the ESP32 being literally 6 inches away from the SwitchBot. I just ordered an ESP32 with an external 2.4GHz antenna (Seeed Xiao ESP32C6), maybe it'll do better.
Thanks @DigiH for this PR theengs/decoder#482
Hi @melyux,
Glad to hear this is working better for you, but for the mentioned reasons this PR was not applicable to be merged into the Decoder development branch.
If you wouldn't mind testing a version which could make it into a release, I have this branch to allow for manufacturerdata or servicedata only, but still requires the name for unique identification.
It would be great if you could test the branch and let us know how frequently you receive your SwitchBot Outdoor Meter with it. Hopefully with the same frequency, so it can be merged.
https://github.com/DigiH/decoder/tree/sbot-test
You can easily test it by replacing the line
decoder = https://github.com/theengs/decoder.git#development
in platformio.ini, with
decoder = https://github.com/DigiH/decoder.git#sbot-test
Thanks
This issue is stale because it has been open for 90 days with no activity.
Sure thing, I'll test it when I have access to the device again
Hello community!
I use openmqttGW already for some temperature sensors and it works quite well. Now I tried to include SwitchBot outdoor temperature sensors, as they are listed on the "supported" list here
I have the issue, that the sensor is appearing in the mqtt list, but typically without data:
{ "id": "D5:4F:3B:88:E8:1E", "rssi": -28 }
No more information. Enabling the logs, there is something like
Sometimes, but only sometimes there will be the correct decoder selected and the correct values published. Next scan, the values are overwritten again with the minimum info above. Using a default bluetooth-scanner, the data-packages are looking good. All necessary information are transmitted. I tried already two different sensors and got the same behavior.
The logs are not helping much for debug. Seems that some of the checks are failing (but not always). As far as I can see, this sensor was introduced in 1.6.0. See this PR. I'm using 1.7.0.
Do you have any guidance how to debug or help to get this thing run smooth?