search

1technophile / OpenMQTTGateway

MQTT gateway for ESP8266 or ESP32 with bidirectional 433mhz/315mhz/868mhz, Infrared communications, BLE, Bluetooth, beacons detection, mi flora, mi jia, LYWSD02, LYWSD03MMC, Mi Scale, TPMS, BBQ thermometer compatibility & LoRa.
https://docs.openmqttgateway.com
GNU General Public License v3.0
3.61k stars 792 forks source link

Duplicate MQTT message with BLE Xiaomi Mijia Temperature Sensor MJ_HT_V1 #1040

Closed mhtrinh closed 3 years ago

mhtrinh commented 3 years ago

I have duplicate MQTT message when listenning for the MJ_HT_V1 report on my ESP32 devkit V4 Output from MQTT server: 

home/OpenMQTTGateway_ESP32_BLE_C/LWT online
home/OpenMQTTGateway_ESP32_BLE_C/version version_tag
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-88,"model":"LYWSDCGQ","tempc":15.5,"tempf":59.9,"hum":65.2}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-88,"model":"LYWSDCGQ","tempc":15.5,"tempf":59.9,"hum":65.2}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX47712 {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-89,"model":"LYWSDCGQ","tempc":21.2,"tempf":70.16,"hum":60.4}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX47712 {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-89,"model":"LYWSDCGQ","tempc":21.2,"tempf":70.16,"hum":60.4}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX53 {"id":"XX:XX:XX:XX:XX:53","rssi":-97}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-93}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX53 {"id":"XX:XX:XX:XX:XX:53","rssi":-97}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-92}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-96}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX5465E {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-59,"model":"LYWSDCGQ","tempc":20.9,"tempf":69.62,"hum":61.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX5465E {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-59,"model":"LYWSDCGQ","tempc":20.9,"tempf":69.62,"hum":61.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-86,"model":"LYWSDCGQ","tempc":15.5,"tempf":59.9,"hum":65.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-86,"model":"LYWSDCGQ","tempc":15.5,"tempf":59.9,"hum":65.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-73,"model":"LYWSDCGQ","tempc":19.4,"tempf":66.92,"hum":67.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-73,"model":"LYWSDCGQ","tempc":19.4,"tempf":66.92,"hum":67.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-91}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478BC {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":20.9,"tempf":69.62,"hum":62.9}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478BC {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":20.9,"tempf":69.62,"hum":62.9}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX53 {"id":"XX:XX:XX:XX:XX:53","rssi":-96}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX40CAF {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-91,"model":"LYWSDCGQ","tempc":21.2,"tempf":70.16,"hum":48.8}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX40CAF {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-91,"model":"LYWSDCGQ","tempc":21.2,"tempf":70.16,"hum":48.8}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":19.4,"tempf":66.92,"hum":67.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":19.4,"tempf":66.92,"hum":67.3}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-92}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX53 {"id":"XX:XX:XX:XX:XX:53","rssi":-95}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX2E {"id":"XX:XX:XX:XX:XX:2E","rssi":-99}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":15.4,"tempf":59.72,"hum":65.1}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX541BB {"id":"XX:XX:XX:X5:41:BB","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":15.4,"tempf":59.72,"hum":65.1}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":19.5,"tempf":67.1}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478B6 {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":19.5,"tempf":67.1}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-93}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478BC {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-89,"model":"LYWSDCGQ","hum":62.8}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXX478BC {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-89,"model":"LYWSDCGQ","hum":62.8}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX53 {"id":"XX:XX:XX:XX:XX:53","rssi":-95}
home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-96}
home/OpenMQTTGateway_ESP32_BLE_C/SYStoMQTT {"uptime":125,"version":"version_tag","freemem":153540,"freestack":3932,"rssi":-63,"SSID":"XXX","ip":"192.168.1.7","mac":"XXX","lowpowermode":0,"btqblck":0,"btqsum":242,"btqsnd":236,"btqavg":1.025424,"interval":0,"scanbcnct":10,"scnct":117,"modules":["BT"]}

This happen with both vanilla ble and ble-cont configuration:

[env:esp32dev-ble-vanilla]
platform = ${com.esp32_platform}
board = esp32dev
board_build.partitions = min_spiffs.csv
lib_deps =
  ${com-esp.lib_deps}
  ${libraries.ble}
build_flags =
  ${com-esp.build_flags}
  '-DZgatewayBT="BT"'
  '-DLED_SEND_RECEIVE=2'
  '-DLED_SEND_RECEIVE_ON=0'
  '-DGateway_Name="OpenMQTTGateway_ESP32_BLE"'
  '-DESPWifiManualSetup'
  '-Dwifi_ssid="XXX"'
  '-Dwifi_password="XXX"'
  '-DMQTT_USER="XX"'
  '-DMQTT_PASS="XX"'
  '-DMQTT_SERVER="192.168.1.20"'  
upload_protocol = espota
upload_port = 192.168.1.7
upload_flags =
  --auth=OTAPASSWORD
  --port=8266
upload_speed = 921600

[env:esp32dev-ble-cont-vanilla]
platform = ${com.esp32_platform}
board = esp32dev
board_build.partitions = min_spiffs.csv
lib_deps =
  ${com-esp.lib_deps}
  ${libraries.ble}
build_flags =
  ${com-esp.build_flags}
  '-DZgatewayBT="BT"'
  '-DLED_SEND_RECEIVE=2'
  '-DLED_SEND_RECEIVE_ON=0'
  '-DGateway_Name="OpenMQTTGateway_ESP32_BLE_C"'
  '-DTimeBtwRead=0'
  '-DScan_duration=1000'
  '-DAttemptBLECOnnect=false'
  '-DESPWifiManualSetup'
  '-Dwifi_ssid="XXX"'
  '-Dwifi_password="XX"'
  '-DMQTT_USER="XX"'
  '-DMQTT_PASS="XX"'
  '-DMQTT_SERVER="192.168.1.20"'  
upload_protocol = espota
upload_port = 192.168.1.7
upload_flags =
  --auth=OTAPASSWORD
  --port=8266
upload_speed = 921600

I don't think it's coming from the MJ_HT_V1 itself as I have a different code base running on RPi and I don't see this duplication.

I am using v0.9.7 compiled with Platformio.

h2zero commented 3 years ago

Thank you @mhtrinh for reporting this. I am trying to collect data regarding this. Could you tell me the approximate time between the duplicates?

mhtrinh commented 3 years ago

I would say instantaneous. Please let me know if you need any log (and any compilation flag that go with it, as I am new to OMG, since yesterday)

h2zero commented 3 years ago

Looking at the logs I can see that BLE_C is being used here and in your config is '-DTimeBtwRead=0' which will start a new scan immediately, coupled with the short scan time '-DScan_duration=1000' this could be expected behavior. Whenever a scan ends and is restarted the duplicate cache is cleared. With these settings you may be seeing the scan results from the last scan and within a second seeing the same results because the scan restarted immediately.

mhtrinh commented 3 years ago

But this happen also with the non-continuous version, that should be TimeBtwRead=55555 and Scan_duration=10000

home/OpenMQTTGateway_ESP32_BLE/SYStoMQTT {"uptime":244,"version":"version_tag","freemem":152588,"freestack":3924,"rssi":-62,"SSID":"XXX","ip":"192.168.1.7","mac":"XXX","lowpowermode":0,"btqblck":0,"btqsum":40,"btqsnd":32,"btqavg":1.25,"interval":55555,"scanbcnct":10,"scnct":4,"modules":["BT"]}
h2zero commented 3 years ago

Sorry, yes, you did state that and I skim read it (bad habit). I will have a look into it further and post back here.

DigiH commented 3 years ago

Looks like the same/similar experience as I commented with my Read BLE characteristics tests in #906 with every single duplicate being exactly 1 second (down to the millisecond ;) apart.

I have to admit though that I have become to appreciate these 'duplicates' now, since they don't just seem to be exact duplicates, but two separate messages, which with the READ functionality sometimes produce a single "success":false message, but the 2nd message containing "success":true to give me the desired result. This difference might also be the case with @mhtrinh as in

home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-93} … home/OpenMQTTGateway_ESP32_BLE_C/BTtoMQTT/XXXXXXXXXX5C {"id":"XX:XX:XX:XX:XX:5C","rssi":-92}

unless they are for different MAC addresses with the same ending.

mhtrinh commented 3 years ago

The MAC address are representative by their ending. I did make sure of that before replacing with XX

The messages with only rssi seems to come from a different type of device that I do not own, probably my neighbor ones. They seem to not have duplicate issue.

In a different test where I whitelist my devices, the duplicate still happen.

mhtrinh commented 3 years ago

Another log with timestamp so you have a better idea: 

1627422989.995632231 home/OMG/LWT: online
1627422989.995727912 home/OMG/version: version_tag
1627422990.042706595 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","hum":68.2}
1627422990.056958472 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","hum":68.2}
1627422992.531895962 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.1,"tempf":60.98,"hum":67.8}
1627422992.543064308 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.1,"tempf":60.98,"hum":67.8}
1627423004.062491154 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":68.2}
1627423004.115835941 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":68.2}
1627423004.575738542 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.8}
1627423004.583006131 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.8}
1627423004.985155174 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-76,"model":"LYWSDCGQ","hum":71.6}
1627423004.992741337 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-76,"model":"LYWSDCGQ","hum":71.6}
1627423006.525546669 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-81,"model":"LYWSDCGQ","tempc":15.9,"tempf":60.62,"hum":68.3}
1627423006.535729720 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-81,"model":"LYWSDCGQ","tempc":15.9,"tempf":60.62,"hum":68.3}
1627423006.978439025 home/OMG/BTtoMQTT/XXXXXXX1A483: {"id":"XX:XX:XX:X1:A4:83","rssi":-90}
1627423008.158378357 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-82,"model":"LYWSDCGQ","tempc":17.5,"tempf":63.5,"hum":70.3}
1627423008.165689051 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-82,"model":"LYWSDCGQ","tempc":17.5,"tempf":63.5,"hum":70.3}

1627423019.013218621 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-73,"model":"LYWSDCGQ","tempc":14.8,"tempf":58.64,"hum":71.5}
1627423019.061041854 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-73,"model":"LYWSDCGQ","tempc":14.8,"tempf":58.64,"hum":71.5}
1627423020.139246704 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-78,"model":"LYWSDCGQ","tempc":15.8,"tempf":60.44,"hum":68.2}
1627423020.147832872 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-78,"model":"LYWSDCGQ","tempc":15.8,"tempf":60.44,"hum":68.2}
1627423021.917949048 home/OMG/BTtoMQTT/XXXXXXX1A483: {"id":"XX:XX:XX:X1:A4:83","rssi":-90}
1627423022.188130380 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-84,"model":"LYWSDCGQ","tempc":17.5,"tempf":63.5,"hum":70.1}
1627423022.196221157 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-84,"model":"LYWSDCGQ","tempc":17.5,"tempf":63.5,"hum":70.1}
1627423022.598419733 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-81,"model":"LYWSDCGQ","tempc":15.9,"tempf":60.62,"hum":68.3}
1627423022.603625048 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-81,"model":"LYWSDCGQ","tempc":15.9,"tempf":60.62,"hum":68.3}
1627423025.976313710 home/OMG/SYStoMQTT: {"uptime":125,"version":"version_tag","freemem":153096,"freestack":4092,"rssi":-58,"SSID":"XXX","ip":"192.168.1.7","mac":"XXX","lowpowermode":0,"btqblck":0,"btqsum":53,"btqsnd":49,"btqavg":1.081633,"interval":6000,"scanbcnct":10,"scnct":7,"modules":["BT"]}
1627423026.603642461 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.5}
1627423026.701882698 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.5}

1627423036.217665743 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","hum":69.8}
1627423036.225735951 home/OMG/BTtoMQTT/XXXXXXX40CAF: {"id":"XX:XX:XX:X4:0C:AF","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","hum":69.8}
1627423036.626673680 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-77,"model":"LYWSDCGQ","hum":68.2}
1627423036.636555202 home/OMG/BTtoMQTT/XXXXXXX478B6: {"id":"XX:XX:XX:X4:78:B6","name":"MJ_HT_V1","rssi":-77,"model":"LYWSDCGQ","hum":68.2}
1627423037.037332389 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-70,"model":"LYWSDCGQ","tempc":14.9,"tempf":58.82}
1627423037.048557775 home/OMG/BTtoMQTT/XXXXXXX478BC: {"id":"XX:XX:XX:X4:78:BC","name":"MJ_HT_V1","rssi":-70,"model":"LYWSDCGQ","tempc":14.9,"tempf":58.82}
1627423037.657831135 home/OMG/BTtoMQTT/XXXXXXX1A483: {"id":"XX:XX:XX:X1:A4:83","rssi":-89}
1627423038.161841045 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":68.3}
1627423038.168886122 home/OMG/BTtoMQTT/XXXXXXX5465E: {"id":"XX:XX:XX:X5:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":68.3}
1627423040.619522536 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.3,"tempf":61.34,"hum":67.5}
1627423040.630692206 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.3,"tempf":61.34,"hum":67.5}

This was compiled with :

[env:esp32dev-ble-vanilla]
platform = ${com.esp32_platform}
board = esp32dev
board_build.partitions = min_spiffs.csv
lib_deps =
  ${com-esp.lib_deps}
  ${libraries.ble}
build_flags =
  ${com-esp.build_flags}
  '-DZgatewayBT="BT"'
  '-DLED_SEND_RECEIVE=2'
  '-DLED_SEND_RECEIVE_ON=0'
  '-DTimeBtwRead=6000'
  '-DGateway_Name="OMG"'
  '-DESPWifiManualSetup'
  '-Dwifi_ssid="XXX"'
  '-Dwifi_password="XXX"'
  '-DMQTT_USER="XXX"'
  '-DMQTT_PASS="XXX"'
  '-DMQTT_SERVER="192.168.1.20"'  
upload_protocol = espota
upload_port = 192.168.1.7
upload_flags =
  --auth=OTAPASSWORD
  --port=8266
upload_speed = 921600
mhtrinh commented 3 years ago

Tried with -DBLEScanDuplicateCacheSize=1000 as per #906 but still have the duplicate issue

h2zero commented 3 years ago

I'm not able to reproduce this. Here is a log before and after inserting some extra messages:

{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-70,"txpower":12}
{"id":"CC:B8:6C:D6:A8:82","name":"Temp/Hmid/Pres","rssi":-88}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-77,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-67,"txpower":12}
{"id":"65:6B:BF:17:FB:C3","rssi":-68,"txpower":12}
{"id":"65:6B:BF:17:FB:C3","rssi":-72,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-70,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"FB:B2:02:AC:5D:52","name":"ESP32","rssi":-68}
{"id":"65:6B:BF:17:FB:C3","rssi":-72,"txpower":12}
{"id":"65:6B:BF:17:FB:C3","rssi":-66,"txpower":12}
{"id":"F1:23:C6:A1:FC:77","name":"Flex","rssi":-85,"txpower":-6,"servicedata":"0706"}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
{"id":"65:6B:BF:17:FB:C3","rssi":-67,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-62,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"F1:23:C6:A1:FC:77","name":"Flex","rssi":-84,"txpower":-6,"servicedata":"0706"}
{"id":"65:6B:BF:17:FB:C3","rssi":-69,"txpower":12}

Extra messages:

Scan begin
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-65,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"CC:B8:6C:D6:A8:82","name":"Temp/Hmid/Pres","rssi":-91}
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
Scan end
Scan begin
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
Scan end
Scan begin
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
Scan end
Scan begin
{"id":"CC:B8:6C:D6:A8:82","name":"Temp/Hmid/Pres","rssi":-93}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
Scan end
Scan begin
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"FB:B2:02:AC:5D:52","name":"ESP32","rssi":-53}
{"id":"65:6B:BF:17:FB:C3","rssi":-68,"txpower":12}
Scan end
Scan begin
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
Scan end
Scan begin
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-67,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
{"id":"65:6B:BF:17:FB:C3","rssi":-70,"txpower":12}
Scan end
Scan begin
{"id":"65:6B:BF:17:FB:C3","rssi":-65,"txpower":12}
{"id":"49:C4:76:D7:A4:50","name":"Echo Dot-47M","rssi":-66,"txpower":1,"servicedata":"01f43f84784bd67ce4df9aed876737433430300310"}
Scan end

So what looks like duplicates is not actually duplicates, just that the scan restarted and found them again.

mhtrinh commented 3 years ago

I may be specific to certain device. In my latest example, message from XXXXXXX1A483 (which have no model name, just rssi) are not duplicated.

I believe it's not just left-over from the previous scan because the information is different between scan:

1627422992.531895962 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.1,"tempf":60.98,"hum":67.8}
1627422992.543064308 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.1,"tempf":60.98,"hum":67.8}
(12s gap : sleep between scan)
1627423004.575738542 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.8}
1627423004.583006131 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.8}
(22s gap : sleep between scan)
1627423026.603642461 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.5}
1627423026.701882698 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","tempc":16.2,"tempf":61.16,"hum":67.5}
(14s gap : sleep between scan)
1627423040.619522536 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.3,"tempf":61.34,"hum":67.5}
1627423040.630692206 home/OMG/BTtoMQTT/XXXXXXX47712: {"id":"XX:XX:XX:X4:77:12","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":16.3,"tempf":61.34,"hum":67.5}
mhtrinh commented 3 years ago

Below is a serial output with -DLOG_LEVEL=LOG_LEVEL_VERBOSE

N: Scan begin
T: Creating BLE buffer
N: Device detected: 58:2D:34:34:0C:AF
T: getDeviceByMac 58:2D:34:34:0C:AF
T: Get services data number: 2
T: Converted service data (18) to 5020aa01b2af0c34342d580d100466005103
T: Service data: 5020aa01b2af0c34342d580d100466005103
T: Service data UUID: 0xfe95
T: getDeviceByMac 58:2D:34:34:0C:AF
T: Checking BLE service data validity
T: Searching BLE device data 5020aa01b2af0c34342d580d100466005103 size 36
T: Is it a mokoBeacon?
T: Is it a mokoBeaconX Pro?
T: Is it a mi flora ?
T: Is it a vegtrug ?
T: Is it a LYWSDCGQ?
T: LYWSDCGQ data reading
T: getDeviceByMac 58:2D:34:34:0C:AF
T: add 58:2D:34:34:0C:AF
T: Valid data_length: 8
E: T: value Semaphore NOT taken
0.00
T: Launching discovery of T: value 58:2D:34:34:0C:AF
0.00
T: MiJiaDiscovery
T: value T: CreateDiscoverySensor MiJia-batt
0.00
T: Converted service data (6) to c2852a2ef24c
T: Pub json :T: Service data: c2852a2ef24c
{"stat_t":"+/+/ble/582D34340CAF","name":"MiJia-batt","uniq_id":"582D34340CAF-MiJia-batt","dev_cla":"battery","val_tpl":"{{ value_json.batt | is_defined }}","unit_of_meas":"%","device":{"connections":[["mac","582D34340CAF"]],"identifiers":["582D34340CAF"],"model":"LYWSDCGQ","name":"MiJia","via_device":"OMG"}} into custom topic: homeassistant/sensor/582D34340CAF-MiJia-batt/config
T: Service data UUID: 0xffff
T: getDeviceByMac 58:2D:34:34:0C:AF
T: Checking BLE service data validity
T: Searching BLE device data c2852a2ef24c size 12
T: Is it a mokoBeacon?
T: Is it a mokoBeaconX Pro?
T: Is it a mi flora ?
T: Is it a vegtrug ?
T: Is it a LYWSDCGQ?
T: Is it a JQJCY01YM?
T: Is it a LYWSD02?
T: Is it a MUE4094RT?
T: Is it a CGP1W?
T: Is it a CGG1
T: Is it a CGD1?
T: Is it a CGDK2?
T: Is it a CGPR1?
T: Is it a CGH1?
T: Is it a MHO_C401?
T: Is it a LYWSD03MMC?
T: Is it a custom (pvvx) LYWSD03MMC?
T: Is it a custom (atc1441) LYWSD03MMC?
T: servicedatauuid 0xffff
T: Is it a MiBand?
T: Is it a XMTZC04HM?
T: Is it a XMTZC05HM?
T: Is it an EddystoneTLM?
T: Converted service data (18) to 5020aa01b2af0c34342d580d100466005103
T: Buffersize for 0 : 336 -> 405 -> 447
T: getDeviceByMac 58:2D:34:34:0C:AF
T: Checking BLE service data validity
T: Searching BLE device data 5020aa01b2af0c34342d580d100466005103 size 36
T: Is it a mokoBeacon?
T: Is it a mokoBeaconX Pro?
T: Is it a mi flora ?
T: Is it a vegtrug ?
T: Is it a LYWSDCGQ?
T: LYWSDCGQ data reading
T: Valid data_length: 8
T: value 0.00
T: value 0.00
T: value 0.00
T: Removing servicedatauuid
T: Removing servicedata
T: Converted service data (6) to c2852a2ef24c
T: Buffersize for 1 : 336 -> 381 -> 423
T: getDeviceByMac 58:2D:34:34:0C:AF
T: Checking BLE service data validity
T: Searching BLE device data c2852a2ef24c size 12
T: Is it a mokoBeacon?
T: Is it a mokoBeaconX Pro?
T: Is it a mi flora ?
T: Is it a vegtrug ?
T: Is it a LYWSDCGQ?
T: Is it a JQJCY01YM?
T: Is it a LYWSD02?
T: Is it a MUE4094RT?
T: Is it a CGP1W?
T: Is it a CGG1
T: Is it a CGD1?
T: Is it a CGDK2?
T: Is it a CGPR1?
T: Is it a CGH1?
T: Is it a MHO_C401?
T: Is it a LYWSD03MMC?
T: Is it a custom (pvvx) LYWSD03MMC?
T: Is it a custom (atc1441) LYWSD03MMC?
T: servicedatauuid 0xffff
T: Is it a MiBand?
T: Is it a XMTZC04HM?
T: Is it a XMTZC05HM?
T: Is it an EddystoneTLM?
T: Removing servicedatauuid
T: Removing servicedata
T: CreateDiscoverySensor MiJia-temp
T: Pub json :{"stat_t":"+/+/ble/582D34340CAF","name":"MiJia-temp","uniq_id":"582D34340CAF-MiJia-temp","dev_cla":"temperature","val_tpl":"{{ value_json.tempc | is_defined }}","unit_of_meas":"°C","device":{"connections":[["mac","582D34340CAF"]],"identifiers":["582D34340CAF"],"model":"LYWSDCGQ","name":"MiJia","via_device":"OMG"}} into custom topic: homeassistant/sensor/582D34340CAF-MiJia-temp/config
T: CreateDiscoverySensor MiJia-hum
T: Pub json :{"stat_t":"+/+/ble/582D34340CAF","name":"MiJia-hum","uniq_id":"582D34340CAF-MiJia-hum","dev_cla":"humidity","val_tpl":"{{ value_json.hum | is_defined }}","unit_of_meas":"%","device":{"connections":[["mac","582D34340CAF"]],"identifiers":["582D34340CAF"],"model":"LYWSDCGQ","name":"MiJia","via_device":"OMG"}} into custom topic: homeassistant/sensor/582D34340CAF-MiJia-hum/config
N: Subject: /ble/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-84,"model":"LYWSDCGQ","tempc":10.2,"tempf":50.36,"hum":84.9}
T: jsonPublishing
N: Subject: /ble/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-84,"model":"LYWSDCGQ","tempc":10.2,"tempf":50.36,"hum":84.9}
T: jsonPublishing

I wonder if the issue come from the fact that the sensor advertise both temperature and humidity in one message and somehow OMG parse the message and will send one message for temperature and one for humidity, but with both information in there.

What is this homeassistant topic ? I don't have HASS on my network ...

DigiH commented 3 years ago

Hi @mhtrinh - please have a look at the issue I logged.

https://github.com/1technophile/OpenMQTTGateway/issues/1041

The fix from @h2zero also fixed my duplicate responses. See if it remedies it for you as well.

1technophile commented 3 years ago

What is this homeassistant topic ? I don't have HASS on my network ...

This topic is used to publish the sensor definition (list of attributes) so as to create it automaticaly into Home Assistant, if you want to remove this you can deactivate it at build time or at runtime.

I will check with my Mijia + the dev version tomorrow

mhtrinh commented 3 years ago

Nope. Issue still there. I pulled the latest development branch. Made sure that a4a85ff14b6fbbc92e23a3b8463d7c9602a6508a is there. Made sure that break; is in line 774:ZgatewayBT.ino

N: Subject: /SYStoMQTT
N: Received json : {"uptime":4,"version":"version_tag","freemem":147348,"freestack":4228,"rssi":-29,"SSID":"XXX","ip":"192.168.1.90","mac":"XXX","lowpowermode":0,"btqblck":0,"btqsum":0,"btqsnd":0,"btqavg":0,"interval":6000,"scanbcnct":10,"scnct":0,"modules":["BT"]}
N: Scan begin
N: Device detected: 58:2D:34:35:46:5E
E: Semaphore NOT taken
N: Subject: /BTtoMQTT/582D3435465E
N: Received json : {"id":"58:2D:34:35:46:5E","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","batt":35}
N: Subject: /BTtoMQTT/582D3435465E
N: Received json : {"id":"58:2D:34:35:46:5E","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","batt":35}
N: Device detected: 58:2D:34:34:0C:AF
E: Semaphore NOT taken
N: Subject: /BTtoMQTT/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":19.3,"tempf":66.74,"hum":68.1}
N: Subject: /BTtoMQTT/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-85,"model":"LYWSDCGQ","tempc":19.3,"tempf":66.74,"hum":68.1}
N: Device detected: 58:2D:34:34:78:B6
E: Semaphore NOT taken
N: Subject: /BTtoMQTT/582D343478B6
N: Received json : {"id":"58:2D:34:34:78:B6","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":17.7,"tempf":63.86,"hum":68.6}
N: Subject: /BTtoMQTT/582D343478B6
N: Received json : {"id":"58:2D:34:34:78:B6","name":"MJ_HT_V1","rssi":-79,"model":"LYWSDCGQ","tempc":17.7,"tempf":63.86,"hum":68.6}
N: Device detected: 58:2D:34:34:78:BC
E: Semaphore NOT taken
N: Subject: /BTtoMQTT/582D343478BC
N: Received json : {"id":"58:2D:34:34:78:BC","name":"MJ_HT_V1","rssi":-87,"model":"LYWSDCGQ","tempc":17.8,"tempf":64.04,"hum":67.8}
N: Subject: /BTtoMQTT/582D343478BC
N: Received json : {"id":"58:2D:34:34:78:BC","name":"MJ_HT_V1","rssi":-87,"model":"LYWSDCGQ","tempc":17.8,"tempf":64.04,"hum":67.8}
N: Device detected: 58:2D:34:35:41:BB
E: Semaphore NOT taken
N: Subject: /BTtoMQTT/582D343541BB
N: Received json : {"id":"58:2D:34:35:41:BB","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26}
N: Subject: /BTtoMQTT/582D343541BB
N: Received json : {"id":"58:2D:34:35:41:BB","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26}
N: Found 5 devices, scan number 1 end
N: BLE Connect begin
N: BLE Connect end
N: Scan begin
N: Device detected: 58:2D:34:35:46:5E
N: Subject: /BTtoMQTT/582D3435465E
N: Received json : {"id":"58:2D:34:35:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":17.9,"tempf":64.22,"hum":67.8}
N: Subject: /BTtoMQTT/582D3435465E
N: Received json : {"id":"58:2D:34:35:46:5E","name":"MJ_HT_V1","rssi":-83,"model":"LYWSDCGQ","tempc":17.9,"tempf":64.22,"hum":67.8}
N: Device detected: 58:2D:34:34:77:12
E: Semaphore NOT taken
N: Device detected: 58:2D:34:34:78:B6
N: Subject: /BTtoMQTT/582D34347712
N: Received json : {"id":"58:2D:34:34:77:12","name":"MJ_HT_V1","rssi":-88,"model":"LYWSDCGQ","tempc":18.1,"tempf":64.58,"hum":68}
N: Subject: /BTtoMQTT/582D34347712
N: Received json : {"id":"58:2D:34:34:77:12","name":"MJ_HT_V1","rssi":-88,"model":"LYWSDCGQ","tempc":18.1,"tempf":64.58,"hum":68}
N: Subject: /BTtoMQTT/582D343478B6
N: Received json : {"id":"58:2D:34:34:78:B6","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","batt":60}
N: Subject: /BTtoMQTT/582D343478B6
N: Received json : {"id":"58:2D:34:34:78:B6","name":"MJ_HT_V1","rssi":-80,"model":"LYWSDCGQ","batt":60}
N: Device detected: 58:2D:34:35:41:BB
N: Subject: /BTtoMQTT/582D343541BB
N: Received json : {"id":"58:2D:34:35:41:BB","name":"MJ_HT_V1","rssi":-82,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":78.5}
N: Subject: /BTtoMQTT/582D343541BB
N: Received json : {"id":"58:2D:34:35:41:BB","name":"MJ_HT_V1","rssi":-82,"model":"LYWSDCGQ","tempc":15.7,"tempf":60.26,"hum":78.5}
N: Device detected: 58:2D:34:34:0C:AF
N: Subject: /BTtoMQTT/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-92,"model":"LYWSDCGQ","tempc":19.3,"tempf":66.74,"hum":68}
N: Subject: /BTtoMQTT/582D34340CAF
N: Received json : {"id":"58:2D:34:34:0C:AF","name":"MJ_HT_V1","rssi":-92,"model":"LYWSDCGQ","tempc":19.3,"tempf":66.74,"hum":68}
N: Found 5 devices, scan number 2 end

When I look at my serial output, sometime I see that the output are munched. It feels like 2 threads are printing in the same time. Is this expected ?

h2zero commented 3 years ago

I believe this is a device issue. It seems this sensor sends multiple packets. I found this description https://blog.ostebaronen.dk/2019/09/mijia-ble.html.

mhtrinh commented 3 years ago

I have some code running on RPi that do passive scan and extract information out of those sensors.

First the a bash script that dump the scan : 

mkfifo $fifoFile

sudo hciconfig $dev reset
sudo hcitool -i $dev lewlclr

# Add to whitelist
for mac in $macList
do
    sudo hcitool -i $dev lewladd "$mac"
done

# Start the scan 
sudo hcitool -i $dev lescan --whitelist --passive --duplicates &>/dev/null &

# Start the dumping 
sudo hcidump -i $dev --raw -t > $fifoFile &

Then extract the dump : 


def parseData(data):    
    temp=None
    hum=None
    batt=None
    dataType=data[12]    
    if (dataType == "0D"):
        hexstr=data[16]+data[15]
        temp = int(hexstr,16)/10.
        hexstr=data[18]+data[17]
        hum=int(hexstr,16)/10.
        return f"temp={temp} hum={hum}"
    elif (dataType == "0A"):
        hexstr=data[15]
        batt = int(hexstr,16)
        return f"battery={batt}"
    elif (dataType == "06"):
        hexstr=data[16]+data[15]
        hum=int(hexstr,16)/10.
        return f"hum={hum}"
    elif (dataType == "04"):
        hexstr=data[16]+data[15]
        temp=int(hexstr,16)/10.
        return f"temp={temp}"

def main(dumpFile):
    fd=open(dumpFile,"r")
    while True:
        line = fd.readline()      
        if line.strip().startswith("FE 50 20 AA 01"):            
            data=line.strip()
            mac=getMac(data)            
            extract=parseData(data.split())
            printDebug(f"{mac} - {data} - {extract}")

if __name__ == '__main__':
    main(sys.argv[1])

This is the output so you have an idea of freqency of messages (no duplicate message within 20ms as in OMG):

2021-07-29 18:58:43.324369 - 582D34340CAF - FE 50 20 AA 01 90 AF 0C 34 34 2D 58 06 10 02 5F 03 A4 - hum=86.3
2021-07-29 18:58:45.324928 - 582D34340CAF - FE 50 20 AA 01 91 AF 0C 34 34 2D 58 0D 10 04 88 00 5F 03 A7 - temp=13.6 hum=86.3
2021-07-29 18:58:49.335941 - 582D34340CAF - FE 50 20 AA 01 93 AF 0C 34 34 2D 58 0D 10 04 87 00 5F 03 B0 - temp=13.5 hum=86.3
2021-07-29 18:58:51.337885 - 582D34340CAF - FE 50 20 AA 01 94 AF 0C 34 34 2D 58 0D 10 04 87 00 5F 03 A5 - temp=13.5 hum=86.3
2021-07-29 18:58:53.345866 - 582D34340CAF - FE 50 20 AA 01 95 AF 0C 34 34 2D 58 04 10 02 87 00 A6 - temp=13.5
2021-07-29 18:58:55.345954 - 582D34340CAF - FE 50 20 AA 01 96 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A7 - temp=13.5 hum=86.1
2021-07-29 18:58:57.348935 - 582D34340CAF - FE 50 20 AA 01 97 AF 0C 34 34 2D 58 0D 10 04 87 00 5E 03 B1 - temp=13.5 hum=86.2
2021-07-29 18:58:59.349893 - 582D34340CAF - FE 50 20 AA 01 98 AF 0C 34 34 2D 58 0D 10 04 87 00 5E 03 B0 - temp=13.5 hum=86.2
2021-07-29 18:59:03.366870 - 582D34340CAF - FE 50 20 AA 01 9A AF 0C 34 34 2D 58 06 10 02 5E 03 A5 - hum=86.2
2021-07-29 18:59:05.371224 - 582D34340CAF - FE 50 20 AA 01 9B AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A4 - temp=13.5 hum=86.1
2021-07-29 18:59:09.386057 - 582D34340CAF - FE 50 20 AA 01 9D AF 0C 34 34 2D 58 0D 10 04 87 00 5E 03 A8 - temp=13.5 hum=86.2
2021-07-29 18:59:11.390936 - 582D34340CAF - FE 50 20 AA 01 9E AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A7 - temp=13.5 hum=86.1
2021-07-29 18:59:13.400909 - 582D34340CAF - FE 50 20 AA 01 9F AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A8 - temp=13.5 hum=86.1
2021-07-29 18:59:15.401819 - 582D34340CAF - FE 50 20 AA 01 A0 AF 0C 34 34 2D 58 04 10 02 87 00 B0 - temp=13.5
2021-07-29 18:59:21.413838 - 582D34340CAF - FE 50 20 AA 01 A3 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A8 - temp=13.5 hum=86.1
2021-07-29 18:59:23.423749 - 582D34340CAF - FE 50 20 AA 01 A4 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A7 - temp=13.5 hum=86.1
2021-07-29 18:59:25.423789 - 582D34340CAF - FE 50 20 AA 01 A5 AF 0C 34 34 2D 58 06 10 02 5D 03 B1 - hum=86.1
2021-07-29 18:59:27.431767 - 582D34340CAF - FE 50 20 AA 01 A6 AF 0C 34 34 2D 58 0D 10 04 88 00 5C 03 B0 - temp=13.6 hum=86.0
2021-07-29 18:59:31.446788 - 582D34340CAF - FE 50 20 AA 01 A8 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A4 - temp=13.5 hum=86.1
2021-07-29 18:59:33.451895 - 582D34340CAF - FE 50 20 AA 01 A9 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A2 - temp=13.5 hum=86.1
2021-07-29 18:59:35.453768 - 582D34340CAF - FE 50 20 AA 01 AA AF 0C 34 34 2D 58 04 10 02 88 00 A7 - temp=13.6
2021-07-29 18:59:37.457675 - 582D34340CAF - FE 50 20 AA 01 AB AF 0C 34 34 2D 58 0A 10 01 32 AF - battery=50
2021-07-29 18:59:39.462721 - 582D34340CAF - FE 50 20 AA 01 AC AF 0C 34 34 2D 58 0D 10 04 87 00 5E 03 B1 - temp=13.5 hum=86.2
2021-07-29 18:59:41.466722 - 582D34340CAF - FE 50 20 AA 01 AD AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 A4 - temp=13.5 hum=86.1
2021-07-29 18:59:45.475674 - 582D34340CAF - FE 50 20 AA 01 AF AF 0C 34 34 2D 58 0D 10 04 87 00 5E 03 A8 - temp=13.5 hum=86.2
2021-07-29 18:59:47.485699 - 582D34340CAF - FE 50 20 AA 01 B0 AF 0C 34 34 2D 58 06 10 02 5F 03 A7 - hum=86.3
2021-07-29 18:59:49.487670 - 582D34340CAF - FE 50 20 AA 01 B1 AF 0C 34 34 2D 58 0D 10 04 86 00 5D 03 B0 - temp=13.4 hum=86.1
2021-07-29 18:59:51.490782 - 582D34340CAF - FE 50 20 AA 01 B2 AF 0C 34 34 2D 58 0D 10 04 87 00 5D 03 B0 - temp=13.5 hum=86.1
2021-07-29 18:59:53.494707 - 582D34340CAF - FE 50 20 AA 01 B3 AF 0C 34 34 2D 58 0D 10 04 86 00 5D 03 A3 - temp=13.4 hum=86.1
h2zero commented 3 years ago

I've created a branch here https://github.com/h2zero/OpenMQTTGateway/tree/duplicate-fix that may resolve the issue at least temporarily. I'm not sure if it will cause issues with other devices yet but if this works I will investigate further.

mhtrinh commented 3 years ago

Yes: that change did fix the my duplicate issue.

h2zero commented 3 years ago

Great, thanks for the quick feedback.

mhtrinh commented 3 years ago

Some raw output generated in case it help debugging without the sensor.

HCI sniffer - Bluetooth packet analyzer ver 5.50
device: hci1 snap_len: 1500 filter: 0xffffffff
2021-07-30 16:48:51.437541 > 04 0E 04 01 0B 20 00 
2021-07-30 16:48:51.437793 < 01 0C 20 02 01 00 
2021-07-30 16:48:51.439491 > 04 0E 04 01 0C 20 00 
2021-07-30 16:48:52.608484 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 00 AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 B0 
2021-07-30 16:48:56.621481 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 02 AF 0C 34 34 2D 58 0D 10 04 A9 00 E4 02 A5 
2021-07-30 16:48:58.621464 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 03 AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 A9 
2021-07-30 16:49:00.628457 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 04 AF 0C 34 34 2D 58 0D 10 04 A8 00 E3 02 AA 
2021-07-30 16:49:02.628457 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 05 AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 AF 
2021-07-30 16:49:06.641451 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 07 AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 A5 
2021-07-30 16:49:10.651443 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 09 AF 0C 34 34 2D 58 0D 10 04 A7 00 E4 02 A8 
2021-07-30 16:49:12.650706 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 0A AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 AF 
2021-07-30 16:49:14.658455 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 0B AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 AF 
2021-07-30 16:49:18.669434 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 0D AF 0C 34 34 2D 58 0D 10 04 A7 00 E4 02 A5 
2021-07-30 16:49:20.675432 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 0E AF 0C 34 34 2D 58 0D 10 04 A9 00 E4 02 A4 
2021-07-30 16:49:22.681469 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 0F AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 A9 
2021-07-30 16:49:24.688424 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 10 AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 A9 
2021-07-30 16:49:26.691424 > 04 3E 23 02 01 00 00 AF 0C 34 34 2D 58 17 02 01 06 13 16 95 
  FE 50 20 AA 01 11 AF 0C 34 34 2D 58 06 10 02 E5 02 AF 
2021-07-30 16:49:28.698420 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 12 AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 AF 
2021-07-30 16:49:30.705420 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 13 AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 AF 
2021-07-30 16:49:32.710416 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 14 AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 A4 
2021-07-30 16:49:34.710435 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 15 AF 0C 34 34 2D 58 0D 10 04 A9 00 E5 02 A8 
2021-07-30 16:49:36.715412 > 04 3E 23 02 01 00 00 AF 0C 34 34 2D 58 17 02 01 06 13 16 95 
  FE 50 20 AA 01 16 AF 0C 34 34 2D 58 04 10 02 A7 00 A9 
2021-07-30 16:49:38.716408 > 04 3E 22 02 01 00 00 AF 0C 34 34 2D 58 16 02 01 06 12 16 95 
  FE 50 20 AA 01 17 AF 0C 34 34 2D 58 0A 10 01 32 B0 
2021-07-30 16:49:40.723405 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 18 AF 0C 34 34 2D 58 0D 10 04 A9 00 E5 02 A5 
2021-07-30 16:49:42.729401 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 19 AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 A5 
2021-07-30 16:49:44.731403 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 1A AF 0C 34 34 2D 58 0D 10 04 A8 00 E5 02 A9 
2021-07-30 16:49:46.740397 > 04 3E 25 02 01 00 00 AF 0C 34 34 2D 58 19 02 01 06 15 16 95 
  FE 50 20 AA 01 1B AF 0C 34 34 2D 58 0D 10 04 A8 00 E4 02 A9 
2021-07-30 16:49:48.747394 > 04 3E 23 02 01 00 00 AF 0C 34 34 2D 58 17 02 01 06 13 16 95 
  FE 50 20 AA 01 1C AF 0C 34 34 2D 58 06 10 02 E4 02 A9

Generated with : 

sudo hcitool -i $dev lescan --whitelist --passive --duplicates &>/dev/null &
sudo hcidump -i $dev --raw -t
h2zero commented 3 years ago

Thanks, I have submitted the change as a PR #1044 as there does not appear to be any serious adverse effects.