[HELP] Can't read data with new BLE connector

[ndrs-cwt]

Describe the issue After update gateway code from master branch, Gateway does not send any data from BLE device to thingsboard but still sending other data like bleconnectorEventsProduced to server and I can't find any error in log.

connector.log

""2021-12-18 06:38:59" - |ERROR| - [device.py] - device - connect_to_device - 155 - Device with address 24:0A:C4:A5:7F:7E was not found."
""2021-12-18 06:39:09" - |INFO| - [device.py] - device - run_client - 164 - Connected to M5StackUART device"
""2021-12-18 06:43:11" - |INFO| - [device.py] - device - run_client - 164 - Connected to M5StackUART device "
""2021-12-18 07:13:01" - |INFO| - [device.py] - device - run_client - 164 - Connected to M5StackUART device "

service.log

""2021-12-18 07:12:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 78 - Gateway starting..."
""2021-12-18 07:12:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 83 - ThingsBoard IoT gateway version: 2.9"
""2021-12-18 07:12:58" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import BLEConnector from /usr/lib/python3/dist-packages/thingsboard_gateway/connectors/ble."
""2021-12-18 07:12:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 153 - Gateway started."

BTW I can read data from BLE device with this code on the same gateway and nRF Connect app still get notify data from device.

import asyncio
from bleak import BleakClient

address = "24:0A:C4:A5:7F:7E"
MODEL_NBR_UUID = "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"

async def main(address):
    async with BleakClient(address) as client:
        model_number = await client.read_gatt_char(MODEL_NBR_UUID)
        print("data: {0}".format("".join(map(chr, model_number))))

asyncio.run(main(address))

>>> data: I

From the config below did I miss some parameter or set something invalid? About how to update code in gateway, Is copy code from master branch and past in /usr/lib/python3/dist-packages/thingsboard_gateway enough or I miss some step?

Configuration ble.json (read every 1 min)

{
    "name": "BLE Connector",
    "passiveScanMode": true,
    "showMap": false,
    "scanner": {
      "timeout": 10000,
      "deviceName": "Device name"
    },
    "devices": [
      {
        "name": "M5StackUART",
        "MACAddress": "24:0A:C4:A5:7F:7E",
        "pollPeriod": 60000,
        "showMap": false,
        "timeout": 10000,
        "telemetry": [
          {
            "key": "UART_Tx",
            "method": "notify",
            "characteristicUUID": "6E400003-B5A3-F393-E0A9-E50E24DCCA9E",
            "byteFrom": 0,
            "byteTo": -1
          }
        ],
        "attributes": [
        ],
        "attributeUpdates": [
        ],
        "serverSideRpc": [
          {
            "methodRPC": "UART_Rx",
            "withResponse": false,
            "characteristicUUID": "6E400002-B5A3-F393-E0A9-E50E24DCCA9E",
            "methodProcessing": "write"
          }
        ]
      }
    ]
  }

Versions (please complete the following information):

• OS: Ubuntu 20.04 LTS 4.14.111
• Thingsboard IoT Gateway version current master (forked)
• Python version : 3.8.10
• bleak 0.13.0
• bluetoothctl: 5.53

[samson0v]

Hi @ndrs-cwt, if you want to read data every 60 sec, you have to use "method": "read", because when you set "method": "notify", BLE Connector subscribe for device and listen to any update of data, so the device can update data in 70 sec or 50 sec and exactly in this way we will get the data. Additionally, you can set the DEBUG level of logs in the config/logs.conf.

Thanks for your interest in ThingsBoard IoT Gateway!

[jayneev]

Hi @ndrs-cwt , @samson0v and @zbeacon , I am also still trying to find right way to get 'update data notifications from BLE peripheral devices, with latest BLE connector remake. Below are few observations and queries , if provided feedback, will definitely help us to get notify data on gateway.

1) OBSERVATION: Below parameters in ble.json now removed from latest one. Are they replaced with new parameter? Old parameters:

"rescanIntervalSeconds": 18000,
"checkIntervalSeconds": 45, 
"scanTimeSeconds": 35,

======================= New parameters: (Is this suffice above requirements?)

"scanner": { "timeout": 10000, "deviceName": "Device name" <==== Is this 'Device name' needs to be replaced with gateway name assigned at Thingsboard application(TB) side? }, "devices": [ {
...... "pollPeriod": 500000, <==== per devices ..... }

2) QUERY: Is the newly added above json parameter tag 'deviceName' value is the gateway name specified on Thingsboard?

3) QUERY: Once device connected to gateway, will they get added in TB dashboard as in previous versions?

4) QUERY: As in QUERY#3 above, are the below JSON parameters still serve purpose of identifying the device connected? "attributes": [ { "key": "name", "method": "read", "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "byteFrom": 0, "byteTo": -1 } ], "attributeUpdates": [ { "attributeOnThingsBoard": "sharedName", "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB" } ],

5. QUERY: Does server side RPC if not used, should the below can be empty? "serverSideRpc": [ { "methodRPC": "rpcMethod1", "withResponse": true, "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "methodProcessing": "read" }, { "methodRPC": "rpcMethod2", "withResponse": true, "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "methodProcessing": "write" }, { "methodRPC": "rpcMethod3", "withResponse": true, "methodProcessing": "scan" } ]

6. QUERY: Does current configuration supports ACTIVE scan mode? If yes, then does setting "passiveScanMode": false, will suffice this requirement?

I appreciate you all for your valuable contribution and help.

Warm Regards, @jayneev

[samson0v]

Hi @jayneev,

1. "rescanIntervalSeconds": 18000, "checkIntervalSeconds": 45, "scanTimeSeconds": 35, deleted and aren't using. The scanner device name is used for searching the mac address of the device if you don't know it and will be printed to console if it is found. A poll period is used for polling device data if it has "method": "read".
2. deviceName is the name of your device, for example, if you are getting data from a Temperature sensor, you can specify "deviceName": "Temperature sensor" or "deviceName": "My sensor 1". deviceName is used to find out which device in ThingsBoard the data is.
3. Yes, it is. This logic didn't change.
4. Yes, it is used to know for who Gateway have to connect, where to search data and how to convert it.
5. If you aren't using RPC, the section serverSideRpc can be empty.
6. If "passiveScanMode": false - Gateway use ACTIVE scan mode and vice versa.

[jayneev]

Hi @samson0v, Thank you for the reply. Regarding below 'deviceName' explanation , there seems confusion in the ble.json configuration file. <<<<<<<<<<Answer from @samson0v <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< deviceName is the name of your device, for example, if you are getting data from a Temperature sensor, you can specify "deviceName": "Temperature sensor" or "deviceName": "My sensor 1". deviceName is used to find out which device in ThingsBoard the data is. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

QUERY: Unable to understand the value of tag "deviceName" under "scanner" tag name. From above explanation it is understood that you mean device connected to the gateway device, as tag "name" under "devices" tag below? Please refer below ble.json extract. <<<<<<<<<<<<<<< ble.json <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< "scanner": { "timeout": 10000, "deviceName": "Device name" <=== Is this gateway name? },

"devices": [ { "name": "Temperature sensor", ..... ''''' } { "name": "My sensor 1", ..... ''''' } <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Warm Regards, @jayneev

[jayneev]

Hi @samson0v , In continuation with my above discussion thread please see below Exception error reported whenever TB gateway runs with CMx9 device connected. Sensor Device name ====> "name": "CMx9", (Please refer to ble.json config below) "deviceName" inside scanner =====> "scanner": { "timeout": 10000, "deviceName": "Device Name" },

<<<<<<<<<<<<<<<<ThingsBoard gateway Error message logs <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ""2021-12-21 06:34:24" - |DEBUG| - [scanner.py] - scanner - _parse_msg - 282 - received D-Bus signal: org.freedesktop.DBus.Properties.PropertiesChanged (/org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e" ""2021-12-21 06:34:24" - |DEBUG| - [client.py] - client - _parse_msg - 973 - received D-Bus signal: org.freedesktop.DBus.Properties.PropertiesChanged (/org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e/c" ""2021-12-21 06:34:24" - |DEBUG| - [client.py] - client - read_gatt_char - 681 - Read Characteristic 1e820106-4a30-45b4-9e17-93f34e283081 | /org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e/char001c: by" ""2021-12-21 06:34:24" - |INFO| - [device.py] - device - __show_map - 204 - MAP FOR CMX9 | [Service] 00001801-0000-1000-8000-00805f9b34fb (Handle: 10): Generic Attribute Profile | [Characteristic] 00002a05-0000-1000-8000-00805f9b34fb (Handle: 11): Service Changed (indicate), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 13): Client Characteristic Configuration) | Value: b'\x02\x00' | [Service] 1e821400-4a30-45b4-9e17-93f34e283081 (Handle: 14): Unknown | [Characteristic] 1e821401-4a30-45b4-9e17-93f34e283081 (Handle: 15): Unknown (notify), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 17): Client Characteristic Configuration) | Value: b'\x00\x00' | [Characteristic] 1e820101-4a30-45b4-9e17-93f34e283081 (Handle: 18): Unknown (read,write), Value: b'P\x00\x00' | [Characteristic] 1e820102-4a30-45b4-9e17-93f34e283081 (Handle: 20): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820103-4a30-45b4-9e17-93f34e283081 (Handle: 22): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820104-4a30-45b4-9e17-93f34e283081 (Handle: 24): Unknown (read,write), Value: b'\x01' | [Characteristic] 1e820105-4a30-45b4-9e17-93f34e283081 (Handle: 26): Unknown (read,write), Value: b'\x03\x00\x02\x00\x02' | [Characteristic] 1e820106-4a30-45b4-9e17-93f34e283081 (Handle: 28): Unknown (read,write), Value: b'\x00\x08\x05\x17\x01'" ""2021-12-21 06:34:24" - |DEBUG| - [scanner.py] - scanner - _parse_msg - 282 - received D-Bus signal: org.freedesktop.DBus.Properties.PropertiesChanged (/org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e" ""2021-12-21 06:34:24" - |DEBUG| - [scanner.py] - scanner - _parse_msg - 282 - received D-Bus signal: org.freedesktop.DBus.Properties.PropertiesChanged (/org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e" ""2021-12-21 06:34:24" - |DEBUG| - [client.py] - client - _parse_msg - 973 - received D-Bus signal: org.freedesktop.DBus.Properties.PropertiesChanged (/org/bluez/hci0/dev_DE_D9_82_1A_78_DC/service000e/c" Exception in thread CMx9: Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 173, in run self.loop.run_until_complete(self.run_client()) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 169, in run_client await self.timer() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 86, in timer await self.process_self() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 125, in process_self data = await self.client.read_gatt_char(char_id) File "/usr/lib/python3.8/site-packages/bleak/backends/bluezdbus/client.py", line 662, in read_gatt_char raise BleakError( bleak.exc.BleakError: Characteristic with UUID 00002A00-0000-1000-8000-00805F9B34FB could not be found! (<== PLEASE see this Exception Error) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< <<<<<<<<<<<<<<<<<<<<<<< ble.json <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< { "name": "BLE Connector", "passiveScanMode": false, "showMap": true, "scanner": { "timeout": 10000, "deviceName": "Device Name" },

"devices": [
{
"name": "CMx9",
"MACAddress": "DE:D9:82:1A:78:DC",
"pollPeriod": 10000,
"showMap": true,
"timeout": 10000,
"telemetry": [
{
"key": "TEMPERATURE",
"method": "notify",
"characteristicUUID": "1E821401-4A30-45B4-9E17-93F34E283081",
"byteFrom": 0,
"byteTo": 3
}
]
"attributes": [
{
"key": "name",
"method": "read",
"characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB",
"byteFrom": 0,
"byteTo": -1
}
],
"attributeUpdates": [
{
"attributeOnThingsBoard": "sharedName",
"characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB"
}
],

"serverSideRpc": [ { "methodRPC": "rpcMethod1", "withResponse": true, "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "methodProcessing": "read" } ] } ] } <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Warm Regards @jayneev

[samson0v]

Hi @jayneev,

1. "deviceName": "Device name" <=== Is this gateway name? - NO, it is the name of your device to find his mac address, if you don't know the mac address
2. The exception is about the characteristic that can't be found. It means that your device hasn't this characteristic. From this screen below

You have the next characteristics:

1. 1e821401-4a30-45b4-9e17-93f34e283081
2. 1e821401-4a30-45b4-9e17-93f34e283081
3. 1e820101-4a30-45b4-9e17-93f34e283081
4. 1e820102-4a30-45b4-9e17-93f34e283081
5. 1e820103-4a30-45b4-9e17-93f34e283081
6. 1e820104-4a30-45b4-9e17-93f34e283081
7. 1e820105-4a30-45b4-9e17-93f34e283081
8. 1e820106-4a30-45b4-9e17-93f34e283081

And I can't see any of those in your config file, so Gateway can't read any data from the device.

You can read the documentation about BLE and about the objects hierarchy if you are a little confusing of the configuration.

[jayneev]

Dear @samson0v, Please find answer embedded [Answer].

1. "deviceName": "Device name" <=== Is this gateway name? - NO, it is the name of your device to find his mac address, if you don't know the mac address [Answer]: Consider the scenario, there are multiple BLE devices connected to single TB BLE gateway. Can you please point out what will be "Scanner" - "deviceName" corresponding value?

2. The exception is about the characteristic that can't be found. It means that your device hasn't this characteristic. From this screen below And I can't see any of those in your config file, so Gateway can't read any data from the device.

[Answer] Below uuid#2-#7 are not required at gateway side. Only uuid#1, above added in ble.json config i.e. "1e821401-4a30-45b4-9e17-93f34e283081" which provides notification data.

1. 1e821401-4a30-45b4-9e17-93f34e283081 <== Provides notification data and required to be notified at gateway end, so mentioned in ble config
2. 1e820101-4a30-45b4-9e17-93f34e283081 <== Since not required, so not mentioned in ble config
3. 1e820102-4a30-45b4-9e17-93f34e283081 <== Since not required, so not mentioned in ble config
4. 1e820103-4a30-45b4-9e17-93f34e283081 <== Since not required, so not mentioned in ble config
5. 1e820104-4a30-45b4-9e17-93f34e283081 <== Since not required. so not mentioned in ble config
6. 1e820105-4a30-45b4-9e17-93f34e283081 <== Since not required. so not mentioned in ble config
7. 1e820106-4a30-45b4-9e17-93f34e283081 <== Since not required, so not mentioned in ble config

I noticed that standard uuid "00002a00-0000-1000-8000-00805f9b34fb" related to "Device Name", TB gateway not able to read and Exception raised by bleak. It was functional in previous TB gateway versions. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 169, in run_client await self.timer() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 86, in timer await self.process_self() File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 125, in process_self data = await self.client.read_gatt_char(char_id) File "/usr/lib/python3.8/site-packages/bleak/backends/bluezdbus/client.py", line 662, in read_gatt_char raise BleakError( bleak.exc.BleakError: Characteristic with UUID 00002A00-0000-1000-8000-00805F9B34FB could not be found! (<== PLEASE see this Exception Error) <======== This Exception may probably cause the issue, not sure? <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Note: For reading uuid 00002A00-0000-1000-8000-00805F9B34FB already in below BLE config files, JSON entry is provide and was tested OK in previous versions i.e device name e.g "CMx9" was identified and no exception raised. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< "attributes": [ { "key": "name", "method": "read", "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "byteFrom": 0, "byteTo": -1 } ], "attributeUpdates": [ { "attributeOnThingsBoard": "sharedName", "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB" } <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Warm Regards, @jayneev

[samson0v]

@jayneev, we can make a call if you can, and together solve your problem. Please, send me info on how we can meet (zoom, google meet, teams, etc) in my email (vbidochka@thingsboard.com). Best regards, Vitalii

[jayneev]

@samson0v, I sent Teams invitation just now scheduled at 09:00 pm (IST) on above mentioned mail.

Best Regards, @jayneev

[samson0v]

@jayneev, sorry, but don't receive an email, can you please send me your email and I will write to you Sorry for the inconvenience.

[samson0v]

Sorry, it isn't vbidochka@thingsboard.com, it is vbidochka@thingsboard.io

[jayneev]

ok

[jayneev]

just resent with your correct email id, please check and join.

[samson0v]

@jayneev received, I will wait for call, thanks

[jayneev]

We are already on the call. please join, click on the Teams links provide in meetings requests.

[jayneev]

@samson0v, Are you able to get the link in Teams meeting requests?

[samson0v]

Yes, 5 min

[jayneev]

ok

[jayneev]

Hi @samson0v,

Thanks for participation in our live issue diagnosis and discussion, can this issue be discussed and concluded with issue originator @ndrs-cwt, and separate issue be raised for Bleak handling BLE data convertor to resolve it further?

Warm Regards, Rohit Singh

[jayneev]

@samson0v, as discussed separate issue raised - https://github.com/thingsboard/thingsboard-gateway/issues/660#issue-1086463979

Warm Regards, Rohit