[Help Wanted] BLE connector remake not working with fixes 'Fixed data converting for BLE Connector #669'

[jayneev]

Describe the bug BLE attributes read for standard BLE 'device name' characteristics UUID "00002A00-0000-1000-8000-00805F9B34FB" reports Exception error. 'bleak.exc.BleakError: Characteristic with UUID 00002A00-0000-1000-8000-00805F9B34FB could not be found!' (Please refer corresponding UUID, below ble.json config - "attributes" section) This issue is linked to earlier issue named "[BUG] BLE data converter not working with latest BLE connector remake from master branch #660" Also we noticed in BLE.json configuration new entries, can you review below entries as per new changes?

Connector name (If bug in the some connector): BLE Connector

Error traceback: BLE Device name ====> "name": "CMx4", (Please refer to ble.json config below)

<<<<<<<<<<<<<<<< ThingsBoard gateway Error message logs START<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
tb_gateway_service - __init__ - 77 - Gateway starting..."
""2021-12-31 11:06:43" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 82 - ThingsBoard IoT gateway version: 2.9"
""2021-12-31 11:06:43" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 178 - Subscribed to *|* with id 1 for device *"
""2021-12-31 11:06:43" - |INFO| - [event_storage_reader.py] - event_storage_reader - read_state_file - 148 - FileStorage_reader -- Initializing from state file: [./data/data_1640947770384.txt:0]"
""2021-12-31 11:06:44" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 138 - connection SUCCESS"
""2021-12-31 11:06:45" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import BLEConnector from /usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble."
""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 65 - FOUND DEVICES"
""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 74 - E2:E1:C2:57:D7:09: CMx4"
""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 74 - 62:9F:15:CD:EC:17: 62-9F-15-CD-EC-17"
""2021-12-31 11:06:56" - |INFO| - [event_storage_reader.py] - event_storage_reader - delete_read_file - 172 - FileStorage_reader -- Cleanup old data file: ./data/data_1640947770384.txt!"
""2021-12-31 11:06:56" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 154 - Gateway started."
""2021-12-31 11:06:59" - |INFO| - [device.py] - device - run_client - 164 - Connected to CMx4 device"
""2021-12-31 11:07:00" - |INFO| - [device.py] - device - __show_map - 204 - MAP FOR CMX4
....
....
Exception in thread CMx4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 173, in run
    self.loop.run_until_complete(self.run_client())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 169, in run_client
    await self.timer()
  File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 86, in timer
    await self.__process_self()
  File "/usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble/device.py", line 125, in __process_self
    data = await self.client.read_gatt_char(char_id)
  File "/usr/lib/python3.8/site-packages/bleak/backends/bluezdbus/client.py", line 662, in read_gatt_char
    raise BleakError(
bleak.exc.BleakError: Characteristic with UUID 00002A00-0000-1000-8000-00805F9B34FB could not be found!
<<<<<<<<<<<<<<<< ThingsBoard gateway Error message logs END <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

<<<<<<<<<<<<<<<< ble.json START <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
{
  "name": "BLE Connector",
  "passiveScanMode": false,
  "showMap": true,
  "devices": [
    {
      "name": "CMx4",
      "MACAddress": "E2:E1:C2:57:D7:09",
      "pollPeriod": 5000,
      "showMap": true,
      "timeout": 10000,
      "telemetry": [
        {
          "key": "Temp",
          "method": "notify",
          "characteristicUUID": "1E821401-4A30-45B4-9E17-93F34E283081",  <=== custom notification UUID
          "valueExpression": "[0]"
        }
      ],
      "attributes": [
        {
          "key": "name",
          "method": "read",
          "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB",
          "valueExpression": "[0:2]cm [2:]A"      <====== It's not clear what 'cm' and 'A' stands for             
        },                                                             
        {                                                              
          "key": "values",           <===== This  JSON section is new entry, is the below 'valueExpression' entry OK?
          "method": "read",                                            
          "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", <== standard 'device name' characteristics UUID
          "valueExpression": "[:]"                         
        }                                                              
      ],                                                               
      "attributeUpdates": [                                            
        {                                                              
          "attributeOnThingsBoard": "sharedName",                      
          "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB" 
        }                                                              
      ],                                                               
      "serverSideRpc": [                                               
      ]                                                                
    }                                                                  
  ]                                                                    
}                                                                      
<<<<<<<<<<<<<<<<<<<<<<<< ble.json END <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
**Versions (please complete the following information):**
 - OS:  Linux 4.9
 - Thingsboard IoT Gateway version 2.9 latest master branch
 - Python version 3.8

[jayneev]

@samson0v, it seems that the BLE converter still not working with latest fixes named 'Fixed data converting for BLE Connector #669'. Could you please check the above logs and ble.json as per new updated configurations? If possible, can we setup online meeting for triage, as per your availability?

Warm Regards, @jayneev

[samson0v]

Hi @jayneev, it seems that your BLE device hasn't that characteristic, make sure that you are using the right one. To see all characteristics set "showMap": true on your device config section and use them (we were doing those things on our call).

"valueExpression": "[:]" - it is in what format data will be sent to the Gateway. When the data come from the device to Gateway, it is a byte array, so you must specify what specific byte to read. You can easily set [:] to read all bytes in the array if you don't know what exactly the data device send.

[jayneev]

@samson0v, Can we have discussion on Teams if you are available now?

[samson0v]

@jayneev today no, but on Tuesday (04.01.2022) we can make a call.

[jayneev]

@samson0v, 1) It's standard BLE device Name characteristic UUID - "00002A00-0000-1000-8000-00805F9B34FB" which is default generic access attributes sent as default by BLE device connected. In all past releases it worked OK and returned CMx4 as 'Device Name', which was published from Thingsboard Gateway to Thingsboard. ""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 65 - FOUND DEVICES" ""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 74 - E2:E1:C2:57:D7:09: CMx4" 2) "valueExpression": "[:]" - we set it to read all bytes in the array. Not sure what meant by "[0:2]cm [2:]A"?

[samson0v]

@jayneev, use showMap to find out what characteristics exist and send the list here.

[jayneev]

""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - show_map - 65 - FOUND DEVICES" ""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 74 - E2:E1:C2:57:D7:09: CMx4" ""2021-12-31 11:06:56" - |INFO| - [ble_connector.py] - ble_connector - show_map - 74 - 62:9F:15:CD:EC:17: 62-9F-15-CD-EC-17" ""2021-12-31 11:06:56" - |INFO| - [event_storage_reader.py] - event_storage_reader - delete_read_file - 172 - FileStorage_reader -- Cleanup old data file: ./data/data_1640947770384.txt!" ""2021-12-31 11:06:56" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 154 - Gateway started." ""2021-12-31 11:06:59" - |INFO| - [device.py] - device - run_client - 164 - Connected to CMx4 device" ""2021-12-31 11:07:00" - |INFO| - [device.py] - device - __show_map - 204 - MAP FOR CMX4 | [Service] 00001801-0000-1000-8000-00805f9b34fb (Handle: 10): Generic Attribute Profile | [Characteristic] 00002a05-0000-1000-8000-00805f9b34fb (Handle: 11): Service Changed (indicate), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 13): Client Characteristic Configuration) | Value: b'\x02\x00' | [Service] 1e821400-4a30-45b4-9e17-93f34e283081 (Handle: 14): Unknown | [Characteristic] 1e821401-4a30-45b4-9e17-93f34e283081 (Handle: 15): Unknown (notify), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 17): Client Characteristic Configuration) | Value: b'\x00\x00' | [Characteristic] 1e820101-4a30-45b4-9e17-93f34e283081 (Handle: 18): Unknown (read,write), Value: b'P\x00\x00' | [Characteristic] 1e820102-4a30-45b4-9e17-93f34e283081 (Handle: 20): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820103-4a30-45b4-9e17-93f34e283081 (Handle: 22): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820104-4a30-45b4-9e17-93f34e283081 (Handle: 24): Unknown (read,write), Value: b'\x01' | [Characteristic] 1e820105-4a30-45b4-9e17-93f34e283081 (Handle: 26): Unknown (read,write), Value: b'\x03\x00\x02\x00\x02' | [Characteristic] 1e820106-4a30-45b4-9e17-93f34e283081 (Handle: 28): Unknown (read,write), Value: b'\x00\x08\x05\x17\x01'" Exception in thread CMx4:

[samson0v]

@jayneev and send, pls, the config file that you are using

[jayneev]

  {

"name": "BLE Connector", "passiveScanMode": false, "showMap": true, "devices": [ { "name": "CMx4", "MACAddress": "E2:E1:C2:57:D7:09", "pollPeriod": 5000, "showMap": true, "timeout": 10000, "telemetry": [ { "key": "Temp", "method": "notify", "characteristicUUID": "1E821401-4A30-45B4-9E17-93F34E283081", "valueExpression": "[0]" } ], "attributes": [ { "key": "name", "method": "read", "characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "valueExpression": "[0:2]cm [2:]A"
},
{
"key": "values",
"method": "read",
"characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB", "valueExpression": "[:]"
}
],
"attributeUpdates": [
{
"attributeOnThingsBoard": "sharedName",
"characteristicUUID": "00002A00-0000-1000-8000-00805F9B34FB" }
],
"serverSideRpc": [
]
}
]
}

[samson0v]

@jayneev I don't understand why you are using a characteristic UUID that doesn't exist in the device

The first telemetry section is OK because you use the right characteristic, the next attributes section is not OK, so Gateway raises the exception there are no exist characteristics.

To fix it, easily delete all obj in the attributes section (but not the attributes section, it can be empty but not without it).

[jayneev]

Deleting attribute section, will update on it soon online

[samson0v]

@jayneev don't send logs that make no sense, I think there is no problem firstly check device connection, check logs and check if these logs relate to the problem of this issue.

[jayneev]

Do you understand that this UUID (00002A00-0000-1000-8000-00805F9B34FB) devices sent which leads to it's identification as CMx4? Device was reconnected and tested. Below are the logs. Note that no CMx4 device updates received on the THINGSBOARD dashboard side.

<<<<<<<<<<<<<<<<<<<<<< logs START <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< thingsboard_gateway# ""2021-12-31 13:48:28" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 77 - Gateway starting..." ""2021-12-31 13:48:28" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 82 - ThingsBoard IoT gateway version: 2.9" ""2021-12-31 13:48:28" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 178 - Subscribed to | with id 1 for device *" ""2021-12-31 13:48:28" - |INFO| - [event_storage_reader.py] - event_storage_reader - read_state_file - 148 - FileStorage_reader -- Initializing from state file: [./data/data_1640957420864.txt:0]" ""2021-12-31 13:48:28" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 138 - connection SUCCESS" ""2021-12-31 13:48:30" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import BLEConnector from /usr/lib/python3.8/site-packages/thingsboard_gateway/connectors/ble." ""2021-12-31 13:48:40" - |INFO| - [ble_connector.py] - ble_connector - show_map - 65 - FOUND DEVICES" ""2021-12-31 13:48:40" - |INFO| - [ble_connector.py] - ble_connector - __show_map - 74 - E2:E1:C2:57:D7:09: CMx4" ""2021-12-31 13:48:40" - |INFO| - [ble_connector.py] - ble_connector - show_map - 74 - 7C:90:CA:01:46:92: 7C-90-CA-01-46-92" ""2021-12-31 13:48:40" - |INFO| - [event_storage_reader.py] - event_storage_reader - delete_read_file - 172 - FileStorage_reader -- Cleanup old data file: ./data/data_1640957420864.txt!" ""2021-12-31 13:48:40" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 154 - Gateway started." ""2021-12-31 13:48:43" - |INFO| - [device.py] - device - run_client - 164 - Connected to CMx4 device" ""2021-12-31 13:48:44" - |INFO| - [device.py] - device - __show_map - 204 - MAP FOR CMX4 | [Service] 00001801-0000-1000-8000-00805f9b34fb (Handle: 10): Generic Attribute Profile | [Characteristic] 00002a05-0000-1000-8000-00805f9b34fb (Handle: 11): Service Changed (indicate), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 13): Client Characteristic Configuration) | Value: b'\x02\x00' | [Service] 1e821400-4a30-45b4-9e17-93f34e283081 (Handle: 14): Unknown | [Characteristic] 1e821401-4a30-45b4-9e17-93f34e283081 (Handle: 15): Unknown (notify), Value: None | [Descriptor] 00002902-0000-1000-8000-00805f9b34fb (Handle: 17): Client Characteristic Configuration) | Value: b'\x00\x00' | [Characteristic] 1e820101-4a30-45b4-9e17-93f34e283081 (Handle: 18): Unknown (read,write), Value: b'P\x00\x00' | [Characteristic] 1e820102-4a30-45b4-9e17-93f34e283081 (Handle: 20): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820103-4a30-45b4-9e17-93f34e283081 (Handle: 22): Unknown (read,write), Value: b'\x00' | [Characteristic] 1e820104-4a30-45b4-9e17-93f34e283081 (Handle: 24): Unknown (read,write), Value: b'\x01' | [Characteristic] 1e820105-4a30-45b4-9e17-93f34e283081 (Handle: 26): Unknown (read,write), Value: b'\x03\x00\x02\x00\x02' | [Characteristic] 1e820106-4a30-45b4-9e17-93f34e283081 (Handle: 28): Unknown (read,write), Value: b'\x00\x08\x05\x17\x01'" <<<<<<<<<<<<<<<<<<<<<< logs END <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

[samson0v]

@jayneev I can't see any problems, you are using notify method, so make sure that data on your device changing.

[jayneev]

@samson0v, device sent updated data notification tested OK with TB gateway previous release versions 2.5.2 and onwards till release v2.9 (with bluepy) . So, why with same devices sending data to TB gateway (latest version 2.9 from master branch using bleak), updated data not received?

[samson0v]

@jayneev we released a new version of Gateway and updated documentation where describe all config parameters. You are welcome https://thingsboard.io/docs/iot-gateway/config/ble/

[samson0v]

Opened another issue