[BUG] Error in multiple telemetry processing

[mircopz]

Describe the bug My device sent the following array of telemetry

[ 
  {
      "ts": 1638808643845, 
      "values": {
        "temperature": 23.6, 
        "humidity": 38.6
      }
  },
  {
      "ts": 1638808643845, 
      "values": {
        "temperature": 23.6, 
        "humidity": 38.6
      }
  },
  {
      "ts": 1638808643845, 
      "values": {
        "temperature": 23.6, 
        "humidity": 38.6
      }
  },
  {
      "ts": 1638808643845, 
      "values": {
        "temperature": 23.6, 
        "humidity": 38.6
      }
  }
]

It is processed by my custom function, and the final message (before validation) has always the following format:

{
  "deviceName": "MY-DEVICE_21/000-000005",
  "deviceType": "MY-DEVICE", 
  "attributes": [], 
  "telemetry": [
    {
      "ts": 1638808643845, 
      "values": {
        "temperature": 23.6, 
        "humidity": 38.6
      }
    }
  ]
}

As you can see from logs, the first three elements are correctly processed, while the fourth cannot be validated, even if it has the same format but function validate_converted_data received an empty array. Is it a problem of a multiple telemetry message received? I've not been able to reproduce it anymore.

Connector name (If bug in the some connector): MQTT Connector

Error traceback (If available):

""2021-12-06 16:37:35" - |DEBUG| - [my_mqtt_uplink_converter.py] - my_mqtt_uplink_converter - convert - 39 - Converted data: {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638807681845, 'values': {'temperature': 23.3, 'humidity': 39.3}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638807681845, 'values': {'temperature': 23.3, 'humidity': 39.3}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic device/28630045-00-21-000-000005/telemetry"
""2021-12-06 16:37:35" - |DEBUG| - [my_mqtt_uplink_converter.py] - my_mqtt_uplink_converter - convert - 39 - Converted data: {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638807980845, 'values': {'temperature': 23.1, 'humidity': 39.4}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638807980845, 'values': {'temperature': 23.1, 'humidity': 39.4}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic device/28630045-00-21-000-000005/telemetry"
""2021-12-06 16:37:35" - |DEBUG| - [my_mqtt_uplink_converter.py] - my_mqtt_uplink_converter - convert - 39 - Converted data: {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638808280845, 'values': {'temperature': 23.3, 'humidity': 39.2}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638808280845, 'values': {'temperature': 23.3, 'humidity': 39.2}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic device/28630045-00-21-000-000005/telemetry"
""2021-12-06 16:37:35" - |DEBUG| - [my_mqtt_uplink_converter.py] - my_mqtt_uplink_converter - convert - 39 - Converted data: {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638808643845, 'values': {'temperature': 23.6, 'humidity': 38.6}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceType': 'MY-DEVICE', 'attributes': [], 'telemetry': [{'ts': 1638808643845, 'values': {'temperature': 23.6, 'humidity': 38.6}}], 'deviceName': 'MY-DEVICE_21/000-000005'}"
""2021-12-06 16:37:35" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic device/28630045-00-21-000-000005/telemetry"
""2021-12-06 16:37:35" - |ERROR| - [tb_utility.py] - tb_utility - validate_converted_data - 69 - No telemetry and attributes in data: {"deviceType": "MY-DEVICE", "attributes": [], "telemetry": [], "deviceName": "MY-DEVICE_21/000-000005"}"
""2021-12-06 16:37:35" - |ERROR| - [tb_gateway_service.py] - tb_gateway_service - __send_to_storage - 389 - Data from MQTT Broker Connector connector is invalid."

Versions (please complete the following information):

• OS: Ubuntu 18.04
• Thingsboard IoT Gateway version 2.9
• Python version 3.7

[samson0v]

Hi @mircopz, pls, send your config file and if you are using a custom converter, send it too. Also, I see that there are 4 messages which successful converted and 5 is unsuccessful. Are you sure that you send the correct data to the Gateway?

Thanks for your interest in ThingsBoard IoT Gateway!

[mircopz]

Hi @samson0v

Data sent to the gateway is the first 4-elements array reported in my first message. I think the unsuccessful data validation is related to my 4 message, the log is right after data conversion. For further information, on thingsboard I received only 2 and 3 message, missing the 1 and 4.

This is my MQTT config file:

{
  "broker": {
    "name": "MQTT Broker",
    "host": "**********",
    "port": 1883,
    "clientId": "TB_gateway",
    "maxMessageNumberPerWorker": 10,
    "maxNumberOfWorkers": 100,
    "security": {
      "type": "basic",
      "username": "*****",
      "password": "*****"
    }
  },
  "mapping": [
    {
      "topicFilter": "device/+/attributes",
      "converter": {
        "type": "custom",
        "extension": "MyMqttUplinkConverter",
        "extension-config": {
        }
      }
    },
    {
      "topicFilter": "device/+/telemetry",
      "converter": {
        "type": "custom",
        "extension": "MyMqttUplinkConverter",
        "extension-config": {
        }
      }
    }
  ],
  "connectRequests": [
    {
      "topicFilter": "device/+/connect",
      "deviceNameTopicExpression": "(?<=device\/)(.*?)(?=\/connect)",
      "deviceTypeJsonExpression": "${deviceType}"
    }
  ],
  "disconnectRequests": [
    {
      "topicFilter": "device/+/disconnect",
      "deviceNameTopicExpression": "(?<=device\/)(.*?)(?=\/disconnect)"
    }
  ]
}

and that is MyMqttUplinkConverter function

from thingsboard_gateway.connectors.converter import Converter, log
import re

class MyMqttUplinkConverter(Converter):
    def __init__(self, config):
        self.__config = config
        self.result_dict = {
            "deviceType": config.get("deviceType", "MY-DEVICE"),
            "attributes": [],
            "telemetry": []
            }

    def convert(self, topic, data):
        self.result_dict["attributes"] = []
        self.result_dict["telemetry"] = []
        deviceName = re.search("(?<=airy\/)(.*?)(?=\/)", topic).group()
        deviceName = deviceName[:14] + "/" + deviceName[15:]
        deviceName = self.result_dict["deviceType"] + "_" + deviceName[12:]
        msgType = re.search("([^\/]+$)", topic).group()
        self.result_dict["deviceName"] = deviceName
        self.result_dict[msgType] = []
        self.result_dict[msgType].append(data)
        log.debug("Converted data: %s", self.result_dict)
        return self.result_dict

[samson0v]

@mircopz We tested your converter and not be able to reproduce your problem

[mircopz]

@samson0v thank you anyway, I haven't been able to reproduce it either