thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.74k stars 844 forks source link

[HELP] MQTT Error in transferring data from client #674

Closed l33tkrew closed 2 years ago

l33tkrew commented 2 years ago

Describe the issue When I push data from the client, I could see the result in MQTT explorer. But I cannot push the data more than once. I do not encounter this problem in deb package (2.9). I cannot spam data when I install tb-gateway latest image or tb-gateway latest master branch. Again the error below only occurs when I send the first data.

The command I run in client: Again I run the same command multiple times in 2.9 (deb package) version. It works well. But I want it to work in the newest version of tb-gateway as well.

mosquitto_pub -h "192.168.1.11" -p "1883" -t "/sensor/SN-101/data" -m '{"sensorType": "Thermometer", "sensorModel": "T1000", "temp": 42, "hum": 58}'

Configuration (Attach your configuration file) I only changed security section s2

Connector name (If you need help with some connector/converter): MQTT Connector

Error traceback (If it was raised):

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3330, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 79, in _on_message
    self._on_decoded_message(content, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 97, in _on_decoded_message
    target = content["device"] + "|*"
KeyError: 'device'

Versions (please complete the following information):

l33tkrew commented 2 years ago

I encounter the same output when I push data through REST connector. I think it appears when gateway tries to send to main server. Interestingly, The 2.9 version works well. I want to use master branch because docker image and 2.9 version gives error when I want to use REST connector.

As MQTT protocol, even if it gives an error in tb-gateway server, I can see the first data transfer in MQTT Explorer. But, when I run the same command again, I cannot see the data in my MQTT explorer.

======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
""2022-01-02 17:44:06" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 154 - Gateway started."
""2022-01-02 17:44:28" - |INFO| - [rest_connector.py] - rest_connector - __call__ - 319 - CONVERTER CONFIG: {'type': 'json', 'deviceNameExpression': 'Device ${name}', 'deviceTypeExpression': 'default', 'attributes': [{'type': 'string', 'key': 'model', 'value': '${sensorModel}'}], 'timeseries': [{'type': 'double', 'key': '${sensorModel}', 'value': '${temp}'}, {'type': 'double', 'key': 'humidity', 'value': '${hum}'}, {'type': 'string', 'key': 'combine', 'value': '${hum}:${temp}'}]}"
""2022-01-02 17:44:28" - |INFO| - [rest_connector.py] - rest_connector - __call__ - 324 - CONVERTED_DATA: {'deviceName': 'Device SN-101', 'deviceType': 'default', 'attributes': [{'model': 'Thermometer'}], 'telemetry': [{'Thermometer': '123'}, {'humidity': '123'}, {'combine': '123:123'}]}"
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3330, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 79, in _on_message
    self._on_decoded_message(content, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 97, in _on_decoded_message
    target = content["device"] + "|*"
KeyError: 'device'
""2022-01-02 17:44:44" - |INFO| - [rest_connector.py] - rest_connector - __call__ - 319 - CONVERTER CONFIG: {'type': 'json', 'deviceNameExpression': 'Device ${name}', 'deviceTypeExpression': 'default', 'attributes': [{'type': 'string', 'key': 'model', 'value': '${sensorModel}'}], 'timeseries': [{'type': 'double', 'key': '${sensorModel}', 'value': '${temp}'}, {'type': 'double', 'key': 'humidity', 'value': '${hum}'}, {'type': 'string', 'key': 'combine', 'value': '${hum}:${temp}'}]}"
""2022-01-02 17:44:44" - |INFO| - [rest_connector.py] - rest_connector - __call__ - 324 - CONVERTED_DATA: {'deviceName': 'Device SN-102', 'deviceType': 'default', 'attributes': [{'model': 'Thermometer'}], 'telemetry': [{'Thermometer': '123'}, {'humidity': '123'}, {'combine': '123:123'}]}"
imbeacon commented 2 years ago

Hi @l33tkrew ,

Thank you for your interest in ThingsBoard IoT gateway, it looks like the gateway receives a message to topic v1/gateway/rpc. Expected payload in this message is {"device":"Device A", "data":"some_data"}. But this message doesn't contain device name in the message. Please check, do you have some RPC sending in your Rule Chains. It may produce this issue.

In order to investigate this issue, please enable DEBUG logging level instead of INFO in file logs.conf (located in config folder).

l33tkrew commented 2 years ago

I do not have RPC sending in my Rule Chains. Btw I could not understand how I can enable DEBUG logging.

For MQTT, I use this command;

mosquitto_pub -h "192.168.1.11" -p "1883" -t "/sensor/SN-101/data" -m '{"sensorType": "Thermometer", "sensorModel": "T1000", "temp": 42, "hum": 58}'

I can send this message multiple times in 2.9 version deb package. I do same thing with same configurations in master branch version but I cannot push the data multiple times. I also use REST connector, it also converts data but it cannot push to thingsboard mqtt broker. The problem occurs when sending converted data to thingsboard through mqtt protocol.

This error occurs also in the docker image version.

Please, see the image below; (Just for testing purposes, I use mqtt explorer) s1

I can see the first message but I cannot push data more because it gives an error (attached in the previous messages) I can use 2.9 version to get rid of this message but while using 2.9 version I could not use rest.connector because a bug. But in the master branch I can use MQTT, REST connectors but the gateway cannot push data to thingsboard ip address more than once.

I also saw something weird in logs. If I run "sudo python3 tb_gateway.py", I saw this output. I trimmed info outputs.

""2022-01-04 10:00:05" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'topicExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-01-04 10:00:05" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'valueExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-01-04 10:00:05" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - attributeRequests handler is missing some mandatory keys => rejected: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", 

The mqtt config file contains;

{
  "broker": {
    "name": "Default Local Broker",
    "host": "127.0.0.1",
    "port": 1883,
    "clientId": "ThingsBoard_gateway",
    "maxMessageNumberPerWorker": 10,
    "maxNumberOfWorkers": 100,
    "security": {
      "type": "anonymous"
    }
  },
  "mapping": [
    {
      "topicFilter": "/sensor/data",
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "${serialNumber}",
        "deviceTypeJsonExpression": "${sensorType}",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          },
          {
            "type": "string",
            "key": "${sensorModel}",
            "value": "on"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          },
          {
            "type": "string",
            "key": "combine",
            "value": "${hum}:${temp}"
          }
        ]
      }
    },
    {
      "topicFilter": "/sensor/+/data",
      "converter": {
        "type": "json",
        "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/data)",
        "deviceTypeTopicExpression": "Thermometer",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          }
        ]
      }
    },
    {
      "topicFilter": "/custom/sensors/+",
      "converter": {
        "type": "custom",
        "extension": "CustomMqttUplinkConverter",
        "extension-config": {
          "temperatureBytes": 2,
          "humidityBytes": 2,
          "batteryLevelBytes": 1
        }
      }
    }
  ],
  "connectRequests": [
    {
      "topicFilter": "sensor/connect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/connect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)"
    }
  ],
  "disconnectRequests": [
    {
      "topicFilter": "sensor/disconnect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/disconnect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)"
    }
  ],
  "attributeRequests": [
    {
      "retain": false,
      "topicFilter": "v1/devices/me/attributes/request",
      "deviceNameTopicExpression": "${SerialNumber}",
      "attributeNameJsonExpression": "${sensorModel}"
    }
  ],
  "attributeUpdates": [
    {
      "retain": true,
      "deviceNameFilter": "SmartMeter.*",
      "attributeFilter": "uploadFrequency",
      "topicExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ],
  "serverSideRpc": [
    {
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
      "responseTimeout": 10000,
      "valueExpression": "${params}"
    },
    {
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "valueExpression": "${params}"
    }
  ]
}

Overall the output is when sending first message;

""2022-01-04 10:06:22" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 77 - Gateway starting..."
""2022-01-04 10:06:22" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 82 - ThingsBoard IoT gateway version: 2.9"
""2022-01-04 10:06:22" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 178 - Subscribed to *|* with id 1 for device *"
""2022-01-04 10:06:22" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import MqttConnector from /usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/connectors/mqtt."
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted mapping handlers: 3"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected mapping handlers: 0"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted serverSideRpc handlers: 2"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected serverSideRpc handlers: 0"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted connectRequests handlers: 2"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected connectRequests handlers: 0"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted disconnectRequests handlers: 2"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected disconnectRequests handlers: 0"
""2022-01-04 10:06:22" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'topicExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-01-04 10:06:22" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 138 - connection SUCCESS"
""2022-01-04 10:06:22" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'valueExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-01-04 10:06:22" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - attributeRequests handler is missing some mandatory keys => rejected: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted attributeRequests handlers: 0"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected attributeRequests handlers: 1"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted attributeUpdates handlers: 1"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected attributeUpdates handlers: 0"
""2022-01-04 10:06:22" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 154 - Gateway started."
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 241 - MQTT Broker Connector connected to 127.0.0.1:1883 - successfully."
""2022-01-04 10:06:22" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import JsonMqttUplinkConverter from /usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/connectors/mqtt."
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /sensor/data"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /sensor/+/data"
""2022-01-04 10:06:22" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import CustomMqttUplinkConverter from /usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/extensions/mqtt."
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /custom/sensors/+"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /sensor/data, subscription message id = 1"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /sensor/+/data, subscription message id = 2"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /custom/sensors/+, subscription message id = 3"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/connect, subscription message id = 4"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/+/connect, subscription message id = 5"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/disconnect, subscription message id = 6"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-01-04 10:06:22" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/+/disconnect, subscription message id = 7"
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3330, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 79, in _on_message
    self._on_decoded_message(content, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 97, in _on_decoded_message
    target = content["device"] + "|*"
KeyError: 'device'

Btw, I could not understand why the last row of the log above is

target = content["device"] + "|*"
I think it should be *|* rather than |* 

In case of any permission issue, I attached the permissions below

-rw-rw-r-- 1 myuser myuser 1435 Jan  4 09:53 bacnet.json
-rw-rw-r-- 1 myuser myuser 1907 Jan  4 09:53 ble.json
-rw-rw-r-- 1 myuser myuser 2217 Jan  4 09:53 can.json
-rw-r--r-- 1 root       root          0 Jan  4 10:09 connected_devices.json
-rw-rw-r-- 1 myuser myuser  685 Jan  4 09:53 custom_serial.json
-rw-rw-r-- 1 myuser myuser 1304 Jan  4 09:53 ftp.json
-rw-rw-r-- 1 myuser myuser 2307 Jan  4 09:53 logs.conf
-rw-rw-r-- 1 myuser myuser 5916 Jan  4 09:53 modbus.json
-rw-rw-r-- 1 myuser myuser  615 Jan  4 09:53 modbus_serial.json
-rw-rw-r-- 1 myuser myuser 3746 Jan  4 09:57 mqtt.json
-rw-rw-r-- 1 myuser myuser 1260 Jan  4 09:53 odbc.json
-rw-rw-r-- 1 myuser myuser 1302 Jan  4 09:53 opcua.json
-rw-rw-r-- 1 myuser myuser 4134 Jan  4 09:53 request.json
-rw-rw-r-- 1myuser myuser 3744 Jan  4 09:53 rest.json
-rw-rw-r-- 1 myuser myuser 3259 Jan  4 09:53 snmp.json
-rw-rw-r-- 1 myuser myuser 1682 Jan  4 09:56 tb_gateway.yaml

The connected_devices.json file has different permission. Is that reason for my error? I wrote many things just to help u to understand my problem. I do not know it is a bug or not.

Note: I installed the master branch through this link "https://thingsboard.io/docs/iot-gateway/install/source-installation/"

imbeacon commented 2 years ago

Could you try to stop the gateway, remove file "connected_devices.json" and then restart the gateway? The part of code with the error it is callback for messages from ThingsBoard and by some, unknown at the moment, for me reason ThingsBoard sends some attribute update message, but without "device" field, could you check topic "v1/gateway/attributes" with mqtt explorer on your instance? It may contain some wrong messages, or please collect a packets dump via wireshark.

l33tkrew commented 2 years ago

I deleted the file and reran the python script. The same error occurred. I restart the gateway 3 times to see messages. That's the reason why there are 3 messages in mqtt explorer. If I did not restart the gateway, I could not send 3 messages. tq1

I will try to fix the REST connector in the 2.9 deb package. I hope I can

l33tkrew commented 2 years ago

I ran my server1 (with 2.9 deb package) and a new server2 (with the latest master branch version)

Both of them are connected to the same thingsboard. When I sent the mqtt data to my server1 (deb package), the server2 (master branch) showed error logs. (But I did not use server2 ip while sending mqtt-client request).

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3330, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.8/dist-packages/paho_mqtt-1.6.1-py3.8.egg/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 79, in _on_message
    self._on_decoded_message(content, message)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-2.9-py3.8.egg/thingsboard_gateway/tb_client/tb_gateway_mqtt.py", line 97, in _on_decoded_message
    target = content["device"] + "|*"
KeyError: 'device'

Maybe it may help you

imbeacon commented 2 years ago

Could you try to connect to thingsboard.cloud and send the data? Because unfortunately, I cannot reproduce the issue.

imbeacon commented 2 years ago

I was wrong about RPC's, do you save something to shared attributes in Rule Chains?

l33tkrew commented 2 years ago

I copied paho client.py from deb package into paho client.py in the master branch version. The problem is fixed. I did not do anything in gateway confs.

Before changing the content of client.py, I checked the differences between two client.py files. https://www.diffchecker.com/P1HDQ8Ui (Left side is client.py on master branch version, right one is client.py on 2.9 deb package version)

Now, I can use both MQTT and REST connectors. Also, I can spam data. But I did a very poor solution to fix the problem. It is important to use the latest client.py version. But, it seems impossible for my scenario for now. I do not have time to examine the client.py code. Maybe a small code block produces that error message.

I need an access token to connect .cloud. Let me create an account first to connect the cloud. Btw, I did not change any conf in web page. Therefore, I did not touch Rules Chains, RPC or etc.

l33tkrew commented 2 years ago

I could not find a way to get an access token for the gateway. I was able to create a device (click on is gateway checkbox). But i do not know how can i achieve this on the cloud system.

Btw, do I have to connect cloud? Because it is obvious that there is a compatibility issue with new paho client.py

imbeacon commented 2 years ago

No, it is not necessary. It may be fixed also with upgrading paho library via pip3 install paho-mqtt --upgrade .

l33tkrew commented 2 years ago

I have already the newest version of paho-mqtt. Whatever, I will use old version of client.py or paho-mqtt. I created bug issue. Maybe you may want to check it out.