search

thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.72k stars 829 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 4: invalid continuation byte #722

Closed santobedi closed 2 years ago

santobedi commented 2 years ago

Describe the issue Random value (unexpected) as telemetry while using Modbus connector. The Last Update Time always toggles with fixed past times. However, everything is fine with the Attributes (Client).

Connector name (If you need help with some connector/converter): Modbus Connector

Error traceback (If it was raised): 

""2022-02-02 09:58:18" - |ERROR| - [bytes_modbus_uplink_converter.py] - bytes_modbus_uplink_converter - convert - 90 - 'utf-8' codec can't decode byte 0xe2 in position 4: invalid continuation byte"
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-3.0.1-py3.8.egg/thingsboard_gateway/connectors/modbus/bytes_modbus_uplink_converter.py", line 76, in convert
    decoded_data = self.__decode_from_registers(decoder, configuration)
  File "/usr/local/lib/python3.8/dist-packages/thingsboard_gateway-3.0.1-py3.8.egg/thingsboard_gateway/connectors/modbus/bytes_modbus_uplink_converter.py", line 154, in __decode_from_registers
    result_data = decoded.decode('UTF-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 4: invalid continuation byte

Versions (please complete the following information):

samson0v commented 2 years ago

Hi @santobedi, send, please, your Modbus config file. Thanks for your interest in ThingsBoard IoT Gateway!

santobedi commented 2 years ago

Hello @samson0v, Following in my Modbus config file.

{
  "master": {
    "slaves": [
      {
        "unitId": 150,
        "deviceName": "energy_meter_electrical_1p_1_1a",
        "attributesPollPeriod": 5000,
        "timeseriesPollPeriod": 5000,
        "sendDataOnlyOnChange": false,
        "connectAttemptTimeMs": 5000,
        "connectAttemptCount": 5,
        "waitAfterFailedAttemptsMs": 300000,
        "attributes": [
          {
            "tag": "Vendor_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 16,
            "address": 31044
          },
          {
            "tag": "RF-ID",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 4,
            "address": 31026
          },
          {
            "tag": "Product_Code",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 16,
            "address": 31060
          },
          {
            "tag": "Firmware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31076
          },
          {
            "tag": "Hardware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31082
          },
          {
            "tag": "Serial_Number",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 10,
            "address": 31088
          },
          {
            "tag": "Asset_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 10,
            "address": 50
          },
          {
            "tag": "Label",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 3,
            "address": 31010
          }
        ],
        "timeseries": [
          {
            "tag": "current_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 2999
          },
          {
            "tag": "voltage_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3027
          },
          {
            "tag": "active_power",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3059,
            "divider": 1000
          },
          {
            "tag": "power_factor",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3083
          },
          {
            "tag": "active_energy_import",
            "type": "64int",
            "functionCode": 3,
            "objectsCount": 4,
            "address": 3203,
            "divider": 1000
          }
        ],
        "attributeUpdates": [
          {
            "tag": "shared_attribute_write",
            "type": "32int",
            "functionCode": 6,
            "objectsCount": 2,
            "address": 29
          }
        ],
        "rpc": [
          {
            "tag": "setValue",
            "type": "bits",
            "functionCode": 5,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "getValue",
            "type": "bits",
            "functionCode": 1,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "setCPUFanSpeed",
            "type": "32int",
            "functionCode": 16,
            "objectsCount": 2,
            "address": 33
          },
          {
            "tag": "getCPULoad",
            "type": "32int",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 35
          }
        ],
        "host": "10.0.1.244",
        "port": 502,
        "type": "tcp",
        "method": "socket",
        "timeout": 35,
        "byteOrder": "BIG",
        "wordOrder": null,
        "retries": null,
        "retryOnEmpty": null,
        "retryOnInvalid": null,
        "baudrate": null,
        "pollPeriod": 5000
      },
      {
        "unitId": 151,
        "deviceName": "energy_meter_electrical_1p_1_2a",
        "attributesPollPeriod": 5000,
        "timeseriesPollPeriod": 5000,
        "sendDataOnlyOnChange": false,
        "connectAttemptTimeMs": 5000,
        "connectAttemptCount": 5,
        "waitAfterFailedAttemptsMs": 300000,
        "attributes": [
          {
            "tag": "Vendor_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 16,
            "address": 31044
          },
          {
            "tag": "RF-ID",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 4,
            "address": 31026
          },
          {
            "tag": "Product_Code",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 16,
            "address": 31060
          },
          {
            "tag": "Firmware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31076
          },
          {
            "tag": "Hardware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31082
          },
          {
            "tag": "Serial_Number",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 10,
            "address": 31088
          },
          {
            "tag": "Asset_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 10,
            "address": 50
          },
          {
            "tag": "Label",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 3,
            "address": 31010
          }
        ],
        "timeseries": [
          {
            "tag": "current_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 2999
          },
          {
            "tag": "voltage_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3027
          },
          {
            "tag": "active_power",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3059,
            "divider": 1000
          },
          {
            "tag": "power_factor",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3083
          },
          {
            "tag": "active_energy_import",
            "type": "64int",
            "functionCode": 3,
            "objectsCount": 4,
            "address": 3203,
            "divider": 1000
          }
        ],
        "attributeUpdates": [
          {
            "tag": "shared_attribute_write",
            "type": "32int",
            "functionCode": 6,
            "objectsCount": 2,
            "address": 29
          }
        ],
        "rpc": [
          {
            "tag": "setValue",
            "type": "bits",
            "functionCode": 5,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "getValue",
            "type": "bits",
            "functionCode": 1,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "setCPUFanSpeed",
            "type": "32int",
            "functionCode": 16,
            "objectsCount": 2,
            "address": 33
          },
          {
            "tag": "getCPULoad",
            "type": "32int",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 35
          }
        ],
        "host": "10.0.1.244",
        "port": 502,
        "type": "tcp",
        "method": "socket",
        "timeout": 35,
        "byteOrder": "BIG",
        "wordOrder": null,
        "retries": null,
        "retryOnEmpty": null,
        "retryOnInvalid": null,
        "baudrate": null,
        "pollPeriod": 5000
      },
      {
        "unitId": 152,
        "deviceName": "energy_meter_electrical_3p_1_3a",
        "attributesPollPeriod": 5000,
        "timeseriesPollPeriod": 5000,
        "sendDataOnlyOnChange": false,
        "connectAttemptTimeMs": 5000,
        "connectAttemptCount": 5,
        "waitAfterFailedAttemptsMs": 300000,
        "attributes": [
          {
            "tag": "Vendor_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 16,
            "address": 31044
          },
          {
            "tag": "RF-ID",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 4,
            "address": 31026
          },
          {
            "tag": "Product_Code",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 16,
            "address": 31060
          },
          {
            "tag": "Firmware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31076
          },
          {
            "tag": "Hardware_Revision",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 6,
            "address": 31082
          },
          {
            "tag": "Serial_Number",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 10,
            "address": 31088
          },
          {
            "tag": "Asset_Name",
            "type": "string",
            "functionCode": 3,
            "objectsCount": 10,
            "address": 50
          },
          {
            "tag": "Label",
            "type": "string",
            "functionCode": 4,
            "objectsCount": 3,
            "address": 31010
          }
        ],
        "timeseries": [
          {
            "tag": "current_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 2999
          },
          {
            "tag": "voltage_phase_1",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3027
          },
          {
            "tag": "active_power",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3059,
            "divider": 1000
          },
          {
            "tag": "power_factor",
            "type": "32float",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 3083
          },
          {
            "tag": "active_energy_import",
            "type": "64int",
            "functionCode": 3,
            "objectsCount": 4,
            "address": 3203,
            "divider": 1000
          }
        ],
        "attributeUpdates": [
          {
            "tag": "shared_attribute_write",
            "type": "32int",
            "functionCode": 6,
            "objectsCount": 2,
            "address": 29
          }
        ],
        "rpc": [
          {
            "tag": "setValue",
            "type": "bits",
            "functionCode": 5,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "getValue",
            "type": "bits",
            "functionCode": 1,
            "objectsCount": 1,
            "address": 31
          },
          {
            "tag": "setCPUFanSpeed",
            "type": "32int",
            "functionCode": 16,
            "objectsCount": 2,
            "address": 33
          },
          {
            "tag": "getCPULoad",
            "type": "32int",
            "functionCode": 4,
            "objectsCount": 2,
            "address": 35
          }
        ],
        "host": "10.0.1.244",
        "port": 502,
        "type": "tcp",
        "method": "socket",
        "timeout": 35,
        "byteOrder": "BIG",
        "wordOrder": null,
        "retries": null,
        "retryOnEmpty": null,
        "retryOnInvalid": null,
        "baudrate": null,
        "pollPeriod": 5000
      }
    ]
  },
  "slave": null
}
santobedi commented 2 years ago

@samson0v On the MQTT connector within the same IoT gateway (device: raspberry pi 3, Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-1050-raspi armv7l) ), the attribute is fine and updated. However, telemetry is not.

samson0v commented 2 years ago

Hi @santobedi, can you, please, write steps to reproduce the bug and also send the registers data (using for example modpoll).

santobedi commented 2 years ago

Hello @samson0v ,

The Modbus TCP power tags are from Schneider. Attached herewith are the screenshots of Telemetry from Gateway device and one of the Modbus power tags. I've noticed that the time shown in Logs of Gateway is same as the time displayed as last time update of the Modbus device (devices created by the gateway in the TB). This last time update is not real time. Whereas the time and values displayed in the attribute section are fine.

Moreover, in the case of the MQTT connector, the values (time-series telemetry and attributes) are fine but the last time update in the telemetry is wrong similar to the Modbus connector. Gateway_telemetry Modbus_TCP

samson0v commented 2 years ago

@santobedi it seems that your system use the wrong time and it is not a Gateway bug because it uses a system date and time.

santobedi commented 2 years ago

@samson0v What about the wrong telemetry? And the raised error?

samson0v commented 2 years ago

@santobedi, can you, please, write steps to reproduce the error, wrong telemetry and also send the registers data (using for example modpoll). Thanks!

santobedi commented 2 years ago

@samson0v , I installed the gateway from source. Configured thetb_gateway.yaml ,modbus.json, and mqtt.json (/home/myDevice/thingsboard-gateway/thingsboard_gateway/config/), then ran python3 ./thingsboard_gateway/tb_gateway.py command. After this, the devices are created in the TB.

The wrong telemetry from one of the devices (created in TB via the gateway) is shown in the attached screenshot. I've attached the screenshot showing the client attribute (which has no issue at all) of the same device for your reference. If it's due to the system time, how come it affects only Telemetry, but not the Attributes? At this moment, I could not run the modpoll in our device (deployed in distance location), however, the same device had no such issues before (the earlier version of the gateway). Capture Capture1

samson0v commented 2 years ago

Hi @santobedi, is this bug still actual for you?

santobedi commented 2 years ago

Hello @samson0v ,

I have not experienced such bugs lately.

santobedi commented 2 years ago

Hello @samson0v, @imbeacon,

The same issue occurred again. For the devices created through the MQTT connector, only the attribute gets updated, Telemetry does not. This time the gateway is DEB.

santobedi commented 2 years ago

Hi @samson0v , @imbeacon ,

This time, the issue occurred when the same MQTT config file (that results in the same virtual devices) has been used for two different gateways (although only one gateway is present in the server and in practice) connecting to the same TB server. In other words, the attempt to update (with telemetry) already existing virtual devices (those were created from the old gateway) with the new gateway caused the issue. Removing all those devices including the gateway from the server, reinstalling the gateway, and several attempts of rebooting the gateway device finally solved the issue.