MQTT with raw (binary) payload

[iw2lsi]

Hi all,

I spent some time trying to understand why in my custom connector I was not able to decode some raw payloads; the problem appear as soon as one of the ASCII char is > 127:

eg: echo -ne "\x84\xf4\xbd\x1e\x59\x5b\x11\x21" | mosquitto_pub -h my.broker.comt -t "gate/v1/raw" -s

/etc/thingsboard-gateway/config/mqtt_gate.json

{
  "broker": {
    "name":"my Broker",
    "host":"my.broker.com",
    "port":1883,
    "clientId": "ThingsBoard_gateway_raw",
    "security": {
      "type": "anonymous"
    }
  },
  "mapping": [
    {
      "topicFilter": "gate/v1/raw",
      "converter": {
        "type": "custom",
        "extension": "RawMqttUplinkConverter"
        }
    }
  ],
...

/usr/lib/python3/dist-packages/thingsboard_gateway/extensions/mqtt/raw_mqtt_uplink_converter.py

class RawMqttUplinkConverter(MqttUplinkConverter):
    def __init__(self, config):
        self.__config = config.get('converter')
        self.dict_result = {}

    def convert(self, config, data):
        try:
            self.dict_result["deviceName"] = "RAW" 
            self.dict_result["deviceType"] = "GATE"
            self.dict_result["telemetry"] = [] 

            log.info('RAW converter... received %d bytes', len(data))

at the end it turns out that the problem was in

/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt/mqtt_connector.py

and in more detail in the TBUtility.decode() function

    ...
    def _on_message(self, client, userdata, message):
        self.statistics['MessagesReceived'] += 1

        #content = TBUtility.decode(message) # does not work
        content = message.payload # works
    ...

which seems to convert my payload in utf-8...

Looking at TBUtility.decode() is seems that utf-8 conversion is on by default... and I wonder if there is a way to specify at connector/converter level that the payload should be threated, instead, as a raw sequence of bytes...

Best Regards

        Giampaolo

[imbeacon]

Hi @iw2lsi ,

Thank you for your interest in ThingsBoard IoT gateway. It was a bug please try to use the version from the master branch, also it will be available in the next release.

[iw2lsi]

thanks Illia... I'll check it ASAP...

    Giampaolo