[HELP] custom converters not found with docker image

[ilseva]

Describe the bug I create custom converters for shelly devices and I copy them in <tb-extensions-volume>/mqtt.
I modify the mqtt.json file adding configuration for my topics

{
      "topicFilter": "shellies/+/sensor/#",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttDwConverter"
      }
    },
    {
      "topicFilter": "shellies/+/relay/#",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttPlugConverter"
      }
    }

When I restart the container I see this logs

""2021-11-03 09:49:49" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/sensor/# topic" ""2021-11-03 09:49:49" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/relay/# topic"

I have to copy the converters in /usr/local/lib/python3.7/site-packages/thingsboard_gateway-2.9-py3.7.egg/thingsboard_gateway/extensions/mqtt

Versions (please complete the following information):

• OS: [e.g. Ubuntu 21.04]
• docker image thingsboard/tb-gateway:2.9

Thanks for your help.

[samson0v]

Hi @ilseva, it seems that your Gateway using a not right config file. When you are using Gateway with docker, config files located at /etc/thingsboard-gateway/config/ and /var/lib/thingsboard_gateway/extensions/ . Please check these paths and make sure that there are the right config files.

Thanks for your interest in ThingsBoard IoT Gateway.

[ilseva]

Hi @samson0v, thanks for your quick reply. To use gateway with docker I followed this guide: https://thingsboard.io/docs/iot-gateway/install/docker-installation/

I mount these 3 volumes:

• .tb-gateway/config:/thingsboard_gateway/config
• .tb-gateway/extensions:/thingsboard_gateway/extensions
• .tb-gateway/logs:/thingsboard_gateway/logs

The config files contained in my local folder are correctly read by gateway when it starts.

I looked at Dockerfile in the repository and I saw that the entrypoint is a bash script that sets the config folder, but not the extensions one.

Thanks.

[samson0v]

@ilseva Can you, please, send the code of your custom converter to the email vbidochka@thingsboard.io for investigation.

[ilseva]

Hi @samson0v, there is nothing secrete :-) I copied the code below, maybe it could help someone else

I have to change the topic filters because we have some problems regarding the data validation and consequently with the dispatch of the telemetry data, but I have to understand the reason before opening an issue. This is my "new" configuration

mqtt.json config (inside mapping[])

{
      "topicFilter": "shellies/+/sensor/battery",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttDwConverter"
      }
    },
    {
      "topicFilter": "shellies/+/sensor/temperature",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttDwConverter"
      }
    },
    {
      "topicFilter": "shellies/+/sensor/state",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttDwConverter"
      }
    },
    {
      "topicFilter": "shellies/+/sensor/lux",
      "converter": {
        "type": "custom",
        "extension": "ShellyMqttDwConverter"
      }
    }

ShellyMqttDwConverter Converter

import re
from thingsboard_gateway.connectors.mqtt.mqtt_uplink_converter import MqttUplinkConverter, log

class ShellyMqttDwConverter(MqttUplinkConverter):
    def __init__(self, config):
        self.__config = config.get('converter')
        self.dict_result = {}

    def convert(self, topic, body):
        try:
            matched = re.search(r'shellies/(shellydw2-\w*?)/sensor/(\w+)', topic)
            if matched:
                key = matched.group(2)
                self.dict_result["deviceName"] = matched.group(1)
                self.dict_result["deviceType"] = "Shelly Door/Window"
                self.dict_result["telemetry"] = []
                dw_status = {key: body}
                self.dict_result["telemetry"].append(dw_status)
            log.info('send %s telemetry for %s value %s', self.dict_result["deviceName"], key, body)
            return self.dict_result
        except Exception as e:
            log.exception('Error in converter message: \n%s\n', body)
            log.exception(e)

Thanks for your help

[samson0v]

Hi @ilseva,

I have to copy the converters in /usr/local/lib/python3.7/site-packages/thingsboard_gateway-2.9-py3.7.egg/thingsboard_gateway/extensions/mqtt

Please, copy converters to the mount point at .tb-gateway/extensions and start your docker image

For example: If I have the next mount points

It means that my extensions files have to locate at /Users/user/.tb-gateway/...

If I have the next mount points:

It means that the .tb-gateway folder has to be in the same directory as your docker file.

After a try, let us know if it was helping

[ilseva]

Hi @samson0v,

Please, copy converters to the mount point at .tb-gateway/extensions and start your docker image

this is exactly what I did.

If I have the next mount points:

It means that the .tb-gateway folder has to be in the same directory as your docker file.

this is my actual configuration. I have a folder with a docker-compose file and this folder structure

As I wrote in an other message of this thread, config files contained in .tb-gateway/config are correctly read by tb-gateway process. I try to copy my custom converters both in .tb-gateway/extensions and in .tb-gateway/extensions/mqtt Always same result

""2021-11-09 08:32:09" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/sensor/battery topic"
""2021-11-09 08:32:09" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/sensor/temperature topic"
""2021-11-09 08:32:09" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/sensor/state topic"
""2021-11-09 08:32:09" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for shellies/+/sensor/lux topic"

Thanks.

[samson0v]

@ilseva can you, pls, send what the extensions folder contain?

[ilseva]

Hi @samson0v, this is the output of ls -laR of my host volume .tb-gateway/extensions

[ilseva]

A clarification: I'm not recreating any docker image from thingsboard-gateway source. I'm using the one deployed on docker hub.

[samson0v]

Hi @ilseva, please, send your docker-compose file. Thanks!

[ilseva]

here it is

version: '3.3'
services:
  tb-gateway:
    container_name: tb-gateway
    image: thingsboard/tb-gateway:2.9
    volumes:
      - .tb-gateway/config:/thingsboard_gateway/config
      - .tb-gateway/extensions:/thingsboard_gateway/extensions
      - .tb-gateway/logs:/thingsboard_gateway/logs

[samson0v]

@ilseva please, add the next volume to your docker-compose file - .tb-gateway/extensions:/usr/local/lib/python3.7/site-packages/thingsboard_gateway-2.9-py3.7.egg/thingsboard_gateway/extensions

And don't forget to recreate containers, so use the next for the first time starting: docker-compose down docker-compose up --build -d

The next thing is to make sure that the converter is copied to the right folder:

1. docker ps
2. choose gateway container id
3. docker exec -it _container_id_ bash
4. cd /usr/local/lib/python3.7/site-packages/thingsboard_gateway-2.9-py3.7.egg/thingsboard_gateway/extensions/mqtt
5. ls
6. and you have to see shelly_converter.py

Have fun) Let us know if that work for you! It is a bug, we will fix it but for now, use the above.

Sorry for inconvenience!

[ilseva]

Thanks @samson0v ! With this workaround custom converters are loaded by the gateway.

I look at the code, but I'm not a python expert: why do you install the gateway as a module with setup during docker image creation?

Thanks for you precious support.

[samson0v]

Hi @ilseva, we fixed this bug so in the next release you can use docker image without extra configurations.