search

thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.75k stars 844 forks source link

[BUG] Gateway docker image is not loading mqtt extensions (wrong internal pythonpath?) #770

Closed rduivenvoorde closed 2 years ago

rduivenvoorde commented 2 years ago

Describe the bug

In a setup running containers I fail to use a custom mqtt python extension in the mounted /thingsboard_gateway/extensions directory.

The gateway only 'finds' the custom python code when I copy it IN the running container in: /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions/mqtt

To reproduce, I went back to a minimal setup by just using a official docker image on a laptop and copy the custom extension in it:

1) run vanilla official image:

docker run -it -v ~/.tb-gateway/logs:/thingsboard_gateway/logs -v ~/.tb-gateway/extensions:/thingsboard_gateway/extensions -v ~/.tb-gateway/config:/thingsboard_gateway/config --name tb-gateway thingsboard/tb-gateway

2) bring down, edit mqtt.json so it connects to an actual local broker, edit tb_gateway.yaml to connect to thingsboard.cloud for example

3) copy the available default ~/.tb-gateway/extensions:/thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter.py and create a custom_mqtt_uplink_converter2.py next to it (also edit the classname in it from CustomMqttUplinkConverter to CustomMqttUplinkConverter2

4) now edit the mqtt.json in ~/.tb-gateway/config:/thingsboard_gateway/config to include a new topic, copy the custom/sensors one so it should use the new CustomMqttUplinkConverter2 class

    {
      "topicFilter": "/custom/sensors/+",
      "converter": {
        "type": "custom",
        "extension": "CustomMqttUplinkConverter",
        "extension-config": {
          "temperatureBytes": 2,
          "humidityBytes": 2,
          "batteryLevelBytes": 1
        }
      }
    },
    {
      "topicFilter": "/custom/sensors2/+",
      "converter": {
        "type": "custom",
        "extension": "CustomMqttUplinkConverter2",
        "extension-config": {
          "temperatureBytes": 2,
          "humidityBytes": 2,
          "batteryLevelBytes": 1
        }
      }
    }

Now fire up the container again, and note the following in the output: ""2022-03-10 11:35:58" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for /custom/sensors2/+ topic"

I think I went pretty far in debugging this (adding some tools to an images, editing the python files to see some output, see https://github.com/thingsboard/thingsboard-gateway/issues/750), and I think the crux is that the pythonpath in the container is NOT picking up the python code in the /thingsboard_gateway/extensions directory.

By going into the container and trying to call/see TBModuleLoader.find_paths() and TBModuleLoader.import_module(...) I see:

[
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions', 
'/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors'
]

While I think(!) that that should also show the /thingsboard_gateway/extensions directory, isn't it?

Going INTO the container: docker exec -it tb-gateway /bin/bash, you can see the custom_mqtt_uplink_converter2.py is there:

# ls /thingsboard_gateway/extensions/mqtt
__init__.py  custom_mqtt_uplink_converter.py  custom_mqtt_uplink_converter2.py

but apparently not found... But copy it to the python install in the container:

cp /thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter2.py /usr/local//lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions/mqtt/

And restarting the container:

docker start -a -i tb-gateway

and you will see the CustomMqttUplinkConverter2 being loaded:

""2022-03-10 12:22:06" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import CustomMqttUplinkConverter from /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions/mqtt."
""2022-03-10 12:22:06" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /custom/sensors/+"
""2022-03-10 12:22:06" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import CustomMqttUplinkConverter2 from /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions/mqtt."
""2022-03-10 12:22:06" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /custom/sensors2/+"

I hope others can reproduce this in this way, Thanks.

Connector name (If bug in the some connector): mqtt (extension

Error traceback (If available):

$ docker run -it -v ~/.tb-gateway/logs:/thingsboard_gateway/logs -v ~/.tb-gateway/extensions:/thingsboard_gateway/extensions -v ~/.tb-gateway/config:/thingsboard_gateway/config --name tb-gateway thingsboard/tb-gateway
""2022-03-10 11:35:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 93 - Gateway starting..."
""2022-03-10 11:35:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 98 - ThingsBoard IoT gateway version: 3.0.1"
""2022-03-10 11:35:58" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 175 - Subscribed to *|* with id 1 for device *"
""2022-03-10 11:35:58" - |WARNING| - [tb_updater.py] - tb_updater - check_for_new_version - 85 - Cannot connect to the update service. PLease check your internet connection."
""2022-03-10 11:35:58" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import MqttConnector from /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors/mqtt."
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted mapping handlers: 4"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected mapping handlers: 0"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted serverSideRpc handlers: 2"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected serverSideRpc handlers: 0"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted connectRequests handlers: 2"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected connectRequests handlers: 0"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted disconnectRequests handlers: 2"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected disconnectRequests handlers: 0"
""2022-03-10 11:35:58" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'topicExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-03-10 11:35:58" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 151 - Mandatory key 'valueExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-03-10 11:35:58" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - attributeRequests handler is missing some mandatory keys => rejected: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted attributeRequests handlers: 0"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected attributeRequests handlers: 1"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 165 - Number of accepted attributeUpdates handlers: 1"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 169 - Number of rejected attributeUpdates handlers: 0"
""2022-03-10 11:35:58" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 180 - Gateway started."
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 241 - MQTT Broker Connector connected to 192.168.178.138:1883 - successfully."
""2022-03-10 11:35:58" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import JsonMqttUplinkConverter from /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/connectors/mqtt."
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /sensor/data"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /sensor/+/data"
""2022-03-10 11:35:58" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import CustomMqttUplinkConverter from /usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/extensions/mqtt."
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 284 - Connector "MQTT Broker Connector" subscribe to /custom/sensors/+"
""2022-03-10 11:35:58" - |ERROR| - [mqtt_connector.py] - mqtt_connector - _on_connect - 269 - Cannot find converter for /custom/sensors2/+ topic"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /sensor/data, subscription message id = 1"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /sensor/+/data, subscription message id = 2"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic /custom/sensors/+, subscription message id = 3"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/connect, subscription message id = 4"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/+/connect, subscription message id = 5"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/disconnect, subscription message id = 6"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
""2022-03-10 11:35:58" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription success to topic sensor/+/disconnect, subscription message id = 7"
""2022-03-10 11:35:59" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 139 - connection SUCCESS"
""2022-03-10 11:35:59" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 175 - Subscribed to *|* with id 2 for device *"
""2022-03-10 11:37:27" - |ERROR| - [custom_mqtt_uplink_converter.py] - custom_mqtt_uplink_converter - convert - 45 - Error in converter, for config:

Versions (please complete the following information):

LennardWesterveld commented 2 years ago

That's because it runs the setup.py inside the docker build. That is also installing thingsboard-gateway it self inside the site-packages directory. If you should replace the setup.py with pip install -e . then the thingsboard-gateway directory will stay where it is placed.

rduivenvoorde commented 2 years ago

@LennardWesterveld mmm, ok, so then is this considered an issue in the container run?

Then I see 2 options:

Anybody can give info about what would be the best option here?

LennardWesterveld commented 2 years ago

@rduivenvoorde this would help you I guess ;) https://gist.github.com/LennardWesterveld/bb8c2c95c2f6649c1306c56478caebd2 this is my docker file for now. In my opinion thingsboard-gateway, docker file itself should also move towards this setup.

samson0v commented 2 years ago

Hi @rduivenvoorde, you can try to use this solution.

rduivenvoorde commented 2 years ago

@samson0v yes, thanks that is what I do now. In that issue, you say: "Hi ilseva, we fixed this bug so in the next release you can use docker image without extra configurations." That was nov 2021, did that already land in a (docker)-release?

samson0v commented 2 years ago

Hi @rduivenvoorde it partially solved the problem but as you can see there are some exceptions. Will fix.

raomin commented 2 years ago

Hi, I also ran into this issue and I would suggest to keep this issue open until it is fixed. ie docker image users can use the ~/.tb-gateway/extensions folder to put in their custom converter.