thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.75k stars 844 forks source link

[HELP] mqtt-connector could not connect #481

Closed RudolphAschmoneit closed 3 years ago

RudolphAschmoneit commented 3 years ago

Describe the issue Hey, I have a thingsboard CE v3.2 (Ubuntu installation) and a mosquitto mqtt broker v2.0.7 (docker container, via docker-compose) running on the same ubuntu 20.04 server. Thingsboard using mqtt standard port 1883, mosquitto using port 1884. Now I try to get up and running a thingsboard-gateway, to subscribe to the broker and push the data to the thingsboard instance. I try to run the gateway as a docker container via docker-compose up -d --build. I tested my credentials separately before (username and password for the broker via mosquitto_sub -q 1 -h 127.0.0.1 -p 1884 -u _my_user_ -P _my_password_ -t sensor/data -d and access token for thingsboard via mosquitto_pub -d -q 1 -h "127.0.0.1" -p "1883" -t "v1/devices/me/telemetry" -u "_my_access_token_" -m "{"pressure":42}" ) - everything works fine. If I now try to get the gateway up I get the Error ERROR - [mqtt_connector.py] - mqtt_connector - 193 - [Errno 111] Connection refused which means imho the gateway's connection request to the broker is rejecteded. I spent several hours checking my configuration, but I could not find the bug, maybe you could help me. I'm assuming that the name of the broker in the mqtt.json is not relevant for my issue? There are also no log files in my host volume (tb-gateway/logs/), so I attach the container logs. For testing purposes I only try to get some of the data send to the specific topic at the moment, but this should also not be the reason for the error (?) Thank you very much in Advance for any help!

Configuration tb_gateway.yaml:

thingsboard:
  host: 127.0.0.1
  port: 1883
  security:
    accessToken: _my_access_token_
storage:
  type: memory
  read_records_count: 10
  max_records_count: 100
connectors:
  -
    name: MQTT Broker Connector
    type: mqtt
    configuration: mqtt.json

mqtt.json:

{
    "broker": {
      "name":"Mosquitto Develop",
      "host":"127.0.0.1",
      "port":1884,
      "security": {
        "type": "basic",
        "username": "my_user",
        "password": "my_password"
      }
    },
    "mapping": [
      {
        "topicFilter": "sensor/data",
        "converter": {
          "type": "json",
          "deviceNameJsonExpression": "${DeviceID}",
          "deviceTypeJsonExpression": "default",
          "timeout": 60000,
          "attributes": [],
          "timeseries": [
            {
              "type": "double",
              "key": "temperature",
              "value": "${temperature}[0]"
            },
            {
              "type": "double",
              "key": "person",
              "value": "${person}"
            },
            {
                "type": "double",
                "key": "timestamp",
                "value": "${timestamp}"
            },
            {
                "type": "double",
                "key": "Status",
                "value": "${Status}"
            }
          ]
        }
      }
    ],
    "connectRequests": [],
    "disconnectRequests": [],
    "attributeRequests": [],
    "attributeUpdates": [],
    "serverSideRpc": []
}

docker-compose.yml (my tb-gateway directory on the host are in the same directory as the docker-compose.yml):

version: '3.8'

services:
  tb-gateway:
    image: thingsboard/tb-gateway:2.5.5.2
    container_name: thingsboard_mqtt_gateway
    restart: always
    ports:
      - "5000:5000"
    volumes:
      - ./tb-gateway/config:/thingsboard_gateway/config
      - ./tb-gateway/extensions:/thingsboard_gateway/extensions
      - ./tb-gateway/logs:/thingsboard_gateway/logs

Connector name (If you need help with some connector/converter): MQTT Connector

Log files from docker-compose logs -t :

thingsboard_mqtt_gateway | 2021-02-19T14:57:50.274474959Z ""2021-02-19 14:57:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 72 - Gateway starting..."
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.278769488Z ""2021-02-19 14:57:50" - DEBUG - [tb_updater.py] - tb_updater - 69 - Checking for new version"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.279325298Z ""2021-02-19 14:57:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 77 - ThingsBoard IoT gateway version: 2.5.5.2"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.283144263Z ""2021-02-19 14:57:50" - DEBUG - [connectionpool.py] - connectionpool - 971 - Starting new HTTPS connection (1): updates.thingsboard.io:443"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.283737349Z ""2021-02-19 14:57:50" - DEBUG - [tb_client.py] - tb_client - 125 - connecting to ThingsBoard"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284003445Z ""2021-02-19 14:57:50" - DEBUG - [tb_device_mqtt.py] - tb_device_mqtt - 301 - Subscribed to * with id 1"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284898711Z ""2021-02-19 14:57:50" - INFO - [tb_gateway_mqtt.py] - tb_gateway_mqtt - 176 - Subscribed to *|* with id 1"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284912203Z ""2021-02-19 14:57:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log service"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284917100Z ""2021-02-19 14:57:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log extension"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284920934Z ""2021-02-19 14:57:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log converter"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.284971428Z ""2021-02-19 14:57:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log connector"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285000979Z ""2021-02-19 14:57:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log tb_connection"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285173128Z ""2021-02-19 14:57:50" - DEBUG - [memory_event_storage.py] - memory_event_storage - 26 - Memory storage created with following configuration:
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285179911Z Max size: 100
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285183398Z  Read records per time: 10"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285466576Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 96 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3e0b3d2400>, origin='/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/json_mqtt_uplink_converter.py')"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.285603508Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 103 - <module 'MqttConnector' from '/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/json_mqtt_uplink_converter.py'>"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.286906653Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 96 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3e0b3d2100>, origin='/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/mqtt_uplink_converter.py')"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.287087795Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 103 - <module 'MqttConnector' from '/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/mqtt_uplink_converter.py'>"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.287376628Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 96 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3e0b3d2580>, origin='/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/mqtt_connector.py')"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.287460863Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 103 - <module 'MqttConnector' from '/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt/mqtt_connector.py'>"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288187810Z ""2021-02-19 14:57:50" - DEBUG - [tb_utility.py] - tb_utility - 107 - Import MqttConnector from /usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/mqtt."
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288521894Z ""2021-02-19 14:57:50" - DEBUG - [mqtt_connector.py] - mqtt_connector - 142 - Mandatory key 'topicFilter' found in mapping handler: {"topicFilter": "vraktion/airq1/data", "converter": {"type": "json", "deviceNameJsonExpression": "${DeviceID}", "deviceTypeJsonExpression": "AirQ", "timeout": 60000, "attributes": [], "timeseries": [{"type": "double", "key": "temperature", "value": "${temperature}"}, {"type": "double", "key": "person", "value": "${person}"}, {"type": "double", "key": "timestamp", "value": "${timestamp}"}, {"type": "double", "key": "Status", "value": "${Status}"}]}}"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288642927Z ""2021-02-19 14:57:50" - DEBUG - [mqtt_connector.py] - mqtt_connector - 142 - Mandatory key 'converter' found in mapping handler: {"topicFilter": "vraktion/airq1/data", "converter": {"type": "json", "deviceNameJsonExpression": "${DeviceID}", "deviceTypeJsonExpression": "AirQ", "timeout": 60000, "attributes": [], "timeseries": [{"type": "double", "key": "temperature", "value": "${temperature}"}, {"type": "double", "key": "person", "value": "${person}"}, {"type": "double", "key": "timestamp", "value": "${timestamp}"}, {"type": "double", "key": "Status", "value": "${Status}"}]}}"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288760733Z ""2021-02-19 14:57:50" - DEBUG - [mqtt_connector.py] - mqtt_connector - 150 - mapping handler has all mandatory keys => accepted: {"topicFilter": "vraktion/airq1/data", "converter": {"type": "json", "deviceNameJsonExpression": "${DeviceID}", "deviceTypeJsonExpression": "AirQ", "timeout": 60000, "attributes": [], "timeseries": [{"type": "double", "key": "temperature", "value": "${temperature}"}, {"type": "double", "key": "person", "value": "${person}"}, {"type": "double", "key": "timestamp", "value": "${timestamp}"}, {"type": "double", "key": "Status", "value": "${Status}"}]}}"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288857425Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted mapping handlers: 1"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.288953580Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected mapping handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289022648Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted serverSideRpc handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289106400Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected serverSideRpc handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289169938Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted connectRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289254027Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected connectRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289326535Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted disconnectRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289406188Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected disconnectRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289492808Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted attributeRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289587393Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected attributeRequests handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289686489Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 153 - Number of accepted attributeUpdates handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.289724525Z ""2021-02-19 14:57:50" - INFO - [mqtt_connector.py] - mqtt_connector - 157 - Number of rejected attributeUpdates handlers: 0"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.290852574Z ""2021-02-19 14:57:50" - ERROR - [mqtt_connector.py] - mqtt_connector - 193 - [Errno 111] Connection refused"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.290873648Z ""2021-02-19 14:57:50" - DEBUG - [tb_gateway_service.py] - tb_gateway_service - 663 - Loaded devices:
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.290877973Z  {'LOAD_TEST_DEVICE0': 'MQTT Broker Connector'}"
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.291355051Z ""2021-02-19 14:57:50" - DEBUG - [tb_gateway_service.py] - tb_gateway_service - 361 - Send data Thread has been started successfully."
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.291516436Z ""2021-02-19 14:57:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 135 - Gateway started."
thingsboard_mqtt_gateway | 2021-02-19T14:57:50.722989941Z ""2021-02-19 14:57:50" - DEBUG - [connectionpool.py] - connectionpool - 452 - https://updates.thingsboard.io:443 "POST /api/tb-gateway/updates HTTP/1.1" 200 None"
thingsboard_mqtt_gateway | 2021-02-19T14:57:51.285483130Z ""2021-02-19 14:57:51" - DEBUG - [tb_client.py] - tb_client - 125 - connecting to ThingsBoard"
thingsboard_mqtt_gateway | 2021-02-19T14:57:52.287097423Z ""2021-02-19 14:57:52" - DEBUG - [tb_client.py] - tb_client - 125 - connecting to ThingsBoard"
thingsboard_mqtt_gateway | 2021-02-19T14:57:53.288740723Z ""2021-02-19 14:57:53" - DEBUG - [tb_client.py] - tb_client - 125 - connecting to ThingsBoard"
thingsboard_mqtt_gateway | 2021-02-19T14:57:54.290337888Z ""2021-02-19 14:57:54" - DEBUG - [tb_client.py] - tb_client - 125 - connecting to ThingsBoard"

Versions:

buckyy2018 commented 3 years ago

hi In your config , publish mqtt message to tb port in 1884. and topic string"topicFilter": "/sensor/data/", in mosquitto config acl file add usr and same topic: right readwrite.

buckyy2018 commented 3 years ago

{ "type": "double", "key": "timestamp", "value": "${timestamp}" }, only field in mqtt.json can be translate into tb.

RudolphAschmoneit commented 3 years ago

hi In your config , publish mqtt message to tb port in 1884. and topic string"topicFilter": "/sensor/data/", in mosquitto config acl file add usr and same topic: right readwrite.

Hey, thanks for your answer, but I do not really get what you mean by "In your config , publish mqtt message to tb port in 1884". The Port of the thingsboard mqtt is 1883, and this is also the port I used in tb_gateway.yaml, like the tb_gateway documentation says. The Port in mqtt.json is the port of the external broker I want the gateway to connect with (also according to thingsboard_gateway documentation) so this is 1884. Sorry If I'm be a bit slow on that, or getting something wrong.

espring commented 3 years ago

What is mosquitto's log?

buckyy2018 commented 3 years ago

in mosqutto conf file you have to set log file then create log file in setting dir or you start 模式去天堂service in console you can see log

发自我的iPhone

在 2021年3月19日,11:39,espring @.***> 写道:

 What is mosquitto's log?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

imbeacon commented 3 years ago

Hi @RudolphAschmoneit ,

Thank you for your interest in ThingsBoard IoT gateway. This issue may be caused by routing requests, could you try to use your local IP in your network instead of 127.0.0.1?

RudolphAschmoneit commented 3 years ago

Hey @zbeacon, @buckyy2018 and @espring Thanks for your answers and sorry for my late reply. I had to finish some other projects here at work. The hint from @zbeacon fixed the issue. Using the local IP of the server instead of 127.0.0.1. Thanks a lot for your help! Kind Regards