search

thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.72k stars 829 forks source link

[HELP] iot-gateway does not receive MQTT message #645

Closed raomin closed 2 years ago

raomin commented 2 years ago

Describe the issue

I configured the iot-gateway to receive and publish messages from the TTN MQTT broker. While the connection is OK, the messages arriving on the MQTT broker (and confirmed with mosquitto_sub on the same topic) don't seem to be processed by the gateway and do not reach thingsboard.

Configuration (mqtt.json)

{
  "broker": {
    "name": "TTN Test AirData",
    "host": "eu1.cloud.thethings.network",
    "port": 1883,
    "clientId": "ThingsBoard_gateway",
    "maxMessageNumberPerWorker": 10,
    "maxNumberOfWorkers": 100,
    "security": {
      "type": "basic",
      "username": "***************",
      "password": "********************"
    }
  },
  "mapping": [
    {
      "topicFilter": "v3/testairdata@ttn/devices/+/up",
      "converter": {
        "type": "json",
        "filterExpression": "",
        "deviceNameTopicExpression": "^(.*)$",
        "deviceNameJsonExpression": "${$.end_device_ids.device_id}",
        "deviceTypeJsonExpression": "Thermometer",
        "timeout": 60000,
        "attributes": [
                          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          }
        ],
        "timeseries": [
          {
          "type": "double",
          "key": "co2",
          "value": "${$.uplink_message.decoded_payload.co2}"
          },
          {
          "type": "double",
          "key": "eco2",
          "value": "${$.uplink_message.decoded_payload.eco2}"
          },
          {
          "type": "double",
          "key": "rssi",
          "value": "${$.uplink_message.decoded_payload.rssi}"
          },
          {
          "type": "double",
          "key": "snr",
          "value": "${$.uplink_message.decoded_payload.snr}"
          }
        ]
      }
    }
  ]
}

No errors in logs:

tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 76 - Gateway starting..."
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 81 - ThingsBoard IoT gateway version: 2.9"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 178 - Subscribed to *|* with id 1 for device *"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import MqttConnector from /thingsboard_gateway/thingsboard_gateway/connectors/mqtt."
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 167 - Number of accepted mapping handlers: 1"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 171 - Number of rejected mapping handlers: 0"
tb-gateway_1  | ""2021-12-04 23:43:48" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 142 - 'serverSideRpc' section missing from configuration"
tb-gateway_1  | ""2021-12-04 23:43:48" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 142 - 'connectRequests' section missing from configuration"
tb-gateway_1  | ""2021-12-04 23:43:48" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 142 - 'disconnectRequests' section missing from configuration"
tb-gateway_1  | ""2021-12-04 23:43:48" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 142 - 'attributeRequests' section missing from configuration"
tb-gateway_1  | ""2021-12-04 23:43:48" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 142 - 'attributeUpdates' section missing from configuration"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - __init__ - 151 - Gateway started."
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 244 - MQTT Broker Connector connected to eu1.cloud.thethings.network:1883 - successfully."
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [tb_loader.py] - tb_loader - import_module - 66 - Import JsonMqttUplinkConverter from /thingsboard_gateway/thingsboard_gateway/connectors/mqtt."
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_connect - 286 - Connector "MQTT Broker Connector" subscribe to v3/testairdata@ttn/devices/+/up"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
tb-gateway_1  | ""2021-12-04 23:43:48" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 335 - "MQTT Broker Connector" subscription success to topic v3/testairdata@ttn/devices/+/up, subscription message id = 1"
-- Stays there with no more logs

In the meantime:

# mosquitto_sub -v -t '#' -h eu1.cloud.thethings.network -p 1883 -u testairdata@ttn -P *********** -q 1
v3/testairdata@ttn/devices/eui-70b3d57ed00490e5/up {"end_device_ids":{"device_id":"eui-70b3d57ed00490e5","application_ids":{"application_id":"testairdata"},"dev_eui":"70B3D57ED00490E5","join_eui":"0000000000000000","dev_addr":"260BA54B"},"correlation_ids":["as:up:01FP648XYAQCC3SRD4YEXN123Y","gs:conn:01FP5XX4KQTN6MNX5J0EZ331KS","gs:up:host:01FP5XX4M0RKP367DF7X81SY05","gs:uplink:01FP648XQAFPG7N3N4HW6ZW14C","ns:uplink:01FP648XQBH63K0PGD66WZY23F","rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01FP648XQBFV9RR53ZWX4DKWT6","rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01FP648XYAB12K2KP60BWFKEYZ"],"received_at":"2021-12-05T20:23:57.131041832Z","uplink_message":{"session_key_id":"AX2K0aqGapGPKCg7hEqlSA==","f_port":1,"f_cnt":681,"frm_payload":"uGaMxHsAuwvVDwAA+pIDAN4W9wc=","decoded_payload":{"co2":3003,"eco2":4053,"humidity":123,"rssi":2039,"snr":5854,"temp":-1123.2099609375,"uptime":234234},"rx_metadata":[{"gateway_ids":{"gateway_id":"manosque01","eui":"AC1F09FFFE06059A"},"time":"2021-12-05T20:23:56.876948118Z","timestamp":2380791622,"rssi":-82,"channel_rssi":-82,"snr":13.5,"location":{"latitude":43.83953760420872,"longitude":5.785639143810446,"source":"SOURCE_REGISTRY"},"uplink_token":"ChgKFgoKbWFub3NxdWUwMRIIrB8J//4GBZoQxv6f7wgaDAjcv7SNBhCH4oKwAyDw0vSRpcIBKgwI3L+0jQYQltWUogM="}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867900000","timestamp":2380791622,"time":"2021-12-05T20:23:56.876948118Z"},"received_at":"2021-12-05T20:23:56.907478905Z","consumed_airtime":"0.071936s","network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-eu1"}}}

Versions (please complete the following information):

I'm certainly missing something... but what?

imbeacon commented 2 years ago

Hi @raomin,

Please stop the gateway, set logging level from INFO to DEBUG in logs.conf file and run the gateway again. Then please send logs.

raomin commented 2 years ago

Hi @zbeacon, thanks for your feedback. I've put everything in DEBUG and here is what I saw:

tb-gateway_1  | ""2021-12-06 21:49:37" - |DEBUG| - [tb_client.py] - tb_client - run - 124 - connecting to ThingsBoard"
tb-gateway_1  | ""2021-12-06 21:49:38" - |DEBUG| - [tb_client.py] - tb_client - run - 124 - connecting to ThingsBoard"
tb-gateway_1  | ""2021-12-06 21:49:39" - |DEBUG| - [tb_client.py] - tb_client - run - 124 - connecting to ThingsBoard"
tb-gateway_1  | ""2021-12-06 21:49:40" - |DEBUG| - [tb_client.py] - tb_client - run - 124 - connecting to ThingsBoard"

Thing is, tb-gateway is running as docker image, and so is tbio. And my tb-gateway.conf what pointing to:

thingsboard:
  host: localhost
  port: 1883

... which would not lead to thingsboard but the localhost of the docker namespace... I changed it to the public IP of the host, and voilà! 

tb-gateway_1  | ""2021-12-06 21:52:08" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceName': 'eui-70b3d57ed00490e5', 'deviceType': '${uplink_message.version_ids.model_id}', 'attributes': [{'brand': '${uplink_message.version_ids.brand_id}'}], 'telemetry': [{'co2': '3003'}, {'eco2': '4053'}, {'rssi': '2039'}, {'snr': '5854'}]}"
tb-gateway_1  | ""2021-12-06 21:52:08" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic v3/testairdata@ttn/devices/eui-70b3d57ed00490e5/up"
tb-gateway_1  | ""2021-12-06 21:52:08" - |DEBUG| - [tb_gateway_service.py] - tb_gateway_service - __save_persistent_devices - 793 - Saved connected devices."
tb-gateway_1  | ""2021-12-06 21:52:44" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceName': 'eui-70b3d57ed00490e5', 'deviceType': '${uplink_message.version_ids.model_id}', 'attributes': [{'brand': '${uplink_message.version_ids.brand_id}'}], 'telemetry': [{'co2': '3003'}, {'eco2': '4053'}, {'rssi': '2039'}, {'snr': '5854'}]}"
tb-gateway_1  | ""2021-12-06 21:52:44" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic v3/testairdata@ttn/devices/eui-70b3d57ed00490e5/up"
tb-gateway_1  | ""2021-12-06 21:53:20" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - run - 718 - {'deviceName': 'eui-70b3d57ed00490e5', 'deviceType': '${uplink_message.version_ids.model_id}', 'attributes': [{'brand': '${uplink_message.version_ids.brand_id}'}], 'telemetry': [{'co2': '3003'}, {'eco2': '4053'}, {'rssi': '2039'}, {'snr': '5854'}]}"
tb-gateway_1  | ""2021-12-06 21:53:20" - |DEBUG| - [mqtt_connector.py] - mqtt_connector - _save_converted_msg - 352 - Successfully converted message from topic v3/testairdata@ttn/devices/eui-70b3d57ed00490e5/up"

Thanks for this @zbeacon.