[HELP] - Githubissues

RudolphAschmoneit commented 3 years ago

Describe the issue Hey at all. I try to get a thingsboard_gateway up and running, which describes to a mosquitto broker hosted on the same machine and piping it to a thingsboard also hosted on the same machine. Tb and tb_gw are setuped as deamons. I use the mqtt connector. The gateway could connect to the brooker and the gateway succesful, but it doesn't find the devices:

Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 160 - Number of accepted attributeUpdates handlers: 0"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 164 - Number of rejected attributeUpdates handlers: 0"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [tb_gateway_service.py] - tb_gateway_service - 139 - Gateway started."
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 234 - MQTT Broker Connector connected to 127.0.0.1:1884 - successfufully."
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [tb_loader.py] - tb_loader - 66 - Import JsonMqttUplinkConverter from /usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt."
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 277 - Connector "MQTT Broker Connector" subscribe to /telemetry/+"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [tb_device_mqtt.py] - tb_device_mqtt - 141 - connection SUCCESS"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 320 - ()"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [mqtt_connector.py] - mqtt_connector - 327 - "MQTT Broker Connector" /telemetry/+, subscription message id = 1"
Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [tb_gateway_mqtt.py] - tb_gateway_mqtt - 176 - Subscribed to *|* with id 2 for device *"

So I think the regular expression in the mqtt.json "deviceNameTopicExpression":"(?<=telemetry\/)(.*?)" does not fit.

I want to setup everything like in mapping example 2 (https://thingsboard.io/docs/iot-gateway/config/mqtt/), cause the different devices have all the same pretopic "telemetry/" in this case and as subtopic they use their mac addresses. I want to use this mac addresses as the device names in Thingsboard. The addresses contain colons, could that be the reason why the regEx is not working as in the example? I am not not very familiar with RegEx, so please forgive me if the question is a bit stupid

Configuration tb_gateway.yaml

  thingsboard:
  host: 127.0.0.1
  port: 1883
  remoteShell: false
  remoteConfiguration: false
  statsSendPeriodInSeconds: 3600
  checkConnectorsConfigurationInSeconds: 60
  security:
    accessToken: XXXXXXXXXXXX
  qos: 1
storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000
#  type: file
#  data_folder_path: ./data/
#  max_file_count: 10
#  max_read_records_count: 10
#  max_records_per_file: 10000
connectors:
  - name: MQTT Broker Connector
    type: mqtt
    configuration: mqtt.json`

MQTT Connector mqtt.json

{
  "broker": {
    "name":"Default Local Broker",
    "host":"127.0.0.1",
    "port":1884,
    "clientId": "ThingsBoard_gateway",
    "security": {
      "type": "basic",
      "username": "usernameusername",
      "password": "XXXXXXXXXXXXXX"
    }
  },
  "mapping": [
    {
      "topicFilter": "/telemetry/+",
      "converter": {
        "type": "json",
        "deviceNameTopicExpression": "(?<=telemetry\/)(.*?)",
        "deviceTypeTopicExpression": "Device",
        "timeout": 60000,
        "attributes": [],
        "timeseries": [
          {
            "type": "double",
            "key": "gw_mac",
            "value": "${gw_mac}"
          },
          {
            "type": "double",
            "key": "rssi",
            "value": "${rssi}"
          },
          {
            "type": "double",
            "key": "aoa",
            "value": "${aoa}"
          },
          {
            "type": "double",
            "key": "gwts",
            "value": "${gwts}"
          },
          {
            "type": "double",
            "key": "devicets",

Info:

Sep 22 11:01:36 ecs-application-server-scobytec python3[259278]: ""2021-09-22 11:01:36" - INFO - [tb_gateway_mqtt.py] - tb_gateway_mqtt - 176 - Subscribed to *|* with id 2 for device *"

Versions:

OS: Ubuntu 20.04.1 LTS
Thingsboard IoT Gateway version 2.7
Python version Python 3.8.10

samson0v commented 2 years ago

Hi @RudolphAschmoneit, you configured your IoT Gateway right, but if you want to it works, you have to send data to the broker from a device. There are examples with your config on my PC:

If you want to see a message about successful converting and sending messages you have to set DEBUG level instead of INFO in config/logs.conf

RudolphAschmoneit commented 2 years ago

Hi @samson0v I am sending messages from a device to the broker. When I use mosquitto_sub I also see this messages. But thank you for your hint with the DEBUG level in logs, I should have tried this early. I will let you know next week if this could fix my problem. Have a nice weekend.

RudolphAschmoneit commented 2 years ago

HI @all. I still don't get it. I tried different RegEX for the device Name Topic Expression:

"deviceNameTopicExpression": "(?<=telemetry\/)(.*?)"
"deviceNameTopicExpression": "(?<=telemetry\/)(..:..:..:..:..:..)"
"deviceNameTopicExpression": "(?<=telemetry\/)(.................)"
"deviceNameTopicExpression": "(?<=telemetry\/)([A-F0-9][A-F0-9]:[A-F0-9][A-F0-9]:[A-F0-9][A-F0-9]:[A-F0-9][A-F0-9]:[A-F0-9][A-F0-9]:[A-F0-9][A-F0-9])"

according to regex101.com this are all Regexes which should fit the pattern telemetry/MAC-ADDRESS

Then I thought maybe it is the value of the deviceTypeTopicExpression, cause my Devices not actual have this parameter in there messages, so I tried it without any value, with out the key-value pair at all, with different regExes, nothing changes.

I set all the log Levels to Debug, but the only thing I can see is in service.log:

""2021-09-30 15:28:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 74 - Gateway starting..."
""2021-09-30 15:28:50" - DEBUG - [tb_updater.py] - tb_updater - 68 - Checking for new version"
""2021-09-30 15:28:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 79 - ThingsBoard IoT gateway version: 2.7"
""2021-09-30 15:28:50" - DEBUG - [tb_logger.py] - tb_logger - 37 - Added remote handler to log service"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 35 - Root path is: /usr/lib/python3/dist-packages/thingsboard_gateway"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 37 - Debian installation extensions folder exists."
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3455483b50>, origin='/var/lib/thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3455508640>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3455438d00>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt/json_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f3455438ee0>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt/mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='MqttConnector', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f345544f040>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt/mqtt_connector.py')"
""2021-09-30 15:28:50" - INFO - [tb_loader.py] - tb_loader - 66 - Import MqttConnector from /usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt."
""2021-09-30 15:28:50" - DEBUG - [tb_gateway_service.py] - tb_gateway_service - 732 - Loaded devices:
 {}"
""2021-09-30 15:28:50" - DEBUG - [tb_gateway_service.py] - tb_gateway_service - 408 - Send data Thread has been started successfully."
""2021-09-30 15:28:50" - INFO - [tb_gateway_service.py] - tb_gateway_service - 139 - Gateway started."
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='JsonMqttUplinkConverter', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f345439f8b0>, origin='/var/lib/thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='JsonMqttUplinkConverter', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f345439f940>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/extensions/mqtt/custom_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - DEBUG - [tb_loader.py] - tb_loader - 57 - ModuleSpec(name='JsonMqttUplinkConverter', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f345439fa30>, origin='/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt/json_mqtt_uplink_converter.py')"
""2021-09-30 15:28:50" - INFO - [tb_loader.py] - tb_loader - 66 - Import JsonMqttUplinkConverter from /usr/lib/python3/dist-packages/thingsboard_gateway/connectors/mqtt."
""2021-09-30 15:28:50" - DEBUG - [tb_gateway_service.py] - tb_gateway_service - 249 - Received data: {}"

So have I to debug the single Python modules on my own, to see why the module doesn't find any devices?

When I stop the gateway and use the same Credentials the Gateway use for a manual mosquitto_sub command on topic telemetry/+, I get messages from all the devices with the topics telemetry/macadress1, telemetry/macadress2, etc.

So If anyone has any idea what I am doing wrong I would be really greatful! Thanks a lot in Advance!

imbeacon commented 2 years ago

Hi @RudolphAschmoneit ,

You have a wrong topic filter in your configuration: "topicFilter": "/telemetry/+" It looks like you publish to the topic telemetry/+ , but these are different topics, please change topic filter to "topicFilter": "telemetry/+" and run the gateway again. Let us know about the results.

RudolphAschmoneit commented 2 years ago

Hi @zbeacon Thank's for your answer. That helped! And sorry for not seeing this little mistake by myself

thingsboard / thingsboard-gateway

[HELP] #578