search

thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.74k stars 844 forks source link

[HELP] tb-gateway docker container - rest connector error #762

Closed l33tkrew closed 2 years ago

l33tkrew commented 2 years ago

Describe the issue I can connect my mqtt broker container with tb-gateway container without any problem. But, when i enable the rest connector, the tb-gateway container restart itself continously. I attached the error logs and my rest configuration file.

Configuration (Attach your configuration file) tbgateway.conf

thingsboard:
  host: xxxxxxxxxxxxxxx
  port: 1883
  remoteShell: false
  remoteConfiguration: false
  statsSendPeriodInSeconds: 3600
  minPackSendDelayMS: 0
  checkConnectorsConfigurationInSeconds: 60
  security:
    accessToken: xxxxxxxxxxxxxxxxx
  qos: 1
storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000
#  type: file
#  data_folder_path: ./data/
#  max_file_count: 10
#  max_read_records_count: 10
#  max_records_per_file: 10000
#  type: sqlite
#  data_file_path: ./data/data.db
#  messages_ttl_check_in_hours: 1
#  messages_ttl_in_days: 7
grpc:
  enabled: false
  serverPort: 9595
  keepaliveTimeMs: 10000
  keepaliveTimeoutMs: 5000
  keepalivePermitWithoutCalls: true
  maxPingsWithoutData: 0
  minTimeBetweenPingsMs: 10000
  minPingIntervalWithoutDataMs: 5000
connectors:
  -
    name: MQTT Broker Connector
    type: mqtt
    configuration: mqtt.json

#  -
#    name: Modbus Connector
#    type: modbus
#    configuration: modbus.json
#
#  -
#    name: Modbus Connector
#    type: modbus
#    configuration: modbus_serial.json
#
#  -
#    name: OPC-UA Connector
#    type: opcua
#    configuration: opcua.json
#
#  -
#    name: BLE Connector
#    type: ble
#    configuration: ble.json
#
#  -
#    name: REQUEST Connector
#    type: request
#    configuration: request.json
#
#  -
#    name: CAN Connector
#    type: can
#    configuration: can.json
#
#  -
#    name: BACnet Connector
#    type: bacnet
#    configuration: bacnet.json
#
#  -
#    name: ODBC Connector
#    type: odbc
#    configuration: odbc.json
#
   -
    name: REST Connector
    type: rest
    configuration: rest.json

#  -
#    name: SNMP Connector
#    type: snmp
#    configuration: snmp.json
#
#  -
#    name: FTP Connector
#    type: ftp
#    configuration: ftp.json
#
#
# ========= Customization ==========
#
#
#  -
#    name: Custom Serial Connector
#    type: serial
#    configuration: custom_serial.json
#    class: CustomSerialConnector
#
#  -
#    name: GRPC Connector 1
#    key: auto
#    type: grpc
#    configuration: grpc_connector_1.json

Rest connector configuration. The port 5000:5000 is open for tb-gateway container. The host address is container's own address. I also tried 127.0.0.1. It also causes the same error.

{
  "host": "172.18.0.41",
  "port": "5000",
  "SSL": false,
  "security": {
    "cert": "~/ssl/cert.pem",
    "key": "~/ssl/key.pem"
  },
  "mapping": [
    {
      "endpoint": "/device1",
      "HTTPMethods": [
        "POST"
      ],
      "security": {
        "type": "basic",
        "username": "user",
        "password": "passwd"
      },
      "converter": {
        "type": "json",
        "deviceNameExpression": "Device ${name}",
        "deviceTypeExpression": "default",
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "${sensorModel}",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          },
          {
            "type": "string",
            "key": "combine",
            "value": "${hum}:${temp}"
          }
        ]
      }
    },
    {
      "endpoint": "/anon1",
      "HTTPMethods": [
        "GET",
        "POST"
      ],
      "security": {
        "type": "anonymous"
      },
      "converter": {
        "type": "json",
        "deviceNameExpression": "Device 2",
        "deviceTypeExpression": "default",
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "Model2"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          }
        ]
      }
    },
    {
      "endpoint": "/anon2",
      "HTTPMethods": [
        "POST"
      ],
      "security": {
        "type": "anonymous"
      },
      "converter": {
        "type": "custom",
        "deviceNameExpression": "SuperAnonDevice",
        "deviceTypeExpression": "default",
        "extension": "CustomRestUplinkConverter",
        "extension-config": [
          {
            "key": "Totaliser",
            "datatype": "float",
            "fromByte": 0,
            "toByte": 4,
            "byteorder": "big",
            "signed": true,
            "multiplier": 1
          }
        ]
      }
    }
  ],
  "attributeUpdates": [
    {
      "HTTPMethod": "POST",
      "SSLVerify": false,
      "httpHeaders": {
        "CONTENT-TYPE": "application/json"
      },
      "security": {
        "type": "basic",
        "username": "user",
        "password": "passwd"
      },
      "timeout": 0.5,
      "tries": 3,
      "allowRedirects": true,
      "deviceNameFilter": ".*REST$",
      "attributeFilter": "data",
      "requestUrlExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ],
  "serverSideRpc": [
    {
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestUrlExpression": "http://127.0.0.1:5001/${deviceName}",
      "responseTimeout": 1,
      "HTTPMethod": "GET",
      "valueExpression": "${params}",
      "timeout": 10.0,
      "tries": 3,
      "httpHeaders": {
        "Content-Type": "application/json"
      },
      "security": {
        "type": "anonymous"
      }
    },
    {
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestUrlExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "HTTPMethod": "POST",
      "valueExpression": "${params}",
      "httpHeaders": {
        "Content-Type": "application/json"
      },
      "security": {
        "type": "anonymous"
      }
    }
  ]
}

Connector name (If you need help with some connector/converter): [REST Connector]

Error traceback (If it was raised):

Traceback (most recent call last):
  File "//./thingsboard_gateway/tb_gateway.py", line 31, in <module>
    main()
  File "//./thingsboard_gateway/tb_gateway.py", line 23, in main
    TBGatewayService(path.dirname(path.abspath(__file__)) + '/config/tb_gateway.yaml'.replace('/', path.sep))
  File "/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/gateway/tb_gateway_service.py", line 84, in __init__
    self.__config = safe_load(general_config)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 392, in parse_block_sequence_entry
    raise ParserError("while parsing a block collection", self.marks[-1],
yaml.parser.ParserError: while parsing a block collection
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 35, column 3
expected <block end>, but found '<block sequence start>'
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 80, column 4
Traceback (most recent call last):
  File "//./thingsboard_gateway/tb_gateway.py", line 31, in <module>
    main()
  File "//./thingsboard_gateway/tb_gateway.py", line 23, in main
    TBGatewayService(path.dirname(path.abspath(__file__)) + '/config/tb_gateway.yaml'.replace('/', path.sep))
  File "/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/gateway/tb_gateway_service.py", line 84, in __init__
    self.__config = safe_load(general_config)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 392, in parse_block_sequence_entry
    raise ParserError("while parsing a block collection", self.marks[-1],
yaml.parser.ParserError: while parsing a block collection
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 35, column 3
expected <block end>, but found '<block sequence start>'
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 80, column 4
Traceback (most recent call last):
  File "//./thingsboard_gateway/tb_gateway.py", line 31, in <module>
    main()
  File "//./thingsboard_gateway/tb_gateway.py", line 23, in main
    TBGatewayService(path.dirname(path.abspath(__file__)) + '/config/tb_gateway.yaml'.replace('/', path.sep))
  File "/usr/local/lib/python3.9/site-packages/thingsboard_gateway-3.0.1-py3.9.egg/thingsboard_gateway/gateway/tb_gateway_service.py", line 84, in __init__
    self.__config = safe_load(general_config)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.9/site-packages/PyYAML-6.0-py3.9-linux-x86_64.egg/yaml/parser.py", line 392, in parse_block_sequence_entry
    raise ParserError("while parsing a block collection", self.marks[-1],
yaml.parser.ParserError: while parsing a block collection
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 35, column 3
expected <block end>, but found '<block sequence start>'
  in "//thingsboard_gateway/config/tb_gateway.yaml", line 80, column 4

Terminal will be reused by tasks, press any key to close it.

Versions (please complete the following information):

samson0v commented 2 years ago

Hi @l33tkrew, please, send the command that you used to run the container or docker-compose file.

l33tkrew commented 2 years ago

Part of my docker-compose file

  tb-gateway:
    restart: always
    image: "thingsboard/tb-gateway"
    container_name: gateway
    ports:
      - "5000:5000"
    volumes:
      - ./gateway/config:/thingsboard_gateway/config
      - ./gateway/logs:/thingsboard_gateway/logs
      - ./gateway/extensions:/thingsboard_gateway/extensions

I run docker-compose up -d to create non stop containers (restart: always).

I manage the containers in VSCode Docker extension. When I activate the REST connector, and restart container to apply changes. The container restarts itself again and again. Because the container gives error logs after appying the changes. When I disabled the rest connector in tb_gateway.yaml file, the container starts working.

l33tkrew commented 2 years ago

Extra question:

When modifying the REST.connector host address. Should I paste the 127.0.0.1 or the container's own address?

samson0v commented 2 years ago

@l33tkrew try to set host: 0.0.0.0 and expose the port in docker-compose file.

l33tkrew commented 2 years ago

I will set REST.connector's host address to 0.0.0.0 then I will expose 5000:5000 in docker-compose file. Is that correct?

samson0v commented 2 years ago

@l33tkrew yes