[HELP] Gateway Transaction Limit

[l33tkrew]

Describe the issue Is there a transaction limit in Thingsboard gateway? Because I am doing a performance and load testing. The gateway mqtt broker get every message (15 mqtt devices that send message in every 1 seconds continously). But the messages cannot be sent my main broker (host/port specified in tb_gateway.yaml). The main broker get messages in every 40+ seconds. (did not calculate the exact duration. I check the messages through MQTT Explorer).

There may be 2 problems; 1-) TB Gateway cannot convert and sent the all messages to main broker 2-) There is a limit in conversion and sending the messages

I think it is because of the limit you put. If it is, how can i increase that limit?

Configuration (Attach your configuration file) tb_gateway Conf

thingsboard:
  host: *********
  port: 1883
  remoteShell: false
  remoteConfiguration: false
  statsSendPeriodInSeconds: 3600
  minPackSendDelayMS: 0
  checkConnectorsConfigurationInSeconds: 60
  handleDeviceRenaming: true
  checkingDeviceActivity:
    checkDeviceInactivity: false
    inactivityTimeoutSeconds: 120
    inactivityCheckPeriodSeconds: 10
  security:
    accessToken: test
  qos: 1
storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000
#  type: file
#  data_folder_path: ./data/
#  max_file_count: 10
#  max_read_records_count: 10
#  max_records_per_file: 10000
#  type: sqlite
#  data_file_path: ./data/data.db
#  messages_ttl_check_in_hours: 1
#  messages_ttl_in_days: 7
grpc:
  enabled: false
  serverPort: 9595
  keepaliveTimeMs: 10000
  keepaliveTimeoutMs: 5000
  keepalivePermitWithoutCalls: true
  maxPingsWithoutData: 0
  minTimeBetweenPingsMs: 10000
  minPingIntervalWithoutDataMs: 5000
connectors:
  -
    name: MQTT Broker Connector
    type: mqtt
    configuration: mqtt.json

#  -
#    name: Modbus Connector
#    type: modbus
#    configuration: modbus.json
#
#  -
#    name: Modbus Connector
#    type: modbus
#    configuration: modbus_serial.json
#
#  -
#    name: OPC-UA Connector
#    type: opcua
#    configuration: opcua.json
#
#  -
#    name: BLE Connector
#    type: ble
#    configuration: ble.json
#
#  -
#    name: REQUEST Connector
#    type: request
#    configuration: request.json
#
#  -
#    name: CAN Connector
#    type: can
#    configuration: can.json
#
#  -
#    name: BACnet Connector
#    type: bacnet
#    configuration: bacnet.json
#
#  -
#    name: ODBC Connector
#    type: odbc
#    configuration: odbc.json
#
#  -
#    name: REST Connector
#    type: rest
#    configuration: rest.json
#
#  -
#    name: SNMP Connector
#    type: snmp
#    configuration: snmp.json
#
#  -
#    name: FTP Connector
#    type: ftp
#    configuration: ftp.json
#
#  -
#    name: Socket TCP/UDP Connector
#    type: socket
#    configuration: socket.json
#
# ========= Customization ==========
#
#
#  -
#    name: Custom Serial Connector
#    type: serial
#    configuration: custom_serial.json
#    class: CustomSerialConnector
#
#  -
#    name: GRPC Connector 1
#    key: auto
#    type: grpc
#    configuration: grpc_connector_1.json

Connector name (If you need help with some connector/converter): MQTT Conf

{
  "broker": {
    "name": "Default Local Broker",
    "host": "**********",
    "port": 1884,
    "clientId": "Gateway",
    "maxMessageNumberPerWorker": 10,
    "maxNumberOfWorkers": 100,
    "security": {
      "type": "basic",
      "username": "*******",
      "password": "********"
    }
  },
  "mapping": [
    {
      "topicFilter": "/sensor/data",
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "${serialNumber}",
        "deviceTypeJsonExpression": "${sensorType}",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          },
          {
            "type": "string",
            "key": "${sensorModel}",
            "value": "on"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          },
          {
            "type": "string",
            "key": "combine",
            "value": "0"
          }
        ]
      }
    },
    {
      "topicFilter": "/sensor/+/data",
      "converter": {
        "type": "json",
        "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/data)",
        "deviceTypeTopicExpression": "Thermometer",
        "timeout": 60000,
        "attributes": [
          {
            "type": "string",
            "key": "model",
            "value": "${sensorModel}"
          }
        ],
        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${temp}"
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${hum}"
          }
        ]
      }
    },
    {
      "topicFilter": "/custom/sensors/+",
      "converter": {
        "type": "custom",
        "extension": "CustomMqttUplinkConverter",
        "extension-config": {
          "temperatureBytes": 2,
          "humidityBytes": 2,
          "batteryLevelBytes": 1
        }
      }
    }
  ],
  "connectRequests": [
    {
      "topicFilter": "sensor/connect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/connect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)"
    }
  ],
  "disconnectRequests": [
    {
      "topicFilter": "sensor/disconnect",
      "deviceNameJsonExpression": "${SerialNumber}"
    },
    {
      "topicFilter": "sensor/+/disconnect",
      "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)"
    }
  ],
  "attributeRequests": [
    {
      "retain": false,
      "topicFilter": "v1/devices/me/attributes/request",
      "deviceNameTopicExpression": "${SerialNumber}",
      "attributeNameJsonExpression": "${sensorModel}"
    }
  ],
  "attributeUpdates": [
    {
      "retain": true,
      "deviceNameFilter": "SmartMeter.*",
      "attributeFilter": "uploadFrequency",
      "topicExpression": "sensor/${deviceName}/${attributeKey}",
      "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
    }
  ],
  "serverSideRpc": [
    {
      "deviceNameFilter": ".*",
      "methodFilter": "echo",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
      "responseTimeout": 10000,
      "valueExpression": "${params}"
    },
    {
      "deviceNameFilter": ".*",
      "methodFilter": "no-reply",
      "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
      "valueExpression": "${params}"
    }
  ]
}

Versions (please complete the following information):

• OS: Ubuntu 20.04
• Thingsboard IoT Gateway version: latest github version
• Python version: 3.8

[samson0v]

Hi @l33tkrew, you can try to increase the minPackSendDelayMS parameter if you send a lot of data to ThingsBoard. Thanks for your interest in ThingsBoard IoT Gateway!

[l33tkrew]

Hi @samson0v First, thank you for your answer.

What is "minPackSendDelayMS"? What does it do? I want to learn its purpose before setting the value.

Also, is there a max value limit for it? Because I want to push the limits. I appreciate it if you recommend a value for it.

[samson0v]

@l33tkrew, it the delay of sending every packet to the ThingsBoard. For example:

1. First situation You send 100 packets per second and you set minPackSendDelayMS: 0, so it is 100 requests to ThingsBoard every second.
2. Second Situation You send 100 packets per second and you set minPackSendDelayMS: 1000, so it is 1 request to ThingsBoard every second (but the size of packet is bigger).