The Gateway cannot push data to the ThingsBoard

[RenCisco]

Describe the bug **See thingsBoard reply ：URL：(https://github.com/thingsboard/thingsboard/issues/5606)

information description

1. Tb-gateway obtains data from device 1 over Modbus protocol. Device 1 has more than 3000 fields. The configuration file is as follows { "server": { "type": "tcp", "host": "device ip address", "port": 502, "timeout": 35, "method": "socket", "byteOrder": "BIG", "devices": [ { "unitId": 1, "deviceName": "deviceName1", "attributesPollPeriod": 5000, "timeseriesPollPeriod": 5000, "sendDataOnlyOnChange": true, "timeseries": [ {"tag":"A","type": "bits","functionCode": 2,"objectsCount": 1,"byteOrder": "LITTLE","address":0}, {"tag":"B","type": "bits","functionCode": 2,"objectsCount": 1,"byteOrder": "LITTLE","address":1}, ### More than 3,500 lines are omitted {"tag":"C","type": "32float","functionCode": 4,"objectsCount": 2,"wordOrder": "LITTLE","address":1482}, {"tag":"D","type": "32float","functionCode": 4,"objectsCount": 2,"wordOrder": "LITTLE","address":1498}, {"tag":"E","type": "32float","functionCode": 4,"objectsCount": 2,"wordOrder": "LITTLE","address":4594} ] } ] } }

2. tb_gataway.yaml file configuration is as follows. thingsboard: host: thingsboard ip address port: 1883 remoteShell: false remoteConfiguration: false security: accessToken: PUT_YOUR_ACCESS_TOKEN_HERE qos: 1 storage: type: memory read_records_count: 1000 max_records_count: 100000

   type: file

       #  data_folder_path: ./data/
       #  max_file_count: 10
       #  max_read_records_count: 10
       #  max_records_per_file: 10000
       connectors:
       #  -
       #    name: MQTT Broker Connector
       #    type: mqtt
       #    configuration: mqtt.json
         -
           name: ModbusDaxinginput
           type: modbus
           configuration: modbusDxiput.json
       #  -
       ....Is omitted
       #  -
       #    name: Custom Serial Connector
       #    type: serial
       #    configuration: custom_serial.json
       #    class: CustomSerialConnector

3. By capturing packets using tcpdump, the gateway can obtain all field values of the device using Modbus.

issue description

1. When the Modbus profile is only 20 lines long, there is no problem getting the data and displaying it in real time .
2. When a Modbus configuration file is several thousand lines long, ThingsBoard can't display all the fields and keep the data up to date.

3. How can i solve this issue? Thingsboard doesn't display data and doesn't update data in a timely manner when modbus configuration files are thousands of lines long.
4. Is thingsBoard having trouble communicating with the Gateway?
5. How many lines can a Modbus profile support?
   Your Server Environment

• own setup
  • docker deployment
  • ThingsBoard Community Edit latest Version
  • OS centos7
  • DB postgresql

Your Client Environment

Desktop (please complete the following information):

• OS: [win10]
• Browser [chrome,]
• Version [e.g. 22]

Your Device

• Connectivity
  • Gateway -modbus
  • Integration: (Specify name)
• Device vendor and model

Versions (please complete the following information):

• OS: [e.g. centos7]
• Thingsboard IoT Gateway version [e.g. 2.0]
• Python version[e.g. 3.7]

[samson0v]

Hi @RenCisco, we reworked the Modbus connector and in the coming days, we will update Gateway. So wait for a little and try to use the new Modbus connector. I will let you know about the update!

Thanks for your interest in ThingsBoard IoT Gateway!

[samson0v]

Hi @RenCisco, we update Gateway Modbus Connector, so you can try and test it (use version from master branch)!

[RenCisco]

@samson0v Thanks for your help！ I tried to install the master branch,but got an error. I installed it follow url (https://thingsboard.io/docs/iot-gateway/install/source-installation/) So please tell me how to install step by step. this is errors: [xx@yy]#python3 ./thingsboard_gateway/tb_gateway.py ""2021-12-02 16:16:53" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 72 - Gateway starting..." ""2021-12-02 16:16:53" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 77 - ThingsBoard IoT gateway version: 2.6" ""2021-12-02 16:16:53" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 176 - Subscribed to | with id 1" ""2021-12-02 16:16:53" - |INFO| - [tb_utility.py] - tb_utility - check_and_import - 111 - Import MqttConnector from /usr/local/lib/python3.6/site-packages/thingsboard_gateway/connectors/mqtt." ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted mapping handlers: 3" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected mapping handlers: 0" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted serverSideRpc handlers: 2" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected serverSideRpc handlers: 0" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted connectRequests handlers: 2" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected connectRequests handlers: 0" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted disconnectRequests handlers: 2" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected disconnectRequests handlers: 0" ""2021-12-02 16:16:53" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 143 - Mandatory key 'topicExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}" ""2021-12-02 16:16:53" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 143 - Mandatory key 'valueExpression' missing from attributeRequests handler: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}" ""2021-12-02 16:16:53" - |ERROR| - [mqtt_connector.py] - mqtt_connector - load_handlers - 150 - attributeRequests handler is missing some mandatory keys => rejected: {"retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}"}" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted attributeRequests handlers: 0" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected attributeRequests handlers: 1" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 158 - Number of accepted attributeUpdates handlers: 1" ""2021-12-02 16:16:53" - |INFO| - [mqtt_connector.py] - mqtt_connector - load_handlers - 162 - Number of rejected attributeUpdates handlers: 0" ""2021-12-02 16:16:53" - |ERROR| - [mqtt_connector.py] - mqtt_connector - connect - 196 - [Errno 111] Connection refused" ""2021-12-02 16:16:53" - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init__ - 135 - Gateway started." ""2021-12-02 16:16:54" - |ERROR| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 148 - connection FAIL with error 5 not authorised" ""2021-12-02 16:16:54" - |INFO| - [tb_updater.py] - tb_updater - check_for_new_version - 78 - 2.7" ""2021-12-02 16:16:54" - |INFO| - [tb_updater.py] - tb_updater - check_for_new_version - 81 -

[===UPDATE===]

New version 2.7 is available!

[===UPDATE===] " ""2021-12-02 16:16:55" - |ERROR| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 148 - connection FAIL with error 5 not authorised" ""2021-12-02 16:17:03" - |ERROR| - [mqtt_connector.py] - mqtt_connector - connect - 196 - [Errno 111] Connection refused" ""2021-12-02 16:17:06" - |ERROR| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 148 - connection FAIL with error 5 not authorised" ""2021-12-02 16:17:13" - |ERROR| - [mqtt_connector.py] - mqtt_connector - connect - 196 - [Errno 111] Connection refused" ""2021-12-02 16:17:16" - |ERROR| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 148 - connection FAIL with error 5 not authorised" ""2021-12-02 16:17:23" - |ERROR| - [mqtt_connector.py] - mqtt_connector - connect - 196 - [Errno 111] Connection refused" ""2021-12-02 16:17:27" - |ERROR| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 148 - connection FAIL with error 5 not authorised" ""2021-12-02 16:17:33" - |ERROR| - [mqtt_connector.py] - mqtt_connector - connect - 196 - [Errno 111] Connection refused"

[samson0v]

Hi @RenCisco, follow the next steps:

1. Clone the master branch
2. Create virtual env using python3 -m venv venv
3. Activate virtual env source venv/bin/activate
4. Run pip install setup.py
5. Configure Gateway
6. Run Gateway

[RenCisco]

@samson0v My OS is centos7, why create a virtual enviroment?

this is my operation

[root@centos7 thingsboard-gateway]# python3 ./thingsboard_gateway/tb_gateway.py [STREAM ONLY] 2021-12-03 09:53:18,355 - DEBUG - [tb_device_mqtt.py] - tb_device_mqtt - 125 - Disconnected client: <paho.mqtt.client.Client object at 0x7fa793af49e8>, user data: None, result code: 1

[RenCisco]

@samson0v

[root@centos7 thingsboard-gateway]# pip install setup.py ERROR: Could not find a version that satisfies the requirement setup.py (from versions: none) ERROR: No matching distribution found for setup.py WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available. You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

[RenCisco]

@samson0v

[root@centos7 thingsboard-gateway]# pip install setup.py ERROR: Could not find a version that satisfies the requirement setup.py (from versions: none) ERROR: No matching distribution found for setup.py WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available. You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

[imbeacon]

Hi @RenCisco ,

Could you try the following command: python3 setup.py install

[RenCisco]

@zbeacon thanks for your help I executed command (python3 setup.py install) but how can i run thingsboard-gateway?

[RenCisco]

[root@centos7 thingsboard-gateway]# systemctl start thingsboard-gateway Failed to start thingsboard-gateway.service: Unit not found.

[RenCisco]

@zbeacon @samson0v Thanks for your help!

this is error [root@centos7 thingsboard-gateway]# python3 ./thingsboard_gateway/tb_gateway.py [STREAM ONLY] 2021-12-06 10:56:59,249 - DEBUG - [tb_device_mqtt.py] - tb_device_mqtt - 125 - Disconnected client: <paho.mqtt.client.Client object at 0x7fcdf8283f60>, user data: None, result code: 1

[RenCisco]

@samson0v @zbeacon Hi thanks for your help! After I grab packets using tcpdump, I find that the Gateway pushes MQTT data to ThingsBoard every 10 minutes. How can I optimize?

[samson0v]

Hi @RenCisco, after installing Gateway from the source, you can start it using python3 thingsboard_gateway/tb_gateway.py. And don't forget to configure the Gateway and connectors which you will use.

[RenCisco]

@samson0v Thanks for your help ! I've started thingsboard-gateway, but sending MQTT data to Thingsboard is too slow, every 10 minutes. So how do I solve this problem? How do I get performance up to pushing data to thingsBoard every few seconds
this is my configuration tb_gateway.yaml thingsboard: host: ip port: 9909 remoteShell: false remoteConfiguration: false security: accessToken: 1YGsdfserfsQC4asdfmneF qos: 1 storage: type: memory read_records_count: 100 max_records_count: 100000

modbus.json "server": { "type": "tcp", "host": "ip", "port": 589, "timeout": 35, "method": "socket", "byteOrder": "BIG", "devices": [ { "unitId": 1, "deviceName": "daxinginput", "attributesPollPeriod": 5000, "timeseriesPollPeriod": 5000, "sendDataOnlyOnChange": true, "timeseries": [

[samson0v]

Hi @RenCisco, you are using "sendDataOnlyOnChange": true, maybe the data doesn't change?

[RenCisco]

@samson0v Thanks for your help!

i have used "sendDataOnlyOnChange": false ,But problems remain. The ThingsBoard still updates its data every 10 minutes.

[samson0v]

Hi @RenCisco, Do you see data streams using Wireshark when Gateway send data to ThingsBoard? How many devices do you use? Probably you can try to use "minPackSendDelayMS": <for example 200> in main tb_gataway.yaml config file as on the picture below.

[samson0v]

Hi @RenCisco , did it help, can we close the issue?

[RenCisco]

@samson0v Thanks a lot for your help and support!