search

thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.72k stars 829 forks source link

[HELP] Assessment on Thingsboard Gateway capability #399

Closed ysit-thingsboard closed 3 years ago

ysit-thingsboard commented 3 years ago

We are a customer for the Thingsboard PE version. Our plan was to use the PE version feautre, Integration but TB engineers told us in customer portal that our payload with data rate of 1KB per seconds does not fit the Integration feature designed to fit data of 1MB per second. It doesn't any sense at all.

Now, we decide to evaluate on the Thingsboard Gateway because it's open-source so we are able to check on the code if there is any problem. However, we still have some questions on the design side like the question came up for integration, "What kind of payload is Thingsboard Gateway designed to fit?" We came across with an interesting reply from TB and we thought it will be useful to share is to anyone who is interesting in Thingsboard Gateway. So we are discussing this topic on Github.

For TB engineers, here is the original ticket on the customer portal.

The interesting part is how much resources tb-gateway need and how many data it can process with that resource. image

To continue with this topic, Vladyslav @vvlladd28 , please share your configuration and examples of your data in and data out for Thingsboard Gateway.

ysit-thingsboard commented 3 years ago

Please be specific on what the issue that you found. I found a problem while I'm trying the Thingsboard Gateway, so I will report a bug here if there is no sufficient information to determine that the issue is known.

image

imbeacon commented 3 years ago

Hi @ysit-thingsboard ,

Thank you for your interest in ThingsBoard IoT gateway. We have tested the gateway payload on start of summer, and it was some about 2000 messages per second using MQTT connector. I decided to repeat the test on Saturday and I discovered some decreasing of the gateway productivity. The reason of this issue is a long conversion in converters. I work to solve this issue.

I will write here when I will find and push solution for this.

Also please do not add people which you don't know to conversation, because vvlladd28 is not Vlad Ponoiko user.

ysit-thingsboard commented 3 years ago

Sorry about tagging the wrong person.

Then your issue is not the same problem since we are using custom converter. According to your code ( I'm not very experience with Python so please correct me if I'm wrong), Thingsboard Gateway opens a new thread for each connector configured but only has one thread connected to TB to send data. You can configure multiple connectors in one Thingsboard Gateway so each TB Gateway service performance should be limited to this send data thread even you installed the service on an one hundred threads server. Please provide a reasonable hardware suggestion based on your code.

You have a minimum of 0.01 sleep time in the loop of sending data to TB so the service can only send 99 times even if it's running on a super powerful computer. The default max_read_records_count value is only 10 which mean the default configuration can only send a maximum of 990 messages per second so please share your configuration.

We are generating data at the rate of 100 messages per second and each message converts to 11 datapoints. That's 1100 datapoints per second. The statistic data sent to TB has an update of 500 events every 5 seconds. All the statistic numbers are the same so all the events/messages produced on gateway should be sent to TB. However, we have a 30 min delay on the data arrival.

image

our configuration:

storage:
 type: file
 data_folder_path: ./data/
 max_file_count: 1000
 max_read_records_count: 10000
 max_records_per_file: 1000000

The "time" is device time.

The environment is a new CE cluster but installed on the same server and only receiving these data. For most of the time, the CPU resources stay very low for the TB related services, less than 10% for core and kafka and less than 1% for postgres.

The good thing is that the services do not crash and have no significant increase in data file size after running for 2 days compared to the PE Integration.

ysit-thingsboard commented 3 years ago

Hi @zbeacon

Are you aware of that I'm having a different problem from yours?

Should I open another issue to track my own problem?