[HELP] ThingsBoard stops getting data

[tikarei]

Describe the issue ThingsBoard stops getting data from gateway after running for a couple of days without logging or showing any errors. No errors in the gateway's connector.log, service.log, or tb_connection.log. Also, thingsboard.log is empty. No logging messages other than stating that the connection to the OPCUA server was successful. RemoteDebugLogging is in DEBUG. I know the data from the OPCUA server is still streaming when using another OPCUA client to check.

Both ThingsBoard and gateway are running in their own docker containers. This happened before a few days ago, I restarted the gateway container multiple times but that doesn't solve the issue. Only when I restarted the ThingsBoard container did it start working again. This morning, I can see the issue has happened again. I have tried restarting the gateway to no success. I'm fairly sure it will work once I restart the ThingsBoard container but I want to figure out why this keeps happening. When attached to the ThingsBoard container, here's the example console output I get:

2020-09-11 08:48:27.930  INFO 78 --- [TB-Scheduling-2] o.t.s.service.queue.TbCoreConsumerStats  : Total [42] sessionEvents [0] getAttr [0] subToAttr [0] subToRpc [0] toDevRpc [0] subInfo [42] claimDevice [0] deviceState [0] subMgr [0] coreNfs [0]

2020-09-11 08:48:41.163  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [17] totalSaved [17] totalFailed [0]

2020-09-11 08:49:11.163  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [24] totalSaved [24] totalFailed [0]

2020-09-11 08:49:27.930  INFO 78 --- [TB-Scheduling-2] o.t.s.service.queue.TbCoreConsumerStats  : Total [41] sessionEvents [4] getAttr [0] subToAttr [0] subToRpc [0] toDevRpc [0] subInfo [37] claimDevice [0] deviceState [0] subMgr [0] coreNfs [0]

2020-09-11 08:49:41.163  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [20] totalSaved [20] totalFailed [0]

2020-09-11 08:50:11.163  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [22] totalSaved [22] totalFailed [0]

2020-09-11 08:50:27.930  INFO 78 --- [TB-Scheduling-1] o.t.s.service.queue.TbCoreConsumerStats  : Total [42] sessionEvents [0] getAttr [0] subToAttr [0] subToRpc [0] toDevRpc [0] subInfo [42] claimDevice [0] deviceState [0] subMgr [0] coreNfs [0]

2020-09-11 08:50:41.163  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [20] totalSaved [20] totalFailed [0]

2020-09-11 08:50:51.168  INFO 78 --- [-log-2-thread-1] o.t.server.dao.sql.TbSqlBlockingQueue    : [Attributes] queueSize [0] totalAdded [2] totalSaved [2] totalFailed [0]

Any help to troubleshoot this is much appreciated.

Configuration (Attach your configuration file)

thingsboard:
  host: tbce_1
  port: 1883
  remoteConfiguration: true
  security:
    accessToken: token
storage:
  type: file
  data_folder_path: ./data/
  max_file_count: 10
  max_read_records_count: 1000
  max_records_per_file: 100000
connectors:
  -
    name: OPC-UA Connector
    type: opcua
    configuration: opcua.json

Connector name: OPCUA

Error traceback (If it was raised): None

Versions (please complete the following information):

• OS: docker host is Ubuntu 18.04
• Thingsboard IoT Gateway version: latest docker image 2.4.1
• Python version: 3.7, whichever comes with the docker image

[imbeacon]

Hi @Tikarei ,

One of the possible reasons is the same like in the issue #386 . To check this, please check values of the gateway device telemetry key LOG on ThingsBoard. Probably some errors will be there. Please let me know about the results.

[tikarei]

Do you mean this? The last error was a few days ago.

[tikarei]

@zbeacon Any other ideas on how I can debug the issue?

[tikarei]

@zbeacon It has happened again and I have just realized it's not just the TB gateway. The TB server itself stops receiving attribute and telemetry data. I connected a device through MQTT without TB gateway, the device details shows it is active and connected but no other data is received. Is there a way to move this issue to the main TB repo issues section or should I close this one can recreate it there?

[imbeacon]

Hi @Tikarei ,

I apologize for the long-term answer, it would be better to create another issue in the general ThingsBoard repository.