search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.65k stars 3.26k forks source link

Doris 0.14.7 version, there is a backlog of stream load tasks #5635

Closed hf200012 closed 3 years ago

hf200012 commented 3 years ago

This problem did not occur in version 0.13.15 The specific phenomenon is as follows ： Stream load task error, report: failed to call frontend service The number of txns quickly reached the upper limit：errCode = 2, detailMessage = current running txns on db 11001 is 150, larger than limit 150

View backends heartbeat loss https://github.com/hf200012/images/blob/master/doris/20210412145426.png

Restart BE, the log shows that the heartbeat of FE has not been received

To check transactions, it shows that many tasks are not over ： https://github.com/hf200012/images/blob/master/doris/20210412145444.png

Finally, only restart FE, and then return to normal

Fe master keeps refreshing the following logs ： 2021-04-12 11:45:43,053 WARN (thrift-server-pool-297|673) [ReportHandler.putToQueue():182] the report queue size exceeds the limit: 120. current: 121 2021-04-12 11:45:43,060 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD 2021-04-12 11:45:43,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902] 2021-04-12 11:45:43,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip 2021-04-12 11:45:43,119 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605972] is expired, remove it from transaction manager 2021-04-12 11:45:43,165 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605973] is expired, remove it from transaction manager 2021-04-12 11:45:43,211 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605974] is expired, remove it from transaction manager 2021-04-12 11:45:43,257 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605979] is expired, remove it from transaction manager 2021-04-12 11:45:43,302 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605981] is expired, remove it from transaction manager 2021-04-12 11:45:43,348 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605984] is expired, remove it from transaction manager 2021-04-12 11:45:43,400 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605985] is expired, remove it from transaction manager 2021-04-12 11:45:43,451 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17605986] is expired, remove it from transaction manager 2021-04-12 11:45:43,502 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606000] is expired, remove it from transaction manager 2021-04-12 11:45:43,554 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606001] is expired, remove it from transaction manager 2021-04-12 11:45:43,605 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606007] is expired, remove it from transaction manager 2021-04-12 11:45:43,663 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606008] is expired, remove it from transaction manager 2021-04-12 11:45:43,720 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606009] is expired, remove it from transaction manager 2021-04-12 11:45:43,771 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606017] is expired, remove it from transaction manager 2021-04-12 11:45:43,823 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606018] is expired, remove it from transaction manager 2021-04-12 11:45:43,874 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606031] is expired, remove it from transaction manager 2021-04-12 11:45:43,926 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606033] is expired, remove it from transaction manager 2021-04-12 11:45:43,977 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606034] is expired, remove it from transaction manager 2021-04-12 11:45:44,029 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606040] is expired, remove it from transaction manager 2021-04-12 11:45:44,061 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD 2021-04-12 11:45:44,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902] 2021-04-12 11:45:44,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip 2021-04-12 11:45:44,080 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606041] is expired, remove it from transaction manager 2021-04-12 11:45:44,132 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606045] is expired, remove it from transaction manager 2021-04-12 11:45:44,183 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606046] is expired, remove it from transaction manager 2021-04-12 11:45:44,235 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606047] is expired, remove it from transaction manager 2021-04-12 11:45:44,286 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606074] is expired, remove it from transaction manager 2021-04-12 11:45:44,338 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606075] is expired, remove it from transaction manager 2021-04-12 11:45:44,389 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606081] is expired, remove it from transaction manager 2021-04-12 11:45:44,441 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606082] is expired, remove it from transaction manager 2021-04-12 11:45:44,492 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606083] is expired, remove it from transaction manager 2021-04-12 11:45:44,544 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606089] is expired, remove it from transaction manager 2021-04-12 11:45:44,595 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606092] is expired, remove it from transaction manager 2021-04-12 11:45:44,647 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606107] is expired, remove it from transaction manager 2021-04-12 11:45:44,698 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606109] is expired, remove it from transaction manager 2021-04-12 11:45:44,750 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606111] is expired, remove it from transaction manager 2021-04-12 11:45:44,801 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606119] is expired, remove it from transaction manager 2021-04-12 11:45:44,853 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606120] is expired, remove it from transaction manager 2021-04-12 11:45:44,904 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606124] is expired, remove it from transaction manager 2021-04-12 11:45:44,956 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606125] is expired, remove it from transaction manager 2021-04-12 11:45:45,007 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606126] is expired, remove it from transaction manager 2021-04-12 11:45:45,022 WARN (thrift-server-pool-852|14940) [ReportHandler.putToQueue():182] the report queue size exceeds the limit: 120. current: 121 2021-04-12 11:45:45,062 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():105] get number of low load paths: 10, with medium: HDD 2021-04-12 11:45:45,063 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():186] select alternative tablets for cluster: default_cluster, medium: HDD, num: 2, detail: [5092902, 5092902] 2021-04-12 11:45:45,063 INFO (tablet scheduler|39) [BeLoadRebalancer.selectAlternativeTabletsForCluster():84] cluster is balance: default_cluster with medium: SSD. skip 2021-04-12 11:45:45,066 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606135] is expired, remove it from transaction manager 2021-04-12 11:45:45,123 INFO (txnCleaner|69) [DatabaseTransactionMgr.removeExpiredTxns():1101] transaction [17606136] is expired, remove it from transaction manager

hf200012 commented 3 years ago

This is caused by my stream load task

lordk911 commented 2 years ago

so how to limit stream load task to avoid exceed the FE limit ?