[INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 149, maxActive 100, creating 0, createErrorCount 3050[Bug]

[civic123321]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Version

1.2.4.1

What's Wrong?

I have create a jdbc catalog to connect to a postgresql database;Firstly , the query for catalog database is ok;After some days，the query occur below problem，the detail message : UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 149, maxActive 100, creating 0, createErrorCount 3050 the create message of the catalog is below: CREATE CATALOG catalog_xxxx PROPERTIES ( "password" = "XXX", "driver_class" = "org.postgresql.Driver", "only_specified_database" = "false", "checksum" = "xxxxx", "driver_url" = "file:///opt/postgresql-42.2.9.jar", "lower_case_table_names" = "false", "type" = "jdbc", "user" = "xxxx", "jdbc_url" = "xxxxx" );

What You Expected?

why this happened? how to resolve this problem? In my program, I use jdbc dataSource(database connection pool) to connect doirs,doris uses catalog to connect postgresql database; I want to know how catalog runs? catalog connect to external database through jdbc or jdbc database connection pool?

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[accont]

连接mysql外部表，1.2.6版本，提示的信息一样

[LongWangXX]

errCode = 2, detailMessage = (192.168.8.39)[INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 11923, active 1, maxActive 100, creating 1, createElapseMillis 148427

[LongWangXX]

我也遇到了这个问题 ，1.2.6 和 1.2.6的上一个版本，当doris运行了几天之后，查询mysql外部表就需要很长时间，感觉整个集群不工作；

[LongWangXX]

be.gc.log.20230823-144134 在做fullgc 504150.803: [Full GC (Ergonomics) 716489K->716489K(724480K), 0.9853802 secs] 504151.789: [Full GC (Ergonomics) 716489K->716489K(724480K), 0.9664271 secs] 504152.756: [Full GC (Ergonomics) 716494K->716491K(724480K), 0.9688096 secs] 504153.725: [Full GC (Ergonomics) 716496K->716490K(724480K), 0.9946603 secs] 504154.729: [Full GC (Ergonomics) 716490K->716490K(724480K), 0.9552805 secs] 504155.685: [Full GC (Ergonomics) 716490K->716490K(724480K), 0.9844935 secs] 504156.670: [Full GC (Ergonomics) 716494K->716492K(724480K), 0.9875643 secs] 504157.663: [Full GC (Ergonomics) 716499K->716492K(724480K), 0.9714049 secs] 504158.635: [Full GC (Ergonomics) 716498K->716493K(724480K), 1.0079401 secs] 504159.643: [Full GC (Ergonomics) 716493K->716493K(724480K), 1.0016210 secs] 504160.645: [Full GC (Ergonomics) 716493K->716493K(724480K), 1.0056263 secs] 504161.652: [Full GC (Ergonomics) 716493K->716478K(724480K), 1.0045419 secs] 504162.657: [Full GC (Ergonomics) 716483K->716479K(724480K), 1.5848425 secs] 504164.243: [Full GC (Ergonomics) 716482K->716478K(724480K), 0.9883516 secs] 504165.232: [Full GC (Ergonomics) 716479K->716474K(724480K), 1.5279067 secs] 504166.764: [Full GC (Ergonomics) 716478K->716471K(724480K), 1.6416849 secs] 504168.406: [Full GC (Ergonomics) 716476K->716472K(724480K), 1.0005823 secs] 504169.412: [Full GC (Ergonomics) 716476K->716473K(724480K), 1.0454194 secs] 504170.458: [Full GC (Ergonomics) 716476K->716473K(724480K), 0.9684831 secs] 504171.427: [Full GC (Ergonomics) 716476K->716472K(724480K), 1.0270565 secs] 504172.455: [Full GC (Ergonomics) 716475K->716471K(724480K), 0.9739450 secs] 504173.429: [Full GC (Ergonomics) 716475K->716471K(724480K), 0.9695286 secs] 504174.399: [Full GC (Ergonomics) 716474K->716471K(724480K), 1.0014711 secs] 504175.401: [Full GC (Ergonomics) 716474K->716472K(724480K), 1.0260675 secs] 504176.428: [Full GC (Ergonomics) 716474K->716472K(724480K), 0.9626161 secs] 504177.391: [Full GC (Ergonomics) 716474K->716473K(724480K), 0.9814544 secs] 504178.373: [Full GC (Ergonomics) 716474K->716473K(724480K), 1.0153033 secs] 504179.389: [Full GC (Ergonomics) 716474K->716473K(724480K), 1.0111815 secs] 504180.401: [Full GC (Ergonomics) 716474K->716474K(724480K), 1.0034611 secs] 504181.405: [Full GC (Ergonomics) 716474K->716474K(724480K), 0.9885663 secs] 504182.394: [Full GC (Ergonomics) 716474K->716474K(724480K), 1.0108041 secs] 504183.405: [Full GC (Ergonomics) 716474K->716474K(724480K), 1.0314112 secs] 504184.437: [Full GC (Ergonomics) 716474K->716474K(724480K), 0.9817878 secs] 504185.419: [Full GC (Ergonomics) 716474K->716473K(724480K), 1.0284817 secs] 504186.448: [Full GC (Ergonomics) 716474K->716474K(724480K), 1.0138165 secs] 504187.463: [Full GC (Ergonomics) 716474K->716473K(724480K), 1.0152616 secs] 504188.478: [Full GC (Ergonomics) 716474K->716473K(724480K), 1.0675273 secs] 504189.547: [Full GC (Ergonomics) 716474K->716474K(724480K), 0.9797167 secs] 504190.527: [Full GC (Ergonomics)

[erichou2010]

连接SQL SERVER外表，版本：1.2.7.1，也遇到相关错误： [INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 100, creating 1, createElapseMillis 10000 2023-11-27 07:58:31,686 WARN (thrift-server-pool-8|414) [Coordinator.updateStatus():875] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 8024bfd2fd204458-a0d2cd276775c07a, instance id: 8024bfd2fd204458-a0d2cd276775c07b, error message: (192.168.1.20)[INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 100, creating 1, createElapseMillis 10000 2023-11-27 07:58:31,686 WARN (mysql-nio-pool-701|52302) [StmtExecutor.handleInsertStmt():1495] insert failed: (192.168.1.20)[INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 100, creating 1, createElapseMillis 10000 2023-11-27 07:58:31,686 INFO (thrift-server-pool-8|414) [Coordinator.cancelInternal():974] unfinished instance: 8024bfd2fd204458-a0d2cd276775c07b 2023-11-27 07:58:31,687 WARN (mysql-nio-pool-701|52302) [StmtExecutor.handleInsertStmt():1536] handle insert stmt fail: insert_8024bfd2fd204458_a0d2cd276775c07a org.apache.doris.common.DdlException: errCode = 2, detailMessage = (192.168.1.20)[INTERNAL_ERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: GetConnectionTimeoutException: wait millis 5000, active 0, maxActive 100, creating 1, createElapseMillis 10000 at org.apache.doris.common.ErrorReport.reportDdlException(ErrorReport.java:68) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleInsertStmt(StmtExecutor.java:1496) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:555) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:409) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:333) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:476) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:703) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]

[tsj68]

doris版本：2.0.6，用一段时间后，报同样的错误，重启集群后，报错消失。

[echo-hhj]

重新创建这个catalog，带上connection_pool_max_size属性 设置为200

[wangkezhong12]

我在创建MySQL 的catalog也遇到了相同的问题，doris版本2.0.3 ，MySQL 版本8.0.37， jdbc版本8.0.32

Execution failed: Error Failed to execute sql: java.sql.SQLException: (conn=605) errCode = 2, detailMessage = (xx.xx.xx.xx)[CANCELLED][INTERNALERROR]UdfRuntimeException: Initialize datasource failed: CAUSED BY: CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. CAUSED BY: CJCommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. CAUSED BY: ConnectException: Connection timed out (Connection timed out) 0# doris::JniUtil::GetJniExceptionMsg(JNIEnv, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /root/src/doris-2.0/be/src/util/jni-util.h:110 1# doris::vectorized::JdbcConnector::open(doris::RuntimeState, bool) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187 2# doris::vectorized::NewJdbcScanner::open(doris::RuntimeState) at /root/src/doris-2.0/be/src/common/status.h:442 3# doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler, doris::vectorized::ScannerContext, std::shared_ptr) at /root/src/doris-2.0/be/src/common/status.h:354 4# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext)::$_1::operator()() const::{lambda()#4}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701 5# doris::WorkThreadPool::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646 6# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85 7# start_thread 8# __clone