search

polardb / polardbx-sql

PolarDB-X is a cloud native distributed SQL Database designed for high concurrency, massive storage, complex querying scenarios.
Apache License 2.0
1.54k stars 325 forks source link

ERROR jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns #51

Open wsm12138 opened 2 years ago

wsm12138 commented 2 years ago

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
    at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
    at jTPCCTData.execute(jTPCCTData.java:95)
    at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
    at jTPCCTerminal.run(jTPCCTerminal.java:88)
    at java.lang.Thread.run(Thread.java:748)

Try to fix

  1. restart cdc,dn,cn,gms of PolarDB-x in docker —— Errors still occur
  2. related #34 ,check variable —— galaxyXProtocol=1 
    [admin@centos71 drds-server]$ cat conf/server.properties
serverPort=3306
managerPort=3406
rpcPort=9090
charset=utf-8
processors=4
processorHandler=16
processorKillExecutor=128
timerExecutor=8
managerExecutor=256
serverExecutor=1024
idleTimeout=
trustedIps=127.0.0.1
slowSqlTime=1000
maxConnection=20000
allowManagerLogin=1
allowCrossDbQuery=true
galaxyXProtocol=1
metaDbAddr=127.0.0.1:3306
metaDbXprotoPort=13306
metaDbUser=my_polarx
metaDbName=polardbx_meta_db_polardbx
instanceId=polardbx-polardbx
  3. Telnet —— network & port alive image
wsm12138 commented 2 years ago

when terminals=50 image

when terminals=200 Errors still occur

target isterminals=500

wsm12138 commented 2 years ago

According to the https://www.alibabacloud.com/help/zh/polardb-x/latest/tpc-c-test proved data integrity. image

F-ca7 commented 2 years ago

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
  at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
  at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
  at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
  at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
  at jTPCCTData.execute(jTPCCTData.java:95)
  at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
  at jTPCCTerminal.run(jTPCCTerminal.java:88)
  at java.lang.Thread.run(Thread.java:748)

What about the hardware resources allocated for PolarDB-X? This error (XClientPool to admin@10.16.9.192:44881 get XConnection timeout) might be caused by a DN crash or hang in the case of high concurrency pressure. You can also check the log file for more information.

wsm12138 commented 2 years ago

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to admin@10.16.9.192:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to admin@10.16.9.192:44881 get XConnection timeout. 5000000000ns
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
    at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
    at jTPCCTData.execute(jTPCCTData.java:95)
    at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
    at jTPCCTerminal.run(jTPCCTerminal.java:88)
    at java.lang.Thread.run(Thread.java:748)

What about the hardware resources allocated for PolarDB-X? This error (XClientPool to admin@10.16.9.192:44881 get XConnection timeout) might be caused by a DN crash or hang in the case of high concurrency pressure. You can also check the log file for more information.

hardware resources 32c 64G

version: v1
type: polardbx
cluster:
  name: polardb_4dn
  gms:
    image: polardbx/galaxyengine:latest
    host_group: [10.16.9.224]
  cn:
    image: polardbx/galaxysql:latest
    replica: 1
    nodes:
      - host: 10.16.9.12
    resources:
      mem_limit: 20G
  dn:
    image: polardbx/galaxyengine:latest
    replica: 4
    nodes:
      - host_group: [10.16.9.62]
      - host_group: [10.16.9.192]
      - host_group: [10.16.9.120]
      - host_group: [10.16.9.16]
    resources:
      mem_limit: 20G
  cdc:
    image: polardbx/galaxycdc:latest
    replica: 1
    nodes:
      - host: 10.16.9.16
    resources:
      mem_limit: 20G

When running polardb-x cluster ，No other application is running, that is, the server resources are fully allocated to polardb-x cluster.

might be caused by a DN crash or hang in the case of high concurrency pressure. With the above resources, I don't think PolarDB-X can't pressure more than 200 concurrent. Standalone MySQL is fine with the same configuration