Closed galaxylqx closed 2 years ago
Actually, I thought about a similar solution. Referring to the processing ideas of https://github.com/baomidou/dynamic-datasource-spring-boot-starter , I think it is possible to add the global configuration Map
of hikaricp
, dbcp
and tomcat dbcp
under spring.datasource.shardingsphere
(with these JDBC pool is used as an example because the source code of shardingsphere
imports these three components as the implementation of the metadata spi). The data source instance can automatically extends these automatic configurations, and the configuration in the data source instance will override the globally configured JDBC pool properties. But the shardingsphere proxy
is being retrofitted with vert.x client
, which makes me wonder if it will be too complicated.
spring:
datasource:
dynamic:
hikari:
catalog:
connection-timeout:
validation-timeout:
idle-timeout:
leak-detection-threshold:
max-lifetime:
max-pool-size:
min-idle:
initialization-fail-timeout:
connection-init-sql:
connection-test-query:
dataSource-class-name:
dataSource-jndi-name:
schema:
transaction-isolation-name:
is-auto-commit:
is-read-only:
is-isolate-internal-queries:
is-register-mbeans:
is-allow-pool-suspension:
data-source-properties:
health-check-properties:
datasource:
master:
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
url: jdbc:mysql://xx.xx.xx.xx:3306/dynamic?characterEncoding=utf8&useSSL=false
hikari:
max-pool-size:
idle-timeout:
@galaxylqx any example/doc for how to config like this? I test with following config in shardingproxy 5.3.2 but can't start server
schemaName: sbtest
dataSources:
ds:
url: jdbc:mysql://127.0.0.1:3306/?characterEncoding=utf8&useSSL=false&useUnicode=true
username: xxx
password: xxx
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
rules:
- !SHARDING
tables:
sbtest1:
actualDataNodes: ds.sbtest_${0..15}.sbtest1_${0..15}
databaseStrategy:
standard:
shardingColumn: k
shardingAlgorithmName: database_k_inline
tableStrategy:
standard:
shardingColumn: k
shardingAlgorithmName: sbtest1_inline
keyGenerateStrategy:
column: id
keyGeneratorName: snowflake
defaultKeyGenerateStrategy:
column: id
keyGeneratorName: snowflake
shardingAlgorithms:
database_k_inline:
type: INLINE
props:
algorithm-expression: sbtest_${k ? k % 16:k % 16}
sbtest1_inline:
type: INLINE
props:
algorithm-expression: sbtest1_${k ? k.intdiv(16) % 16:k.intdiv(16) % 16}
keyGenerators:
snowflake:
type: SNOWFLAKE
props:
worker-id: 1
# autoinc:
# type: AUTOINC
# props:
# dataType: 4
# primaryKey: true
# unsigned: false
here is the log:
[INFO ] 2024-07-03 09:18:27.648 [main] com.zaxxer.hikari.HikariDataSource:110 - HikariPool-1 - Starting...
[INFO ] 2024-07-03 09:18:28.078 [main] com.zaxxer.hikari.HikariDataSource:123 - HikariPool-1 - Start completed.
[INFO ] 2024-07-03 09:18:28.473 [main] com.zaxxer.hikari.HikariDataSource:110 - HikariPool-2 - Starting...
[INFO ] 2024-07-03 09:18:28.750 [main] com.zaxxer.hikari.HikariDataSource:123 - HikariPool-2 - Start completed.
Exception in thread "main" java.sql.SQLException: Unknown exception: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
at org.apache.shardingsphere.infra.util.exception.external.sql.ShardingSphereSQLException.toSQLException(ShardingSphereSQLException.java:62)
at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:72)
at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.loadSchemas(GenericSchemaBuilder.java:102)
at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.build(GenericSchemaBuilder.java:76)
at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.build(GenericSchemaBuilder.java:64)
at org.apache.shardingsphere.infra.metadata.database.ShardingSphereDatabase.create(ShardingSphereDatabase.java:89)
at org.apache.shardingsphere.metadata.factory.ExternalMetaDataFactory.createGenericDatabases(ExternalMetaDataFactory.java:79)
at org.apache.shardingsphere.metadata.factory.ExternalMetaDataFactory.create(ExternalMetaDataFactory.java:67)
at org.apache.shardingsphere.mode.metadata.MetaDataContextsFactory.create(MetaDataContextsFactory.java:91)
at org.apache.shardingsphere.mode.metadata.MetaDataContextsFactory.create(MetaDataContextsFactory.java:68)
at org.apache.shardingsphere.mode.manager.standalone.StandaloneContextManagerBuilder.build(StandaloneContextManagerBuilder.java:53)
at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.createContextManager(BootstrapInitializer.java:69)
at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:59)
at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:54)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:66)
... 12 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:78)
at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.lambda$load$0(SchemaMetaDataLoaderEngine.java:62)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Problem description: Multiple data sources use the same database instance. For example, if the host is 127.0.0.1 and the port is 3306, multiple data sources need to be created. If the same database instance has 100 databases, 100 data sources need to be created.
Solution: The data source connection does not specify a specific database, the data source is only a database connection, and the real database corresponding to the data source is maintained through memory map mapping.
Add map in memory for data source and database mapping relationship