apache / shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Apache License 2.0
19.88k stars 6.73k forks source link

Data source simplification #15881

Closed galaxylqx closed 2 years ago

galaxylqx commented 2 years ago

Problem description: Multiple data sources use the same database instance. For example, if the host is 127.0.0.1 and the port is 3306, multiple data sources need to be created. If the same database instance has 100 databases, 100 data sources need to be created.

schemaName: sharding_db
  dataSources:
    ds_0:
      url: jdbc:mysql://127.0.0.1:3306/order_db_0?serverTimezone=UTC&useSSL=false
      username: root
      password:
      connectionTimeoutMilliseconds: 30000
      idleTimeoutMilliseconds: 60000
      maxLifetimeMilliseconds: 1800000
      maxPoolSize: 50
      minPoolSize: 1
    ds_1:
      url: jdbc:mysql://127.0.0.1:3306/order_db_1?serverTimezone=UTC&useSSL=false
      username: root
      password:
      connectionTimeoutMilliseconds: 30000
      idleTimeoutMilliseconds: 60000
      maxLifetimeMilliseconds: 1800000
      maxPoolSize: 50
      minPoolSize: 1
 rules:
 - !SHARDING
   tables:
     t_order:
       actualDataNodes: ds_${0..1}.t_order_${0..1}
       tableStrategy:
         standard:
           shardingColumn: order_id
           shardingAlgorithmName: t_order_inline

Solution: The data source connection does not specify a specific database, the data source is only a database connection, and the real database corresponding to the data source is maintained through memory map mapping.

schemaName: sharding_db
dataSources:
  ds:
    url: jdbc:mysql://127.0.0.1:3306/?serverTimezone=UTC&useSSL=false
    username: root
    password:
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1
 rules:
- !SHARDING
  tables:
    t_order:
      actualDataNodes: ds.order_db_${0..1}.t_order_${0..1}
      tableStrategy:
        standard:
          shardingColumn: order_id
          shardingAlgorithmName: t_order_inline

Add map in memory for data source and database mapping relationship

key value
ds.order_db_0 order_db_0
ds.order_db_1 order_db_1
linghengqian commented 2 years ago

Actually, I thought about a similar solution. Referring to the processing ideas of https://github.com/baomidou/dynamic-datasource-spring-boot-starter , I think it is possible to add the global configuration Map of hikaricp, dbcp and tomcat dbcp under spring.datasource.shardingsphere (with these JDBC pool is used as an example because the source code of shardingsphere imports these three components as the implementation of the metadata spi). The data source instance can automatically extends these automatic configurations, and the configuration in the data source instance will override the globally configured JDBC pool properties. But the shardingsphere proxy is being retrofitted with vert.x client, which makes me wonder if it will be too complicated.

spring:
  datasource:
    dynamic:
      hikari:  
        catalog:
        connection-timeout:
        validation-timeout:
        idle-timeout:
        leak-detection-threshold:
        max-lifetime:
        max-pool-size:
        min-idle:
        initialization-fail-timeout:
        connection-init-sql:
        connection-test-query:
        dataSource-class-name:
        dataSource-jndi-name:
        schema:
        transaction-isolation-name:
        is-auto-commit:
        is-read-only:
        is-isolate-internal-queries:
        is-register-mbeans:
        is-allow-pool-suspension:
        data-source-properties: 
        health-check-properties:
      datasource:
        master:
          username: root
          password: 123456
          driver-class-name: com.mysql.jdbc.Driver
          url: jdbc:mysql://xx.xx.xx.xx:3306/dynamic?characterEncoding=utf8&useSSL=false
          hikari: 
            max-pool-size:
            idle-timeout:
huntkalio commented 3 months ago

@galaxylqx any example/doc for how to config like this? I test with following config in shardingproxy 5.3.2 but can't start server

schemaName: sbtest

dataSources:
  ds:
    url: jdbc:mysql://127.0.0.1:3306/?characterEncoding=utf8&useSSL=false&useUnicode=true
    username: xxx
    password: xxx
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1

rules:
  - !SHARDING
    tables:
      sbtest1:
        actualDataNodes: ds.sbtest_${0..15}.sbtest1_${0..15}
        databaseStrategy:
          standard:
            shardingColumn: k
            shardingAlgorithmName: database_k_inline
        tableStrategy:
          standard:
            shardingColumn: k
            shardingAlgorithmName: sbtest1_inline
        keyGenerateStrategy:
            column: id
            keyGeneratorName: snowflake
    defaultKeyGenerateStrategy:
      column: id
      keyGeneratorName: snowflake

    shardingAlgorithms:
      database_k_inline:
        type: INLINE
        props:
          algorithm-expression: sbtest_${k ? k % 16:k % 16}
      sbtest1_inline:
        type: INLINE
        props:
          algorithm-expression: sbtest1_${k ? k.intdiv(16) % 16:k.intdiv(16) % 16}

    keyGenerators:
      snowflake:
        type: SNOWFLAKE
        props:
          worker-id: 1
      # autoinc:
      #   type: AUTOINC
      #   props:
      #     dataType: 4
      #     primaryKey: true
      #     unsigned: false

here is the log:

[INFO ] 2024-07-03 09:18:27.648 [main] com.zaxxer.hikari.HikariDataSource:110  - HikariPool-1 - Starting...
[INFO ] 2024-07-03 09:18:28.078 [main] com.zaxxer.hikari.HikariDataSource:123  - HikariPool-1 - Start completed.
[INFO ] 2024-07-03 09:18:28.473 [main] com.zaxxer.hikari.HikariDataSource:110  - HikariPool-2 - Starting...
[INFO ] 2024-07-03 09:18:28.750 [main] com.zaxxer.hikari.HikariDataSource:123  - HikariPool-2 - Start completed.
Exception in thread "main" java.sql.SQLException: Unknown exception: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
        at org.apache.shardingsphere.infra.util.exception.external.sql.ShardingSphereSQLException.toSQLException(ShardingSphereSQLException.java:62)
        at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:72)
        at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.loadSchemas(GenericSchemaBuilder.java:102)
        at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.build(GenericSchemaBuilder.java:76)
        at org.apache.shardingsphere.infra.metadata.database.schema.builder.GenericSchemaBuilder.build(GenericSchemaBuilder.java:64)
        at org.apache.shardingsphere.infra.metadata.database.ShardingSphereDatabase.create(ShardingSphereDatabase.java:89)
        at org.apache.shardingsphere.metadata.factory.ExternalMetaDataFactory.createGenericDatabases(ExternalMetaDataFactory.java:79)
        at org.apache.shardingsphere.metadata.factory.ExternalMetaDataFactory.create(ExternalMetaDataFactory.java:67)
        at org.apache.shardingsphere.mode.metadata.MetaDataContextsFactory.create(MetaDataContextsFactory.java:91)
        at org.apache.shardingsphere.mode.metadata.MetaDataContextsFactory.create(MetaDataContextsFactory.java:68)
        at org.apache.shardingsphere.mode.manager.standalone.StandaloneContextManagerBuilder.build(StandaloneContextManagerBuilder.java:53)
        at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.createContextManager(BootstrapInitializer.java:69)
        at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:59)
        at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:54)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:66)
        ... 12 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.shardingsphere.infra.database.type.DatabaseType.getType()" because the return value of "org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderMaterial.getStorageType()" is null
        at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.load(SchemaMetaDataLoaderEngine.java:78)
        at org.apache.shardingsphere.infra.metadata.database.schema.loader.metadata.SchemaMetaDataLoaderEngine.lambda$load$0(SchemaMetaDataLoaderEngine.java:62)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)