apache / shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Apache License 2.0
19.8k stars 6.7k forks source link

No query condition for the sharding table, it does not working #13541

Closed Jenyow closed 2 years ago

Jenyow commented 2 years ago

Bug Report

Which version of ShardingSphere did you use?

5.0.0

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

ShardingSphere-JDBC

Expected behavior

the sql:

select count(id) from statistic_data

the application.yml fragment:

spring:
  shardingsphere:
    rules:
      sharding:
        tables:
          statistic_data:
            actual-data-nodes: statistics$->{2020..2022}.statistic_data_$->{['1820A01','1820A02','1820A03']}
            database-strategy:
              standard:
                sharding-column: get_time
                sharding-algorithm-name: get-time-database-sharding-algorithm
            table-strategy:
              standard:
                sharding-column: station_code
                sharding-algorithm-name: station-code-table-sharding-algorithm

the log

### The error may involve com.qm.data.storage.mapper.StatisticDataMapper.countStatisticData
### The error occurred while handling results
### SQL: select count(id)         from statistic_data
### Cause: java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitiveFromTables(GroupByMemoryMergedResult.java:134)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitive(GroupByMemoryMergedResult.java:124)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:73)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:53)
    at org.apache.shardingsphere.infra.merge.result.impl.memory.MemoryMergedResult.<init>(MemoryMergedResult.java:51)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.<init>(GroupByMemoryMergedResult.java:56)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.getGroupByMergedResult(ShardingDQLResultMerger.java:108)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.build(ShardingDQLResultMerger.java:78)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.merge(ShardingDQLResultMerger.java:63)
    at org.apache.shardingsphere.infra.merge.MergeEngine.executeMerge(MergeEngine.java:90)
    at org.apache.shardingsphere.infra.merge.MergeEngine.merge(MergeEngine.java:80)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.mergeQuery(ShardingSpherePreparedStatement.java:393)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.getResultSet(ShardingSpherePreparedStatement.java:339)
    at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)

It work fine in version 5.0.0-beta.

Actual behavior

Reason analyze (If you can)

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

Example codes for reproduce this issue (such as a github link).

strongduanmu commented 2 years ago

@Jenyow Thank you for your feedback, I will check it.

strongduanmu commented 2 years ago

Hi @Jenyow, I used the 5.0.0 version and the following similar sharding configuration for testing, and there was no exception.

dataSources:
  ds_0:
    url: jdbc:mysql://127.0.0.1:3306/demo_ds_0?serverTimezone=UTC&useSSL=false
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1
  ds_1:
    url: jdbc:mysql://127.0.0.1:3306/demo_ds_1?serverTimezone=UTC&useSSL=false
    username: root
    password: 123456
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
    minPoolSize: 1

rules:
- !SHARDING
  tables:
    t_order:
      actualDataNodes: ds_${0..1}.t_order_${0..1}
      tableStrategy:
        standard:
          shardingColumn: order_id
          shardingAlgorithmName: t_order_inline
      keyGenerateStrategy:
        column: order_id
        keyGeneratorName: snowflake
    t_order_item:
      actualDataNodes: ds_${0..1}.t_order_item_${0..1}
      tableStrategy:
        standard:
          shardingColumn: order_id
          shardingAlgorithmName: t_order_item_inline
      keyGenerateStrategy:
        column: order_item_id
        keyGeneratorName: snowflake
  bindingTables:
    - t_order,t_order_item
  defaultDatabaseStrategy:
    standard:
      shardingColumn: user_id
      shardingAlgorithmName: database_inline
  defaultTableStrategy:
    none:

  shardingAlgorithms:
    database_inline:
      type: INLINE
      props:
        algorithm-expression: ds_${user_id % 2}
    t_order_inline:
      type: INLINE
      props:
        algorithm-expression: t_order_${order_id % 2}
    t_order_item_inline:
      type: INLINE
      props:
        algorithm-expression: t_order_item_${order_id % 2}

  keyGenerators:
    snowflake:
      type: SNOWFLAKE
      props:
        worker-id: 123
mysql> select count(order_id) from t_order;
+-----------------+
| count(order_id) |
+-----------------+
|               4 |
+-----------------+
1 row in set (0.04 sec)

Is there any difference in your configuration and environment? If you can provide a demo program, I think it will help the positioning of this problem.

Jenyow commented 2 years ago

Hi @strongduanmu , the attch attachment is demo. Please read README.md and do it.

sharding-demo.zip

strongduanmu commented 2 years ago

@Jenyow Thank you for your detailed info, I will check it again.

strongduanmu commented 2 years ago

Hi @Jenyow Thank you very much for the perfect demo, but everything is ok when I run it. Below is my log.

==>  Preparing: select count(id) from statistic_data
==> Parameters: 
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Logic SQL: select count(id)
        from statistic_data
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --SQLStatement: MySQLSelectStatement(limit=Optional.empty, lock=Optional.empty, window=Optional.empty)
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820a01
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820a02
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820a03
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820a01
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820a02
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820a03
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820a01
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820a02
[2021-11-11 18:43:55] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820a03
<==    Columns: count(id)
<==        Row: 0
<==      Total: 1

BTW, your sql-show parameter configuration is wrong, it should be configured as.

spring:
  application:
    name: sharding-demo
  main:
    allow-bean-definition-overriding: true
  shardingsphere:
    props:
      sql-show: true
    datasource:
......
Jenyow commented 2 years ago

Hi @strongduanmu , thandks. I revise sql-show property. It show sql like your log, but still print exception, I don't know what's wrong with me.

Creating a new SqlSession
Registering transaction synchronization for SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@51d7e025]
JDBC Connection [org.apache.shardingsphere.driver.jdbc.core.connection.ShardingSphereConnection@5b2f8b55] will be managed by Spring
==>  Preparing: select count(id) from statistic_data
==> Parameters: 
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Logic SQL: select count(id)
        from statistic_data
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --SQLStatement: MySQLSelectStatement(limit=Optional.empty, lock=Optional.empty, window=Optional.empty)
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A01
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A02
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A03
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A01
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A02
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A03
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A01
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A02
[2021-11-11 20:02:31] [INFO ] http-nio-8080-exec-10 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A03
Releasing transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@51d7e025]
Transaction synchronization deregistering SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@51d7e025]
Transaction synchronization closing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@51d7e025]
[2021-11-11 20:02:31] [ERROR] http-nio-8080-exec-10 --Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: java.lang.NullPointerException
### The error may exist in file [E:\git\sharding-demo\target\classes\mapper\StatisticDataMapper.xml]
### The error may involve org.example.mapper.StatisticDataMapper.countStatisticData
### The error occurred while handling results
### SQL: select count(id)         from statistic_data
### Cause: java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitiveFromTables(GroupByMemoryMergedResult.java:134)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitive(GroupByMemoryMergedResult.java:124)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:73)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:53)
    at org.apache.shardingsphere.infra.merge.result.impl.memory.MemoryMergedResult.<init>(MemoryMergedResult.java:51)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.<init>(GroupByMemoryMergedResult.java:56)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.getGroupByMergedResult(ShardingDQLResultMerger.java:108)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.build(ShardingDQLResultMerger.java:78)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.merge(ShardingDQLResultMerger.java:63)
    at org.apache.shardingsphere.infra.merge.MergeEngine.executeMerge(MergeEngine.java:90)
    at org.apache.shardingsphere.infra.merge.MergeEngine.merge(MergeEngine.java:80)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.mergeQuery(ShardingSpherePreparedStatement.java:393)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.getResultSet(ShardingSpherePreparedStatement.java:339)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:69)
    at com.sun.proxy.$Proxy245.getResultSet(Unknown Source)
    at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getFirstResultSet(DefaultResultSetHandler.java:238)
    at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:188)
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64)
    at com.sun.proxy.$Proxy244.query(Unknown Source)
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:81)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
    at com.sun.proxy.$Proxy243.query(Unknown Source)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
    at com.sun.proxy.$Proxy202.selectOne(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:160)
    at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy$PlainMethodInvoker.invoke(MybatisMapperProxy.java:148)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:89)
    at com.sun.proxy.$Proxy203.countStatisticData(Unknown Source)
    at org.example.service.impl.StatisticDataServiceImpl.countStatisticDatas(StatisticDataServiceImpl.java:20)
    at org.example.service.impl.StatisticDataServiceImpl$$FastClassBySpringCGLIB$$283450a2.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
    at org.example.service.impl.StatisticDataServiceImpl$$EnhancerBySpringCGLIB$$d3ce665.countStatisticDatas(<generated>)
    at org.example.controller.StatisticDataController.countStatisticDatas(StatisticDataController.java:27)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Jenyow commented 2 years ago

Hi @strongduanmu. When I changed the version to 5.0.0-beta, it worked fine. But when I change the version to 5.0.0, it always print the exception. So I'm sure this is a bug in version 5.0.0.

use 5.0.0-beta the log:

==>  Preparing: select count(id) from statistic_data
==> Parameters: 
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Logic SQL: select count(id)
        from statistic_data
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --SQLStatement: MySQLSelectStatement(limit=Optional.empty, lock=Optional.empty, window=Optional.empty)
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A03
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A03
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:18:02] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A03
<==    Columns: count(id)
<==        Row: 0
<==      Total: 1

use 5.0.0 the log:

==>  Preparing: select count(id) from statistic_data
==> Parameters: 
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Logic SQL: select count(id)
        from statistic_data
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --SQLStatement: MySQLSelectStatement(limit=Optional.empty, lock=Optional.empty, window=Optional.empty)
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2020 ::: select count(id)
        from statistic_data_1820A03
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2021 ::: select count(id)
        from statistic_data_1820A03
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A01
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A02
[2021-11-12 00:19:48] [INFO ] http-nio-8080-exec-1 --Actual SQL: statistics2022 ::: select count(id)
        from statistic_data_1820A03
Releasing transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5d6494b7]
Transaction synchronization deregistering SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5d6494b7]
Transaction synchronization closing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5d6494b7]
[2021-11-12 00:19:48] [ERROR] http-nio-8080-exec-1 --Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: java.lang.NullPointerException
### The error may exist in file [E:\git\sharding-demo\target\classes\mapper\StatisticDataMapper.xml]
### The error may involve org.example.mapper.StatisticDataMapper.countStatisticData
### The error occurred while handling results
### SQL: select count(id)         from statistic_data
### Cause: java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitiveFromTables(GroupByMemoryMergedResult.java:134)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.getValueCaseSensitive(GroupByMemoryMergedResult.java:124)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:73)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.init(GroupByMemoryMergedResult.java:53)
    at org.apache.shardingsphere.infra.merge.result.impl.memory.MemoryMergedResult.<init>(MemoryMergedResult.java:51)
    at org.apache.shardingsphere.sharding.merge.dql.groupby.GroupByMemoryMergedResult.<init>(GroupByMemoryMergedResult.java:56)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.getGroupByMergedResult(ShardingDQLResultMerger.java:108)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.build(ShardingDQLResultMerger.java:78)
    at org.apache.shardingsphere.sharding.merge.dql.ShardingDQLResultMerger.merge(ShardingDQLResultMerger.java:63)
    at org.apache.shardingsphere.infra.merge.MergeEngine.executeMerge(MergeEngine.java:90)
    at org.apache.shardingsphere.infra.merge.MergeEngine.merge(MergeEngine.java:80)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.mergeQuery(ShardingSpherePreparedStatement.java:393)
    at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.getResultSet(ShardingSpherePreparedStatement.java:339)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:69)
    at com.sun.proxy.$Proxy232.getResultSet(Unknown Source)
    at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getFirstResultSet(DefaultResultSetHandler.java:238)
    at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:188)
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64)
    at com.sun.proxy.$Proxy231.query(Unknown Source)
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:81)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
    at com.sun.proxy.$Proxy230.query(Unknown Source)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
    at com.sun.proxy.$Proxy202.selectOne(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:160)
    at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy$PlainMethodInvoker.invoke(MybatisMapperProxy.java:148)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:89)
    at com.sun.proxy.$Proxy203.countStatisticData(Unknown Source)
    at org.example.service.impl.StatisticDataServiceImpl.countStatisticDatas(StatisticDataServiceImpl.java:20)
    at org.example.service.impl.StatisticDataServiceImpl$$FastClassBySpringCGLIB$$283450a2.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
    at org.example.service.impl.StatisticDataServiceImpl$$EnhancerBySpringCGLIB$$d3ce665.countStatisticDatas(<generated>)
    at org.example.controller.StatisticDataController.countStatisticDatas(StatisticDataController.java:27)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

Please check it again. Thanks.

strongduanmu commented 2 years ago

@Jenyow Ok, I will try again.

strongduanmu commented 2 years ago

@Jenyow I inserted a piece of data into the table and executed it again, but there was still no exception. Can you get exceptions through demo execution?

For further communication, can you send the wechat id to my email——duanzhengqiang@apache.org?

image

Jenyow commented 2 years ago

Hi @strongduanmu , I sent an email to you, please add me.

strongduanmu commented 2 years ago

@Jenyow This exception is due to the fact that actualDataNodes does not consider case-insensitiveness, which leads to metadata wrong. You can temporarily modify it to be consistent with the actual table name of the database to avoid this problem.

NOLouser commented 2 years ago

@strongduanmu Excuse me, will this bug be fixed in next 5.1.0, and when will it be released?

strongduanmu commented 2 years ago

@NoLouser Yes, this bug has been fixed in master branch. You can try it.

NOLouser commented 2 years ago

@NOLouser Yes, this bug has been fixed in master branch. You can try it.

@strongduanmu This bug still exists in master branch!

I think this is the real cause of the problem! image

strongduanmu commented 2 years ago

@NOLouser Please check whether all the tables configured in your actualDataNode exist. This issue solves the existence of the table, but the metadata is abnormal due to the capitalization issue.

NOLouser commented 2 years ago

@NOLouser Please check whether all the tables configured in your actualDataNode exist. This issue solves the existence of the table, but the metadata is abnormal due to the capitalization issue.

@strongduanmu image image image

strongduanmu commented 2 years ago

@Jenyow From your debug information, I can see that the metadata is somewhat abnormal. Can you create a new issue to report this problem?

NOLouser commented 2 years ago

@Jenyow From your debug information, I can see that the metadata is somewhat abnormal. Can you create a new issue to report this problem?

@strongduanmu Ok, you can look at this question. Aggregations such as max, sum, count got NullPointerException using shardingsphere-jdbc 5.0.0