baomidou / mybatis-plus

An powerful enhanced toolkit of MyBatis for simplify development
https://baomidou.com
Apache License 2.0
16.42k stars 4.31k forks source link

多数据源集成spring-boot-starter-quartz导致数据源切换错误 #3120

Closed wangliujing closed 3 years ago

wangliujing commented 3 years ago

当前使用版本(必填,否则不予处理)

3.2.0

该问题是如何引起的?(确定最新版也有问题再提!!!)

业务库与quartz库是分开的,项目需要采用多数据源的方式开发,则集成了mybatispluse提供的多数据源功能,后发现切换数据源的过程频繁出错

重现步骤(如果有就写完整)

quartz配置类如下 @SpringBootConfiguration public class CustomQuartzAutoConfiguration {

/**
 * 设置schedulerFactory初始化属性
 * @author wangLiuJing
 * Created on 2020/11/30
 *
 * @param dataSource of type DynamicRoutingDataSource
 * @param dataSourceTransactionManager of type DataSourceTransactionManager
 * @return SchedulerFactoryBeanCustomizer
 */
@Order(1)
@Bean
public SchedulerFactoryBeanCustomizer schedulerFactoryBeanCustomizer(DynamicRoutingDataSource dataSource,
                                                                     DataSourceTransactionManager dataSourceTransactionManager) {
    return schedulerFactoryBean -> {
        schedulerFactoryBean.setDataSource(dataSource);
        schedulerFactoryBean.setTransactionManager(dataSourceTransactionManager);
        // 延时5秒启动
        schedulerFactoryBean.setStartupDelay(5);
        schedulerFactoryBean.setAutoStartup(true);
        schedulerFactoryBean.setOverwriteExistingJobs(true);
        // 设置全局的定时任务日志处理
        schedulerFactoryBean.setGlobalJobListeners(new QuartzJobLogListener());
    };
}

/**
 * 通过SchedulerFactoryBean获取Scheduler的实例
 */
@Bean
public Scheduler scheduler(SchedulerFactoryBean schedulerFactoryBean) {
    return schedulerFactoryBean.getScheduler();
}

}

相关核心配置如下 spring: datasource: type: com.baomidou.dynamic.datasource.DynamicRoutingDataSource dynamic: primary: main datasource: main: driverClassName: com.mysql.cj.jdbc.Driver url: jdbc:mysql://top.intellzens.com.cn:3306/honghu_datacenter_quartz?characterEncoding=utf8&connectTimeout=1000&socketTimeout=300000&autoReconnect=true&serverTimezone=Asia/Shanghai username: root password: IntellZens1104 kettle: driverClassName: com.mysql.cj.jdbc.Driver url: jdbc:mysql://top.intellzens.com.cn:3306/honghu_datacenter_kettle?characterEncoding=utf8&connectTimeout=1000&socketTimeout=300000&autoReconnect=true&serverTimezone=Asia/Shanghai username: root password: IntellZens1104

quartz: job-store-type: jdbc properties: org: quartz: threadPool: class: org.quartz.simpl.SimpleThreadPool threadCount: 100 threadPriority: 5 threadsInheritContextClassLoaderOfInitializingThread: true jobStore: class: org.quartz.impl.jdbcjobstore.JobStoreTX driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate dataSource: quartzDataSource tablePrefix: qrtz_ isClustered: true clusterCheckinInterval: 2000 misfireThreshold: 3000 useProperties: false scheduler: instanceName: clusteredScheduler instanceId: AUTO

看如下日志,分析问题出现原因:

2020-12-01 15:56:41.004 ERROR 12676 --- [dulerExecutor-2] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【257】, 绑定数据源【kettle】, 线程【Thread[TaskSchedulerExecutor-2,5,main]】, 获取ArrayDeque【[kettle]】,ArrayDeque【824736048】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-2] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【257】, 获取ArrayDeque【[kettle]】,ArrayDeque【824736048】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-1] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【256】, 获取ArrayDeque【[kettle]】,ArrayDeque【824736048】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-2] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【257】, 获取数据源【kettle】, 线程【Thread[TaskSchedulerExecutor-2,5,main]】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-1] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【256】, 获取数据源【kettle】, 线程【Thread[TaskSchedulerExecutor-1,5,main]】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-2] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【257】, 获取ArrayDeque【[kettle]】,ArrayDeque【824736048】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-1] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【256】, 获取ArrayDeque【[kettle]】,ArrayDeque【824736048】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-2] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【257】, 获取数据源【kettle】, 线程【Thread[TaskSchedulerExecutor-2,5,main]】 2020-12-01 15:56:41.098 ERROR 12676 --- [dulerExecutor-1] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【256】, 获取数据源【kettle】, 线程【Thread[TaskSchedulerExecutor-1,5,main]】

线程256没有绑定数据源,但是获取到数据源key=kettle,很明显看第一行日志数据源kettle首先是有线程257绑定,那257绑定的key为什么被256获取到了,再看栈容器ArrayDeque的hash值可以发现257跟256绑定了同一个栈容器,我们再看如下,前面的日志分析ArrayDeque的容器是被谁创建的

2020-12-01 15:56:37.056 ERROR 12676 --- [ main] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【1】, 创建ArrayDeque【[]】,ArrayDeque【824736048】==================== 2020-12-01 15:56:37.056 ERROR 12676 --- [ main] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【1】, 获取ArrayDeque【[]】,ArrayDeque【824736048】 2020-12-01 15:56:37.057 ERROR 12676 --- [ main] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【1】, 获取数据源【null】, 线程【Thread[main,5,main]】 2020-12-01 15:56:37.261 INFO 12676 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor 2020-12-01 15:56:37.265 INFO 12676 --- [ main] org.quartz.simpl.SimpleThreadPool : Job execution threads will use class loader of thread: main 2020-12-01 15:56:37.294 INFO 12676 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2020-12-01 15:56:37.295 INFO 12676 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.1 created. 2020-12-01 15:56:37.303 ERROR 12676 --- [ main] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【1】, 获取ArrayDeque【[]】,ArrayDeque【824736048】 2020-12-01 15:56:37.303 ERROR 12676 --- [ main] c.b.d.d.t.DynamicDataSourceContextHolder : 线程ID【1】, 获取数据源【null】, 线程【Thread[main,5,main]】

上面日志可以看出ArrayDeque是被线程1创建,这里可以发现后面几步中ThreadLocal执行get的时候并未重新初始化ArrayDeque

修改相关代码,问题解决: `/**

import java.util.ArrayDeque; import java.util.ArrayList; import java.util.Deque; import java.util.List;

import lombok.extern.slf4j.Slf4j; import org.springframework.core.NamedInheritableThreadLocal; import org.springframework.util.StringUtils;

/**

报错信息

Caused by: java.sql.SQLSyntaxErrorException: Table 'ckettle.dc_kettle_task' doesn't exist at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) ~[mysql-connector-java-8.0.17.jar:8.0.17] at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.17.jar:8.0.17] at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.17.jar:8.0.17] at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.17.jar:8.0.17] at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:370) ~[mysql-connector-java-8.0.17.jar:8.0.17] at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3461) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.wall.WallFilter.preparedStatement_execute(WallFilter.java:660) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.2.3.jar:1.2.3] at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[druid-1.2.3.jar:1.2.3] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171] at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59) ~[mybatis-3.5.2.jar:3.5.2] at com.sun.proxy.$Proxy241.execute(Unknown Source) ~[na:na] at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.2.jar:3.5.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171] at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63) ~[mybatis-3.5.2.jar:3.5.2] at com.sun.proxy.$Proxy239.query(Unknown Source) ~[na:na] at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67) ~[mybatis-plus-core-3.2.0.jar:3.2.0] at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[mybatis-3.5.2.jar:3.5.2] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) ~[mybatis-3.5.2.jar:3.5.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ~[mybatis-spring-2.0.2.jar:2.0.2] ... 30 common frames omitted

huayanYu commented 3 years ago

@wangliujing 动态数据源的问题请在那边提,我很少会来看这边的ISSUE。~ 请查看最新文档和示例项目,如仍有问题。建议fork示例项目,给出复现步骤在那边提ISSUE。

wangliujing commented 3 years ago

/**

import lombok.extern.slf4j.Slf4j; import org.springframework.core.NamedThreadLocal; import org.springframework.util.StringUtils;

import java.util.ArrayDeque; import java.util.Deque;

/**

这块又出现bug了,我已经确定,如果并发情况下去切换数据源很容易出现我说的问题,把源码修改成上面这种就好了,我也不知道为什么,看源码感觉跟这种是一样的,只不过变换了一种写法,但是测试的确我这种写法不会有问题,源码的写法会导致数据库切换混乱,比如本来是应该获取A数据源,但是实际获取的是B,最新的我看过了,还是有问题 ------------------ 原始邮件 ------------------ 发件人: "baomidou/mybatis-plus" <notifications@github.com>; 发送时间: 2021年1月15日(星期五) 下午3:21 收件人: "baomidou/mybatis-plus"<mybatis-plus@noreply.github.com>; 抄送: "王柳敬"<273833883@qq.com>;"Mention"<mention@noreply.github.com>; 主题: Re: [baomidou/mybatis-plus] 多数据源集成spring-boot-starter-quartz导致数据源切换错误 (#3120)

@wangliujing 动态数据源的问题请在那边提,我很少会来看这边的ISSUE。~ 请查看最新文档和示例项目,如仍有问题。建议fork示例项目,给出复现步骤在那边提ISSUE。

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

wangliujing commented 3 years ago

------------------ 原始邮件 ------------------ 发件人: "王柳敬" <273833883@qq.com>; 发送时间: 2021年2月1日(星期一) 下午4:49 收件人: "baomidou/mybatis-plus"<reply@reply.github.com>;

主题: 回复: [baomidou/mybatis-plus] 多数据源集成spring-boot-starter-quartz导致数据源切换错误 (#3120)

/**

import lombok.extern.slf4j.Slf4j; import org.springframework.core.NamedThreadLocal; import org.springframework.util.StringUtils;

import java.util.ArrayDeque; import java.util.Deque;

/**

这块又出现bug了,我已经确定,如果并发情况下去切换数据源很容易出现我说的问题,把源码修改成上面这种就好了,我也不知道为什么,看源码感觉跟这种是一样的,只不过变换了一种写法,但是测试的确我这种写法不会有问题,源码的写法会导致数据库切换混乱,比如本来是应该获取A数据源,但是实际获取的是B,最新的我看过了,还是有问题 ------------------ 原始邮件 ------------------ 发件人: "baomidou/mybatis-plus" <notifications@github.com>; 发送时间: 2021年1月15日(星期五) 下午3:21 收件人: "baomidou/mybatis-plus"<mybatis-plus@noreply.github.com>; 抄送: "王柳敬"<273833883@qq.com>;"Mention"<mention@noreply.github.com>; 主题: Re: [baomidou/mybatis-plus] 多数据源集成spring-boot-starter-quartz导致数据源切换错误 (#3120)

@wangliujing 动态数据源的问题请在那边提,我很少会来看这边的ISSUE。~ 请查看最新文档和示例项目,如仍有问题。建议fork示例项目,给出复现步骤在那边提ISSUE。

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

luohaoGit commented 7 months ago

我也遇到类似问题