[x] I have searched the issues of this repository and believe that this is not a duplicate.
Ⅰ. Issue Description
mysql databases has table over 500;
the springboot starting in create sharding-jdbc's datasource enter into deadlock;
because sharding-jdbc query database's table metadata for too long time,
the seata create RMclient which start thread in delay 60s,
the thread get bean into spring container ,waiting lock in singleObject, but the springboot start-main-thread hold this lock
I have to update source code,update SCHEDULE_DELAY_MILLS from 60s to 180s,and start success
io.seata.core.rpc.netty.AbstractNettyRemotingClient.class
private static final long SCHEDULE_DELAY_MILLS = 60 * 1000L;
how easier config SCHEDULE_DELAY_MILLS ,do not modify source code?
stack log:
Found one Java-level deadlock:
"timeoutChecker_2_1":
waiting to lock monitor 0x0000023ff219ab08 (object 0x00000007910e81a8, a java.lang.Class),
which is held by "timeoutChecker_1_1"
"timeoutChecker_1_1":
waiting to lock monitor 0x0000023fedde9908 (object 0x00000006cac04dd8, a java.util.concurrent.ConcurrentHashMap),
which is held by "main"
"main":
waiting to lock monitor 0x0000023ff219ab08 (object 0x00000007910e81a8, a java.lang.Class),
which is held by "timeoutChecker_1_1"
Java stack information for the threads listed above:
"timeoutChecker_2_1":
at io.seata.discovery.registry.RegistryFactory.getInstance(RegistryFactory.java:42)
waiting to lock <0x00000007910e81a8> (a java.lang.Class for io.seata.discovery.registry.RegistryFactory)
at io.seata.core.rpc.netty.NettyClientChannelManager.getAvailServerList(NettyClientChannelManager.java:231)
at io.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:166)
at io.seata.core.rpc.netty.AbstractNettyRemotingClient$1.run(AbstractNettyRemotingClient.java:114)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"timeoutChecker_1_1":
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:986)
waiting to lock <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:884)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:619)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:536)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:503)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:480)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:473)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1159)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:420)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:349)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1127)
at io.seata.spring.boot.autoconfigure.provider.SpringBootConfigurationProvider.get(SpringBootConfigurationProvider.java:93)
at io.seata.spring.boot.autoconfigure.provider.SpringBootConfigurationProvider.access$100(SpringBootConfigurationProvider.java:43)
at io.seata.spring.boot.autoconfigure.provider.SpringBootConfigurationProvider$1.intercept(SpringBootConfigurationProvider.java:56)
at io.seata.config.FileConfiguration$$EnhancerByCGLIB$$862af1eb.getConfig()
at io.seata.discovery.registry.RegistryFactory.buildRegistryService(RegistryFactory.java:52)
at io.seata.discovery.registry.RegistryFactory.getInstance(RegistryFactory.java:43)
locked <0x00000007910e81a8> (a java.lang.Class for io.seata.discovery.registry.RegistryFactory)
at io.seata.core.rpc.netty.NettyClientChannelManager.getAvailServerList(NettyClientChannelManager.java:231)
at io.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:166)
at io.seata.core.rpc.netty.AbstractNettyRemotingClient$1.run(AbstractNettyRemotingClient.java:114)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"main":
at io.seata.discovery.registry.RegistryFactory.getInstance(RegistryFactory.java:42)
waiting to lock <0x00000007910e81a8> (a java.lang.Class for io.seata.discovery.registry.RegistryFactory)
at io.seata.core.rpc.netty.NettyClientChannelManager.getAvailServerList(NettyClientChannelManager.java:231)
at io.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:166)
at io.seata.core.rpc.netty.RmNettyRemotingClient.registerResource(RmNettyRemotingClient.java:195)
at io.seata.rm.AbstractResourceManager.registerResource(AbstractResourceManager.java:121)
at io.seata.rm.datasource.DataSourceManager.registerResource(DataSourceManager.java:120)
at io.seata.rm.DefaultResourceManager.registerResource(DefaultResourceManager.java:114)
at io.seata.rm.datasource.DataSourceProxy.init(DataSourceProxy.java:108)
at io.seata.rm.datasource.DataSourceProxy.(DataSourceProxy.java:94)
at io.seata.rm.datasource.DataSourceProxy.(DataSourceProxy.java:79)
at org.apache.shardingsphere.transaction.base.seata.at.SeataATShardingTransactionManager.init(SeataATShardingTransactionManager.java:68)
at org.apache.shardingsphere.transaction.ShardingTransactionManagerEngine.init(ShardingTransactionManagerEngine.java:67)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.context.ShardingRuntimeContext.(ShardingRuntimeContext.java:52)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.datasource.ShardingDataSource.(ShardingDataSource.java:54)
at org.apache.shardingsphere.shardingjdbc.api.ShardingDataSourceFactory.createDataSource(ShardingDataSourceFactory.java:48)
at org.apache.shardingsphere.shardingjdbc.spring.boot.SpringBootConfiguration.shardingDataSource(SpringBootConfiguration.java:103)
at org.apache.shardingsphere.shardingjdbc.spring.boot.SpringBootConfiguration$$EnhancerBySpringCGLIB$$124d36df.CGLIB$shardingDataSource$0()
at org.apache.shardingsphere.shardingjdbc.spring.boot.SpringBootConfiguration$$EnhancerBySpringCGLIB$$124d36df$$FastClassBySpringCGLIB$$34a71f7b.invoke()
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at org.apache.shardingsphere.shardingjdbc.spring.boot.SpringBootConfiguration$$EnhancerBySpringCGLIB$$124d36df.shardingDataSource()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:483)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$159/606593028.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
locked <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$159/606593028.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
locked <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByType(AbstractAutowireCapableBeanFactory.java:1509)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1404)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$159/606593028.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
locked <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$159/606593028.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
locked <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$159/606593028.getObject(Unknown Source)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
locked <0x00000006cac04dd8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
locked <0x00000006ca8ea9f8> (a java.lang.Object)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.aixin.iteaching.school.SchoolApplication.main(SchoolApplication.java:17)
Ⅰ. Issue Description
mysql databases has table over 500; the springboot starting in create sharding-jdbc's datasource enter into deadlock; because sharding-jdbc query database's table metadata for too long time, the seata create RMclient which start thread in delay 60s,
the thread get bean into spring container ,waiting lock in singleObject, but the springboot start-main-thread hold this lock
I have to update source code,update SCHEDULE_DELAY_MILLS from 60s to 180s,and start success io.seata.core.rpc.netty.AbstractNettyRemotingClient.class private static final long SCHEDULE_DELAY_MILLS = 60 * 1000L;
how easier config SCHEDULE_DELAY_MILLS ,do not modify source code?
stack log: Found one Java-level deadlock:
"timeoutChecker_2_1": waiting to lock monitor 0x0000023ff219ab08 (object 0x00000007910e81a8, a java.lang.Class), which is held by "timeoutChecker_1_1" "timeoutChecker_1_1": waiting to lock monitor 0x0000023fedde9908 (object 0x00000006cac04dd8, a java.util.concurrent.ConcurrentHashMap), which is held by "main" "main": waiting to lock monitor 0x0000023ff219ab08 (object 0x00000007910e81a8, a java.lang.Class), which is held by "timeoutChecker_1_1"
Java stack information for the threads listed above:
"timeoutChecker_2_1": at io.seata.discovery.registry.RegistryFactory.getInstance(RegistryFactory.java:42)
Found 1 deadlock.