search

alibaba / nacos

an easy-to-use dynamic service discovery, configuration and service management platform for building cloud native applications.
https://nacos.io
Apache License 2.0
30.36k stars 12.86k forks source link

nacos从2.2.2升级到2.2.3，然后回滚到2.2.2，集群异常，Fail to init RocksDBLogStorage,是什么问题 #12881

Open HadwinLing opened 5 days ago

HadwinLing commented 5 days ago

问题描述 nacos 从2.2.2 滚动升级到2.2.3 ，升级方案是：先备份2.2.2 整个目录为2.2.2_copy，然后替换jar包。从2.2.3 回滚到2.2.2 ,回滚方案：删除2.2.3整个目录，将先前备份的2.2.2_copy 重新修改nacos 目录。在回滚过程中存在间歇性回滚失败，在alipay-jrafe.log 报这个异常：Fail to init RocksDBLogStorage 具体日志alipay-jrafe.log： 2024-11-21 21:02:33,828 ERROR Fail to init RocksDBLogStorage, path=/app/deploy/nacos/data/protocol/raft/naming_persistent_service/log.

org.rocksdb.RocksDBException: Corruption: IO error: No such file or directory: While open a file for random read: /app/deploy/nacos/data/protocol/raft/naming_persistent_service/log/000010.ldb: No such file or directory in file /app/deploy/nacos/data/protocol/raft/naming_persistent_service/log/MANIFEST-000013 at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:307) at com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage.openDB(RocksDBLogStorage.java:312) at com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage.initAndLoad(RocksDBLogStorage.java:230) at com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage.init(RocksDBLogStorage.java:209) at com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage.init(RocksDBLogStorage.java:72) at com.alipay.sofa.jraft.storage.impl.LogManagerImpl.init(LogManagerImpl.java:184) at com.alipay.sofa.jraft.storage.impl.LogManagerImpl.init(LogManagerImpl.java:76) at com.alipay.sofa.jraft.core.NodeImpl.initLogStorage(NodeImpl.java:591) at com.alipay.sofa.jraft.core.NodeImpl.init(NodeImpl.java:1005) at com.alipay.sofa.jraft.core.NodeImpl.init(NodeImpl.java:141) at com.alipay.sofa.jraft.RaftServiceFactory.createAndInitRaftNode(RaftServiceFactory.java:47) at com.alipay.sofa.jraft.RaftGroupService.start(RaftGroupService.java:129) at com.alibaba.nacos.core.distributed.raft.JRaftServer.createMultiRaftGroup(JRaftServer.java:260) at com.alibaba.nacos.core.distributed.raft.JRaftProtocol.addRequestProcessors(JRaftProtocol.java:163) at com.alibaba.nacos.naming.consistency.persistent.impl.PersistentServiceProcessor.afterConstruct(PersistentServiceProcessor.java:76) at com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl.createPersistentServiceProcessor(PersistentConsistencyServiceDelegateImpl.java:88) at com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl.(PersistentConsistencyServiceDelegateImpl.java:45) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeanByName(AbstractAutowireCapableBeanFactory.java:479) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:550) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:520) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:673) at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:228) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:329) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:362) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:334) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.trafficReviseFilter() at com.alibaba.nacos.naming.web.NamingConfig.trafficReviseFilterRegistration(NamingConfig.java:64) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.CGLIB$trafficReviseFilterRegistration$1() at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2$$FastClassBySpringCGLIB$$ea9bed69.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.trafficReviseFilterRegistration() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:203) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:97) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:86) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:260) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5211) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486) at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.(TomcatWebServer.java:104) at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:479) at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:423) at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at com.alibaba.nacos.Nacos.main(Nacos.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467) 2024-11-21 21:02:33,828 ERROR Fail to init logStorage

2024-11-21 21:02:33,828 ERROR Node <naming_persistent_service/10.208.181.95:7848> initLogStorage failed.

2024-11-21 21:02:33,901 INFO ThreadPool is terminated: JRaft-RPC-Processor, com.alipay.sofa.jraft.util.MetricThreadPoolExecutor@536f2a7e[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0].

start-out.log 日志： org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:423) at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at com.alibaba.nacos.Nacos.main(Nacos.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467) Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142) at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.(TomcatWebServer.java:104) at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:479) at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) ... 16 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'trafficReviseFilterRegistration' defined in class path resource [com/alibaba/nacos/naming/web/NamingConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.web.servlet.FilterRegistrationBean]: Factory method 'trafficReviseFilterRegistration' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'trafficReviseFilter': Unsatisfied dependency expressed through field 'serverStatusManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverStatusManager': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistentConsistencyServiceDelegate' defined in URL [jar:file:/app/deploy/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-naming-2.2.2.jar!/com/alibaba/nacos/naming/consistency/persistent/PersistentConsistencyServiceDelegateImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:203) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:97) at org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:86) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:260) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5211) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486) at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) ... 21 common frames omitted Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.web.servlet.FilterRegistrationBean]: Factory method 'trafficReviseFilterRegistration' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'trafficReviseFilter': Unsatisfied dependency expressed through field 'serverStatusManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverStatusManager': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistentConsistencyServiceDelegate' defined in URL [jar:file:/app/deploy/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-naming-2.2.2.jar!/com/alibaba/nacos/naming/consistency/persistent/PersistentConsistencyServiceDelegateImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ... 61 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'trafficReviseFilter': Unsatisfied dependency expressed through field 'serverStatusManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverStatusManager': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistentConsistencyServiceDelegate' defined in URL [jar:file:/app/deploy/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-naming-2.2.2.jar!/com/alibaba/nacos/naming/consistency/persistent/PersistentConsistencyServiceDelegateImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:362) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:334) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.trafficReviseFilter() at com.alibaba.nacos.naming.web.NamingConfig.trafficReviseFilterRegistration(NamingConfig.java:64) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.CGLIB$trafficReviseFilterRegistration$1() at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2$$FastClassBySpringCGLIB$$ea9bed69.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at com.alibaba.nacos.naming.web.NamingConfig$$EnhancerBySpringCGLIB$$54f8ccb2.trafficReviseFilterRegistration() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 62 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverStatusManager': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistentConsistencyServiceDelegate' defined in URL [jar:file:/app/deploy/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-naming-2.2.2.jar!/com/alibaba/nacos/naming/consistency/persistent/PersistentConsistencyServiceDelegateImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:332) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657) ... 86 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistentConsistencyServiceDelegate' defined in URL [jar:file:/app/deploy/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-naming-2.2.2.jar!/com/alibaba/nacos/naming/consistency/persistent/PersistentConsistencyServiceDelegateImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeanByName(AbstractAutowireCapableBeanFactory.java:479) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:550) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:520) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:673) at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:228) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:329) ... 97 common frames omitted Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ... 113 common frames omitted Caused by: java.lang.IllegalStateException: Fail to init node, please see the logs to find the reason. at com.alipay.sofa.jraft.RaftServiceFactory.createAndInitRaftNode(RaftServiceFactory.java:48) at com.alipay.sofa.jraft.RaftGroupService.start(RaftGroupService.java:129) at com.alibaba.nacos.core.distributed.raft.JRaftServer.createMultiRaftGroup(JRaftServer.java:260) at com.alibaba.nacos.core.distributed.raft.JRaftProtocol.addRequestProcessors(JRaftProtocol.java:163) at com.alibaba.nacos.naming.consistency.persistent.impl.PersistentServiceProcessor.afterConstruct(PersistentServiceProcessor.java:76) at com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl.createPersistentServiceProcessor(PersistentConsistencyServiceDelegateImpl.java:88) at com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl.(PersistentConsistencyServiceDelegateImpl.java:45) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ... 115 common frames omitted 2024-11-21 21:02:33,925 WARN [WatchFileCenter] start close

2024-11-21 21:02:33,926 WARN [WatchFileCenter] start to shutdown this watcher which is watch : /app/deploy/nacos/conf

2024-11-21 21:02:33,930 WARN [WatchFileCenter] already closed

2024-11-21 21:02:33,930 WARN [NotifyCenter] Start destroying Publisher

2024-11-21 21:02:33,931 WARN [NotifyCenter] Destruction of the end

2024-11-21 21:02:33,931 ERROR Nacos failed to start, please see /app/deploy/nacos/logs/nacos.log for more details.

2024-11-21 21:02:34,109 WARN [ThreadPoolManager] Start destroying ThreadPool

2024-11-21 21:02:34,109 WARN [ThreadPoolManager] Destruction of the end

lucky8987 commented 5 days ago

@HadwinLing 我们升级2.4.2 过程中也遇到jraft日志文件相关报错，你可以尝试删除掉：data/protocol 目录下所有文件，注意每次操作一台，且删除后控制台观察服务节点是否正常。

HadwinLing commented 5 days ago

@HadwinLing 我们升级2.4.2 过程中也遇到jraft日志文件相关报错，你可以尝试删除掉：data/protocol 目录下所有文件，注意每次操作一台，且删除后控制台观察服务节点是否正常。

我这边升级和回滚都是每次操作一台的。删除data/protocol 目录是可以的，但感觉这不是最优解？看官网说说/data/protocol 目录主要存放的是与Raft协议相关的数据和日志文件，这些文件是用于Nacos集群中的数据一致性、节点间通信以及故障恢复的。这样直接删除好吗？想知道为啥出现会这个问题，如何规避。

KomachiSion commented 4 days ago

升级会导致rocksdb版本随之升级，再降级时， rocksdb可能不兼容降级，所以会导致init失败的问题， 确实只能删除data/protocol来销毁不兼容的rocksdb数据来解决。

另外为什么需要降级？

HadwinLing commented 4 days ago

我这边是验证升级失败后自动回滚降级，我看nacos 2.2.2 和nacos 2.2.3 都是用jraft-core 的版本是一致的，rocksdb 是嵌套在jraft 里面的，那版本也是一致的。而且这个init 失败不是必现的，我正常对一个集群执行升级回滚是ok，后面再回滚就突然就出现init 这个问题了。所以想了解一下是啥问题造成的

Melod-YI commented 1 day ago

我这边是验证升级失败后自动回滚降级，我看nacos 2.2.2 和nacos 2.2.3 都是用jraft-core 的版本是一致的，rocksdb 是嵌套在jraft 里面的，那版本也是一致的。而且这个init 失败不是必现的，我正常对一个集群执行升级回滚是ok，后面再回滚就突然就出现init 这个问题了。所以想了解一下是啥问题造成的

是不是考虑升级前先把data目录备份出来，然后升级失败的话用备份的data来拉起旧版本？

HadwinLing commented 1 day ago

我这边是验证升级失败后自动回滚降级，我看nacos 2.2.2 和nacos 2.2.3 都是用jraft-core 的版本是一致的，rocksdb 是嵌套在jraft 里面的，那版本也是一致的。而且这个init 失败不是必现的，我正常对一个集群执行升级回滚是ok，后面再回滚就突然就出现init 这个问题了。所以想了解一下是啥问题造成的

是不是考虑升级前先把data目录备份出来，然后升级失败的话用备份的data来拉起旧版本？

我这边也是这样的，升级前把整个nacos 目录copy一份出来，升级失败后，将copy 的那份重新重命名为nacos，这样子升级。是不是因为我这边反复的升级回滚，来回再2.2.2和2.2.3 之间升级回滚，导致/data 的数据不兼容呀？