nsarvi
commented
9 years ago Spring XD admin server/Container or shell throws the same exception stack trace 'Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException' if the Spring XD custom module ( xd.customModule.home ) enabled in the Ambari. If custom module is disabled, i.e either the config parameter xd.customModule.home is removed or empty value, Admin and container starts up fine.
The current workaround is to disable the custom module.
The environment is PHD 3.0 and installed Spring XD using Ambari plugin. This is kerberosed, name node and resource manager HA has enabled.
Looking at the variables in the remote debug mode, the configuration parameter 'dfs.client.failover.proxy.provider.phd3labha' is getting ignored in the Configuration object. Although, all the necessary client HA parameters set in the template file hadoop.properties.j2 and pushed out all the Spring XD nodes.
dfs.nameservices=phd3labha dfs.ha.namenodes.phd3labha=nn1,nn2 dfs.namenode.rpc-address.phd3labha.nn1=nimbus6.domain.com:8020 dfs.namenode.rpc-address.phd3labha.nn2=nimbus7.domain.com:8020 dfs.namenode.http-address.phd3labha.nn1=nimbus6.domain.com:50070 dfs.namenode.http-address.phd3labha.nn2=nimbus7.domain.com:50070 dfs.client.failover.proxy.provider.phd3labha=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
The following are the variables in the Configuration object when custom module enabled. As we can see, the config param dfs.client.failover.proxy.provider.phd3labha is not listed. {fs.defaultFS=hdfs://phd3labha/xd/custom-modules, yarn.resourcemanager.principal=rm/_HOST@PHD.BENTONVILLE.COM, hadoop.security.authentication=kerberos, hadoop.security.authorization=true, dfs.namenode.kerberos.principal=nn/_HOST@PHD.BENTONVILLE.COM, spring.hadoop.userPrincipal=spring-xd@PHD.BENTONVILLE.COM, spring.hadoop.userKeytab=/etc/security/keytabs/spring-xd.keytab spring-xd@PHD.BENTONVILLE.COM}
Here is the complete stack trace
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#3e84a226' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Cannot create filesystem
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
... 20 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot create filesystem
at org.springframework.data.hadoop.fs.HdfsResourceLoader.
trisberg
jvalkeal
It seems like custom module doesn't pickup namenode HA? and still use NameNodeProxies.createNonHAProxy?
015-08-04T18:01:49+0200 1.2.1.RELEASE WARN main annotation.AnnotationConfigApplicationContext - Exception encountered during context initialization - cancelling refresh attempt org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'moduleRegistry' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Cannot create inner bean 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#5b51bacc' of type [org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean] while setting bean property 'delegates' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#5b51bacc' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Cannot create filesystem at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) ~[spring-context-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) ~[spring-context-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:686) [spring-boot-1.2.5.RELEASE.jar:1.2.5.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) [spring-boot-1.2.5.RELEASE.jar:1.2.5.RELEASE] at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) [spring-boot-1.2.5.RELEASE.jar:1.2.5.RELEASE] at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:129) [spring-boot-1.2.5.RELEASE.jar:1.2.5.RELEASE] at org.springframework.xd.dirt.server.admin.AdminServerApplication.run(AdminServerApplication.java:95) [spring-xd-dirt-1.2.1.RELEASE.jar:1.2.1.RELEASE] at org.springframework.xd.dirt.server.admin.AdminServerApplication.main(AdminServerApplication.java:79) [spring-xd-dirt-1.2.1.RELEASE.jar:1.2.1.RELEASE] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#5b51bacc' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Cannot create filesystem at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE] ... 20 common frames omitted Caused by: java.lang.IllegalStateException: Cannot create filesystem at org.springframework.data.hadoop.fs.HdfsResourceLoader.(HdfsResourceLoader.java:112) ~[spring-data-hadoop-core-2.2.0.RELEASE-phd30.jar:2.2.0.RELEASE-phd30]
at org.springframework.data.hadoop.fs.HdfsResourceLoader.(HdfsResourceLoader.java:125) ~[spring-data-hadoop-core-2.2.0.RELEASE-phd30.jar:2.2.0.RELEASE-phd30]
at org.springframework.data.hadoop.fs.HdfsResourceLoader.(HdfsResourceLoader.java:86) ~[spring-data-hadoop-core-2.2.0.RELEASE-phd30.jar:2.2.0.RELEASE-phd30]
at org.springframework.xd.dirt.module.WritableResourceModuleRegistry.afterPropertiesSet(WritableResourceModuleRegistry.java:178) ~[spring-xd-dirt-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean.afterPropertiesSet(CustomModuleRegistryFactoryBean.java:91) ~[spring-xd-dirt-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1633) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570) ~[spring-beans-4.1.7.RELEASE.jar:4.1.7.RELEASE]
... 23 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: homelab
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) ~[hadoop-common-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312) ~[hadoop-hdfs-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178) ~[hadoop-hdfs-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:665) ~[hadoop-hdfs-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:601) ~[hadoop-hdfs-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148) ~[hadoop-hdfs-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2619) ~[hadoop-common-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) ~[hadoop-common-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2653) ~[hadoop-common-2.6.0.3.0.0.0-249.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2635) ~[hadoop-common-2.6.0.3.0.0.0-249.jar:na]
"admin-565179.log" 111L, 18394C