YARN incompatibility with 2.3.GlusterFS stack and ambari 2.2.1.0

Hi,

We are trying to install Hadoop cluster using 2.3.GlusterFS stack through Ambari 2.2.1.0. All provisioning is done with Ambari blueprints like : Blueprint { "configurations" : [ { "core-site": { "hadoop.proxyuser.hcat.groups" : "*", "hadoop.proxyuser.hcat.hosts" : "*", "hadoop.proxyuser.hue.groups" : "*", "hadoop.proxyuser.hue.hosts" : "*", "hadoop.security.authentication" : "simple", "fs.AbstractFileSystem.glusterfs.impl" : "org.apache.hadoop.fs.local.GlusterFs", "fs.trash.interval" : "360", "fs.glusterfs.impl" : "org.apache.hadoop.fs.glusterfs.GlusterFileSystem", "hadoop.security.authorization" : "false", "net.topology.script.file.name" : "/etc/hadoop/conf/topology_script.py" } }, { "ams-hbase-env" : { "regionserver_xmn_size" : "384m", "hbase_regionserver_heapsize" : "1024m", "hbase_master_heapsize" : "1024m", "hbase_master_xmn_size" : "384m" } }, { "ams-env" : { "metrics_collector_heapsize" : "1024m" } }, { "hadoop-env" : { "dtnode_heapsize" : "1024m", "namenode_heapsize" : "2048m", "namenode_opt_maxnewsize" : "384m", "namenode_opt_newsize" : "384m", "namenode_host" : "master01", "snamenode_host" : "master01", "glusterfs_user" : "root", "hdfs_user" : "hdfs", "hdfs_log_dir_prefix" : "/var/log/hadoop" } }, { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/hdfs/data", "dfs.datanode.balance.bandwidthPerSec" : "12500000", "dfs.datanode.max.transfer.threads": "4096", "dfs.datanode.failed.volumes.tolerated" : "1", "dfs.replication" : "true" } }, { "spark-defaults" : { "spark.executor.instances" : "2", "spark.executor.memory" : "7808m", "spark.driver.memory" : "3712m", "spark.yarn.am.memory" : "3712m", "spark.yarn.executor.memoryOverhead" : "384", "spark.yarn.driver.memoryOverhead" : "384", "spark.yarn.am.memoryOverhead" : "384" } }, { "mapred-site" : { "mapreduce.map.java.opts" : "-Xmx1638m", "mapreduce.map.memory.mb" : "2048", "mapreduce.reduce.java.opts" : "-Xmx1638m", "mapreduce.reduce.memory.mb" : "2048", "mapreduce.task.io.sort.mb" : "768", "yarn.app.mapreduce.am.command-opts" : "-Xmx1638m -Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.resource.mb" : "2048" } }, { "tez-site" : { "tez.am.resource.memory.mb" : "2048", "tez.task.resource.memory.mb" : "2048" } }, { "spark-defaults" : { "spark.executor.instances" : "1", "spark.executor.memory" : "3712m", "spark.driver.memory" : "1664m", "spark.yarn.am.memory" : "1664m", "spark.yarn.executor.memoryOverhead" : "384", "spark.yarn.driver.memoryOverhead" : "384", "spark.yarn.am.memoryOverhead" : "384" } }, { "storm-site" : { "logviewer.port" : "8005" } }, { "oozie-site" : { "oozie.service.ProxyUserService.proxyuser.hue.groups" : "*", "oozie.service.ProxyUserService.proxyuser.hue.hosts" : "*" } }, { "webhcat-site" : { "webhcat.proxyuser.hue.groups" : "*", "webhcat.proxyuser.hue.hosts" : "*" } }, { "hive-site" : { "hive.tez.container.size" : "-1", "hive.tez.java.opts": "-1", "fs.file.impl.disable.cache" : "true", "fs.hdfs.impl.disable.cache" : "true", "javax.jdo.option.ConnectionPassword" : "my-pw" } } ], "host_groups" : [ { "name" : "slavenode_simple", "configurations" : [ ], "components" : [ { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "OOZIE_CLIENT" }, { "name" : "HIVE_CLIENT" }, { "name" : "GLUSTERFS_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "TEZ_CLIENT" }, { "name" : "SPARK_CLIENT" }, { "name" : "NODEMANAGER" }, { "name" : "METRICS_MONITOR" } ], "cardinality" : "2" }, { "name" : "masternode_1", "configurations" : [ ], "components" : [ { "name" : "NODEMANAGER" }, { "name" : "SPARK_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "GLUSTERFS_CLIENT" }, { "name" : "METRICS_MONITOR" }, { "name" : "TEZ_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "AMBARI_SERVER" }, { "name" : "SPARK_JOBHISTORYSERVER" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "METRICS_COLLECTOR" }, { "name" : "RESOURCEMANAGER" }, { "name" : "WEBHCAT_SERVER" }, { "name" : "OOZIE_SERVER" }, { "name" : "HIVE_METASTORE" }, { "name" : "HIVE_SERVER" } ], "cardinality" : "1" } ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.3.GlusterFS" } }

Template used for blueprint to be provided to : { "blueprint": "cluster_blueprint", "default_password": "my-pw", "host_groups": [ { "name" : "masternode_1", "hosts" : [ { "fqdn": "master01.domain.com" } ] } ] }

Here is the full stack : 23 Mar 2016 11:05:51,256 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type zookeeper-env 23 Mar 2016 11:05:51,280 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type zookeeper-log4j 23 Mar 2016 11:05:51,323 INFO [qtp-ambari-client-23] ClusterConfigurationRequest:358 - Sending cluster config update request for service = YARN 23 Mar 2016 11:05:51,324 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1353 - Received a updateCluster request, clusterId=null, clusterName=test-cluster2, securityType=null, request={ clusterName=test-cluster2, clusterId=null, provisioningState=null, securityType=null, stackVersion=null, desired_scv=null, hosts=[] } 23 Mar 2016 11:05:51,324 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type hdfs-site 23 Mar 2016 11:05:51,351 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type capacity-scheduler 23 Mar 2016 11:05:51,378 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-env 23 Mar 2016 11:05:51,411 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-site 23 Mar 2016 11:05:51,490 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-log4j 23 Mar 2016 11:05:51,567 ERROR [qtp-ambari-client-23] ClusterImpl:2635 - Updating configs for multiple services by a single API request isn't supported, config version not created 23 Mar 2016 11:05:51,572 ERROR [qtp-ambari-client-23] BaseManagementHandler:66 - Caught a runtime exception while attempting to create a resource: Failed to set configurations on cluster: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported java.lang.RuntimeException: Failed to set configurations on cluster: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.topology.AmbariContext.setConfigurationOnCluster(AmbariContext.java:390) at org.apache.ambari.server.topology.ClusterConfigurationRequest.setConfigurationsOnCluster(ClusterConfigurationRequest.java:359) at org.apache.ambari.server.topology.ClusterConfigurationRequest.setConfigurationsOnCluster(ClusterConfigurationRequest.java:279) at org.apache.ambari.server.topology.ClusterConfigurationRequest.<init>(ClusterConfigurationRequest.java:78) at org.apache.ambari.server.topology.ClusterConfigurationRequest.<init>(ClusterConfigurationRequest.java:83) at org.apache.ambari.server.topology.TopologyManager.provisionCluster(TopologyManager.java:191) at org.apache.ambari.server.controller.internal.ClusterResourceProvider.processBlueprintCreate(ClusterResourceProvider.java:517) at org.apache.ambari.server.controller.internal.ClusterResourceProvider.createResources(ClusterResourceProvider.java:174) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289) at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76) at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36) at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72) at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75) at org.apache.ambari.server.api.services.ClusterService.createCluster(ClusterService.java:131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:196) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:216) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:205) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.utils.RetryHelper.executeWithRetry(RetryHelper.java:110) at org.apache.ambari.server.topology.AmbariContext.setConfigurationOnCluster(AmbariContext.java:381) ... 96 more Caused by: java.lang.IllegalArgumentException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.state.cluster.ClusterImpl.applyConfigs(ClusterImpl.java:2634) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68) at org.apache.ambari.server.state.cluster.ClusterImpl.addDesiredConfig(ClusterImpl.java:2172) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1485) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1337) at org.apache.ambari.server.topology.AmbariContext$5.call(AmbariContext.java:384) at org.apache.ambari.server.utils.RetryHelper.executeWithRetry(RetryHelper.java:95) ... 97 more

When we change GLUSTERFS_CLIENT to HDFS_CLIENT and stack to be 2.4 (HDFS), everything is fine.

Do you have any idea please ? Is there any incompatibility ? Thanks

gluster / glusterfs-hadoop

YARN incompatibility with 2.3.GlusterFS stack and ambari 2.2.1.0 #134