yahoo / storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.
Other
417 stars 161 forks source link

Hadoop 2.2.0 support needed #56

Open anfeng opened 10 years ago

anfeng commented 10 years ago

I am using Apache Hadoop-2.2.0 with storm-yarn from wget https://github.com/yahoo/storm-yarn/archive/master.zip.

When running the storm-yarn using mvn package the tests are failing with below error:

00:34:28.901 [main] INFO com.yahoo.storm.yarn.TestConfig - storm_home: lib/storm-0.9.0-wip21 00:34:28.903 [main] INFO com.yahoo.storm.yarn.TestIntegration - Will be using storm found on PATH at lib/storm-0.9.0-wip21 00:34:29.083 [main] INFO com.yahoo.storm.yarn.TestIntegration - bin/storm-yarn launch ./conf/storm.yaml --stormZip lib/storm.zip --appname storm-on-yarn-test --output target/appId.txt SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/cloud/Manisha/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/cloud/nirmal/storm-yarn-master/lib/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 00:34:31.206 [mainhread] org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)evel org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)ogger{36} - Unable to load native-hadoop library for your platform... using builtin-java classes where applicablesg 00:34:32.465 [Socket Reader #1 for port 37121] INFO org.apache.hadoop.ipc.Server - IPC Server listener on 37121: readAndProcess from client 192.168.41.106 threw exception [org.apache.hadoop.ipc.RpcServerException: Connection context not established] Exception in thread "main" org.apache.hadoop.ipc.RpcServerException: Connection context not established at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy8.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:127) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:135) at com.yahoo.storm.yarn.StormOnYarn.launchApp(StormOnYarn.java:140) at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:372) at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:55) at com.yahoo.storm.yarn.Client.execute(Client.java:139) at com.yahoo.storm.yarn.Client.main(Client.java:145) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException): Connection context not established at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy7.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:167) ... 14 more 00:34:47.523 [main] ERROR com.yahoo.storm.yarn.TestIntegration - setup failure java.io.FileNotFoundException: target/appId.txt (No such file or directory) at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_45] at java.io.FileInputStream.(FileInputStream.java:146) ~[na:1.7.0_45] at java.io.FileInputStream.(FileInputStream.java:101) ~[na:1.7.0_45] at java.io.FileReader.(FileReader.java:58) ~[na:1.7.0_45] at com.yahoo.storm.yarn.TestIntegration.setup(TestIntegration.java:101) ~[test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) [junit-4.10.jar:na] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit-4.10.jar:na] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) [junit-4.10.jar:na] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) [junit-4.10.jar:na] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) [junit-4.10.jar:na] at org.junit.runners.ParentRunner.run(ParentRunner.java:300) [junit-4.10.jar:na] at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) [surefire-junit4-2.10.jar:2.10] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) [surefire-junit4-2.10.jar:2.10] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) [surefire-junit4-2.10.jar:2.10] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45] at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) [surefire-api-2.10.jar:2.10] at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) [surefire-booter-2.10.jar:2.10] at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) [surefire-booter-2.10.jar:2.10] at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) [surefire-booter-2.10.jar:2.10] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) [surefire-booter-2.10.jar:2.10] 00:34:47.524 [main] INFO com.yahoo.storm.yarn.TestIntegration - java.lang.NullPointerException 00:34:47.525 [main] INFO c.yahoo.storm.yarn.EmbeddedZKServer - shutdown embedded zookeeper server with port 60000 00:34:47.527 [main] INFO com.yahoo.storm.yarn.TestIntegration - shutdown MiniYarn cluster 00:34:47.546 [main] INFO org.mortbay.log - Stopped SelectChannelConnector@Impetus-942.impetus.co.in:0 00:34:47.648 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 35805 00:34:47.652 [IPC Server listener on 35805] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 35805 00:34:47.655 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.657 [main] INFO o.a.h.y.s.n.c.m.ContainersMonitorImpl - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread 00:34:47.658 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 60910 00:34:47.661 [IPC Server listener on 60910] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 60910 00:34:47.662 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.664 [Public Localizer] INFO o.a.h.y.s.n.c.l.ResourceLocalizationService - Public cache exiting 00:34:47.678 [main] INFO org.mortbay.log - Stopped SelectChannelConnector@Impetus-942.impetus.co.in:0 00:34:47.780 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 43639 00:34:47.782 [IPC Server listener on 43639] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 43639 00:34:47.782 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.784 [main] INFO o.a.h.y.s.n.c.m.ContainersMonitorImpl - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread 00:34:47.785 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 37231 00:34:47.786 [IPC Server listener on 37231] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 37231 00:34:47.787 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.788 [Public Localizer] INFO o.a.h.y.s.n.c.l.ResourceLocalizationService - Public cache exiting 00:34:47.804 [main] INFO org.mortbay.log - Stopped SelectChannelConnector@Impetus-942.impetus.co.in:0 00:34:47.807 [Thread[Thread-13,5,main]] ERROR o.a.h.s.t.d.AbstractDelegationTokenSecretManager - InterruptedExcpetion recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep interrupted 00:34:47.809 [main] INFO o.a.h.m.impl.MetricsSystemImpl - Stopping NodeManager metrics system... 00:34:47.814 [main] INFO o.a.h.m.impl.MetricsSystemImpl - NodeManager metrics system stopped. 00:34:47.815 [main] INFO o.a.h.m.impl.MetricsSystemImpl - NodeManager metrics system shutdown complete. 00:34:47.816 [ApplicationMaster Launcher] WARN o.a.h.y.s.r.a.ApplicationMasterLauncher - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 00:34:47.816 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 33651 00:34:47.819 [IPC Server listener on 33651] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 33651 00:34:47.819 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.819 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 37121 00:34:47.825 [IPC Server listener on 37121] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 37121 00:34:47.825 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.825 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 33039 00:34:47.832 [IPC Server listener on 33039] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 33039 00:34:47.834 [main] INFO org.apache.hadoop.ipc.Server - Stopping server on 54908 00:34:47.834 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.849 [IPC Server listener on 54908] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server listener on 54908 00:34:47.849 [IPC Server Responder] INFO org.apache.hadoop.ipc.Server - Stopping IPC Server Responder 00:34:47.849 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - NMLivelinessMonitor thread interrupted 00:34:47.850 [ResourceManager Event Processor] ERROR o.a.h.y.s.r.ResourceManager - Returning, interrupted : java.lang.InterruptedException 00:34:47.853 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - AMLivelinessMonitor thread interrupted 00:34:47.854 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - AMLivelinessMonitor thread interrupted 00:34:47.854 [Ping Checker] INFO o.a.h.y.u.AbstractLivelinessMonitor - org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.228 sec <<< FAILURE!

anfeng commented 10 years ago

Pull request #57 created