search

jetoile / hadoop-unit

Hadoop-Unit is a project which allow testing projects which need hadoop ecosysteme like kafka, solr, hdfs, hive, hbase, ...
https://blog.jetoile.fr/hadoop-unit/
Apache License 2.0
51 stars 15 forks source link

Build failure - Yarn test whoami returns nothing #79

Closed yoyocm closed 6 years ago

yoyocm commented 6 years ago

Yarn tests failed when it tests 'whoami' which returned nothing.

2017-09-24 20:33:09 INFO RMAppManager$ApplicationSummary:194 - appId=application_1506277984109_0001,name=simple-yarn-app,user=yoann,queue=default,state=FAILED,trackingUrl=http://snecmaais.home:37004/cluster/app/application_1506277984109_0001,appMasterHost=N/A,startTime=1506277987106,finishTime=1506277989136,finalStatus=FAILED,memorySeconds=632,vcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=YARN java.io.IOException: Is a directory at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.file.Files.read(Files.java:3105) at java.nio.file.Files.readAllBytes(Files.java:3158) at fr.jetoile.hadoopunit.component.YarnBootstrapTest.getStdoutContents(YarnBootstrapTest.java:72) at fr.jetoile.hadoopunit.component.YarnBootstrapTest.testYarnLocalClusterIntegrationTest(YarnBootstrapTest.java:65) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) 2017-09-24 20:33:09 INFO YarnBootstrap:162 - fr.jetoile.hadoopunit.component.YarnBootstrap is stopping 2017-09-24 20:33:09 INFO YarnLocalCluster:243 - YARN: Stopping MiniYarnCluster 2017-09-24 20:33:09 INFO log:67 - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@snecmaais.home:0 2017-09-24 20:33:09 INFO ContainerManagerImpl:519 - Applications still running : [application_1506277984109_0001] 2017-09-24 20:33:09 INFO ContainerManagerImpl:533 - Waiting for Applications to be Finished 2017-09-24 20:33:09 INFO ApplicationImpl:464 - Application application_1506277984109_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2017-09-24 20:33:09 INFO AuxServices:215 - Got event APPLICATION_STOP for appId application_1506277984109_0001 2017-09-24 20:33:09 INFO ApplicationImpl:464 - Application application_1506277984109_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2017-09-24 20:33:09 INFO NonAggregatingLogHandler:167 - Scheduling Log Deletion for application: application_1506277984109_0001, with delay of 10800 seconds 2017-09-24 20:33:13 INFO ContainerManagerImpl:550 - Done waiting for Applications to be Finished. Still alive: [application_1506277984109_0001] 2017-09-24 20:33:13 INFO Server:2752 - Stopping server on 62641 2017-09-24 20:33:13 INFO Server:932 - Stopping IPC Server listener on 62641 2017-09-24 20:33:13 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 INFO ContainersMonitorImpl:212 - ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled. 2017-09-24 20:33:23 INFO Server:2752 - Stopping server on 62642 2017-09-24 20:33:23 INFO Server:932 - Stopping IPC Server listener on 62642 2017-09-24 20:33:23 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 INFO ResourceLocalizationService:883 - Public cache exiting 2017-09-24 20:33:23 ERROR AbstractDelegationTokenSecretManager:659 - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2017-09-24 20:33:23 INFO log:67 - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37004 2017-09-24 20:33:23 INFO Server:2752 - Stopping server on 37001 2017-09-24 20:33:23 INFO Server:932 - Stopping IPC Server listener on 37001 2017-09-24 20:33:23 INFO Server:2752 - Stopping server on 8033 2017-09-24 20:33:23 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 INFO Server:932 - Stopping IPC Server listener on 8033 2017-09-24 20:33:23 INFO ResourceManager:1078 - Transitioning to standby state 2017-09-24 20:33:23 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 WARN ApplicationMasterLauncher:122 - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 2017-09-24 20:33:23 INFO Server:2752 - Stopping server on 37002 2017-09-24 20:33:23 INFO Server:932 - Stopping IPC Server listener on 37002 2017-09-24 20:33:23 INFO Server:2752 - Stopping server on 37003 2017-09-24 20:33:23 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 INFO Server:932 - Stopping IPC Server listener on 37003 2017-09-24 20:33:23 INFO AbstractLivelinessMonitor:139 - NMLivelinessMonitor thread interrupted 2017-09-24 20:33:23 ERROR ResourceManager:708 - Returning, interrupted : java.lang.InterruptedException 2017-09-24 20:33:23 INFO Server:1069 - Stopping IPC Server Responder 2017-09-24 20:33:23 INFO AsyncDispatcher:142 - AsyncDispatcher is draining to stop, igonring any new events. 2017-09-24 20:33:23 INFO AbstractLivelinessMonitor:139 - org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted 2017-09-24 20:33:23 INFO AbstractLivelinessMonitor:139 - AMLivelinessMonitor thread interrupted 2017-09-24 20:33:23 ERROR AbstractDelegationTokenSecretManager:659 - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2017-09-24 20:33:23 INFO AbstractLivelinessMonitor:139 - AMLivelinessMonitor thread interrupted 2017-09-24 20:33:23 INFO AsyncDispatcher:142 - AsyncDispatcher is draining to stop, igonring any new events. 2017-09-24 20:33:23 INFO ResourceManager:1085 - Transitioned to standby state 2017-09-24 20:33:23 INFO FileUtils:21 - FILEUTILS: Deleting contents of directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster/com.github.sakserv.minicluster.impl.YarnLocalCluster-localDir-nm-0_0/filecache 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster/com.github.sakserv.minicluster.impl.YarnLocalCluster-localDir-nm-0_0/nmPrivate 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster/com.github.sakserv.minicluster.impl.YarnLocalCluster-localDir-nm-0_0/usercache 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster/com.github.sakserv.minicluster.impl.YarnLocalCluster-localDir-nm-0_0 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster/com.github.sakserv.minicluster.impl.YarnLocalCluster-logDir-nm-0_0 2017-09-24 20:33:23 INFO FileUtils:36 - Removing directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster 2017-09-24 20:33:23 INFO FileUtils:21 - FILEUTILS: Deleting contents of directory: /Users/yoann/Downloads/hadoop-unit-master/hadoop-unit-yarn/./target/com.github.sakserv.minicluster.impl.YarnLocalCluster 2017-09-24 20:33:23 ERROR FileUtils:41 - FILEUTILS: Unable to remove ./target/com.github.sakserv.minicluster.impl.YarnLocalCluster 2017-09-24 20:33:23 INFO YarnBootstrap:171 - fr.jetoile.hadoopunit.component.YarnBootstrap is stopped Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.148 sec <<< FAILURE! - in fr.jetoile.hadoopunit.component.YarnBootstrapTest testYarnLocalClusterIntegrationTest(fr.jetoile.hadoopunit.component.YarnBootstrapTest) Time elapsed: 2.614 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<[yoann]> but was:<[]> at fr.jetoile.hadoopunit.component.YarnBootstrapTest.testYarnLocalClusterIntegrationTest(YarnBootstrapTest.java:65)

2017-09-24 20:33:23 INFO HadoopBootstrap:61 - All services are going to be stopped

jetoile commented 6 years ago

hum... I do not reproduce this behavior... It seems that you are using a mac but I only have a Linux and a Windows VM.

Do you need Yarn and/or what is your usecase for using Hadoop-unit?

jetoile commented 6 years ago

Hello,

I just reproduced your issue. It seems that you have to set rights to 755.

https://stackoverflow.com/questions/23903113/mapreduce-error-usergroupinformation-priviledgedactionexception

This is quite troublesome just for tests but if you do not need yarn, just do a mvn install -DskipTests

jetoile commented 6 years ago

I close this issue.

Feel free to reopen if needed