search

spring-attic / spring-hadoop-samples

Spring Hadoop Samples
Apache License 2.0
492 stars 466 forks source link

Connection Refused in Mapreduce sample #33

Closed gmunumel closed 6 years ago

gmunumel commented 6 years ago

Hello I am getting the following error when trying to run the Mapreduce sample:

19:47:07,866  WARN t.support.ClassPathXmlApplicationContext: 487 - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'runner': Invocation of init method failed; nested exception is org.springframework.scripting.ScriptCompilationException: Could not compile script [class path resource [copy-files.groovy]]: Execution failure; nested exception is javax.script.ScriptException: javax.script.ScriptException: org.springframework.data.hadoop.HadoopException: Cannot test resource /user/gutenberg/input/word/;Call From Gabriels-MacBook-Pro.local/192.168.1.101 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)

I am running Hadoop version 3 in mac OS X. 

$HADOOP_HOME = /usr/local/Cellar/hadoop
$JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
jps
93667 DataNode
93810 SecondaryNameNode
94004 ResourceManager
94884 Jps
94102 NodeManager
93562 NameNode

Any idea?

gmunumel commented 6 years ago

Changed my core-site.xml solved the issue. 

<configuration>
   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://localhost:8020</value>
   </property>
</configuration>

Now I am facing a problem with:

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hadoop-yarn/staging/gabrielmunumel/.staging/job_1525387543266_0001/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)

But this is another issue I guess.

trisberg commented 6 years ago

Looks like your cluster has some issues with the datanodes. Also, we have not used the framework with Hadoop 3 and have no plans on evolving it further, see notice at https://projects.spring.io/spring-hadoop/

gmunumel commented 6 years ago

Realized that. I have downgraded hadoop to version 2.6.

To solved the last issue I have removed my tmp dir and recreated with hdfs namenode -format. It is working now. The only issue is that the job hangs.

I will close the issue because seems not related to spring.