search

mustafatok / SEBC

1 stars 1 forks source link

Storage labs #3

Closed mustafatok closed 7 years ago

mustafatok commented 7 years ago

Cloudera Manager instance : http://ec2-35-156-89-235.eu-central-1.compute.amazonaws.com:7180/

mustafatok commented 7 years ago

Other than distcp, everything is done! I spent too much time on making distcp work, but didn't manage it.

I tried giving private-ip, private-dns, public-ip and public-dns, but didn't work!

I also tried changing hosts file, but couldn't fix the issue with distcp.

My cluster is in eu and the other cluster is in us. Here is the output of command.

[msmarnaoui@ip-172-31-26-193 ec2-user]$ hadoop distcp hftp://ec2-52-10-0-80.us-west-2.compute.amazonaws.com:50070/user/msmarnaoui/msmarnaoui/part-m-00000 /user/msmarnaoui/target 17/03/07 12:05:25 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hftp://ec2-52-10-0-80.us-west-2.compute.amazonaws.com:50070/user/msmarnaoui/msmarnaoui/part-m-00000], targetPath=/user/msmarnaoui/target, targetPathExists=false, filtersFile='null'} 17/03/07 12:05:26 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-26-193.eu-central-1.compute.internal/172.31.26.193:8032 17/03/07 12:05:27 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0 17/03/07 12:05:27 INFO tools.SimpleCopyListing: Build file listing completed. 17/03/07 12:05:27 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 17/03/07 12:05:27 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 17/03/07 12:05:27 INFO tools.DistCp: Number of paths in the copy list: 1 17/03/07 12:05:27 INFO tools.DistCp: Number of paths in the copy list: 1 17/03/07 12:05:27 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-26-193.eu-central-1.compute.internal/172.31.26.193:8032 17/03/07 12:05:28 INFO mapreduce.JobSubmitter: number of splits:1 17/03/07 12:05:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1488905664666_0006 17/03/07 12:05:28 INFO impl.YarnClientImpl: Submitted application application_1488905664666_0006 17/03/07 12:05:28 INFO mapreduce.Job: The url to track the job: http://ip-172-31-26-193.eu-central-1.compute.internal:8088/proxy/application_1488905664666_0006/ 17/03/07 12:05:28 INFO tools.DistCp: DistCp job-id: job_1488905664666_0006 17/03/07 12:05:28 INFO mapreduce.Job: Running job: job_1488905664666_0006 17/03/07 12:05:33 INFO mapreduce.Job: Job job_1488905664666_0006 running in uber mode : false 17/03/07 12:05:33 INFO mapreduce.Job: map 0% reduce 0% 17/03/07 12:05:44 INFO mapreduce.Job: map 100% reduce 0% 17/03/07 12:05:44 INFO mapreduce.Job: Task Id : attempt_1488905664666_0006_m_000000_0, Status : FAILED Error: java.io.IOException: File copy failed: hftp://ec2-52-10-0-80.us-west-2.compute.amazonaws.com:50070/user/msmarnaoui/msmarnaoui/part-m-00000 --> hdfs://nameservice1/user/msmarnaoui/target at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hftp://ec2-52-10-0-80.us-west-2.compute.amazonaws.com:50070/user/msmarnaoui/msmarnaoui/part-m-00000 to hdfs://nameservice1/user/msmarnaoui/target at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.net.UnknownHostException: ip-172-31-10-48.us-west-2.compute.internal

mustafatok commented 7 years ago

The problem is solved by adding all public ips and private ips of each others to /etc/hosts/ in all nodes in the cluster. Furthermore, we both needed to use datanode host name. The setting is configured under HDFS configuration : 

Use DataNode Hostname
dfs.client.use.datanode.hostname
Use DataNode Hostname
dfs.datanode.use.datanode.hostname