Process hangs at Running job when trying /bin/workloads/micro/wordcount/prepare/prepare.sh

delgadod commented 6 years ago

Hello,

I am using: HiBench 7.0 Hadoop 2.9 Java version 1.8.0_161 Scala code runner version 2.11.6 Apache Maven 3.5.2

All on a three-node Hadoop cluster of OpenStack VM's Ubuntu 16.04.3 LTS VCPUs: 8 RAM: 16GB Size: 10GB

Each has a 100GB volume attached where the dfs storage is kept

When trying to run wordcount prepare script I am stuck at:

hadoop@hadoop0:~/HiBench$ /home/hadoop/HiBench/bin/workloads/micro/wordcount/prepare/prepare.sh patching args= Parsing conf: /home/hadoop/HiBench/conf/hadoop.conf Parsing conf: /home/hadoop/HiBench/conf/hibench.conf Parsing conf: /home/hadoop/HiBench/conf/workloads/micro/wordcount.conf probe sleep jar: /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0-tests.jar start HadoopPrepareWordcount bench hdfs rm -r: /home/hadoop/hadoop/bin/hadoop --config /home/hadoop/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://node-master:9000/HiBench/Wordcount/Input rm: 'hdfs://node-master:9000/HiBench/Wordcount/Input': No such file or directory Submit MapReduce Job: /home/hadoop/hadoop/bin/hadoop --config /home/hadoop/hadoop/etc/hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar randomtextwriter -D mapreduce.randomtextwriter.totalbytes=32000 -D mapreduce.randomtextwriter.bytespermap=4000 -D mapreduce.job.maps=8 -D mapreduce.job.reduces=8 hdfs://node-master:9000/HiBench/Wordcount/Input 18/02/21 04:03:22 INFO mapreduce.Job: Running job: job_1519185680180_0001

I have seen that this is most likely due to a resource issue, but my YARN and MapReduce configurations have been set in order to solve it. Is this a HiBench issue? Are there minimum resource requirements for HiBench wordcount that I am not satisfying?

Below is the output of a few reports while the job is hanging:

hadoop@hadoop0:~$ yarn node -list 18/02/21 04:03:33 INFO client.RMProxy: Connecting to ResourceManager at node-master/10.10.10.7:8032 Total Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers 31-24-168.neu.massopencloud.org:34500 RUNNING 31-24-168.neu.massopencloud.org:8042 0 31-24-168.neu.massopencloud.org:39297 RUNNING 31-24-168.neu.massopencloud.org:8042 0 31-24-168.neu.massopencloud.org:43353 RUNNING 31-24-168.neu.massopencloud.org:8042 1

hadoop@hadoop0:~$ yarn application -list 18/02/21 04:04:05 INFO client.RMProxy: Connecting to ResourceManager at node-master/10.10.10.7:8032 Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1519185680180_0001 random-text-writer MAPREDUCE hadoop default ACCEPTED UNDEFINED 0% N/A

hadoop@hadoop0:~$ hdfs dfsadmin -report Configured Capacity: 316665593856 (294.92 GB) Present Capacity: 300320362240 (279.70 GB) DFS Remaining: 300319195136 (279.69 GB) DFS Used: 1167104 (1.11 MB) DFS Used%: 0.00% Under replicated blocks: 2 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 Pending deletion blocks: 0

Live datanodes (3):

Name: 10.10.10.10:50010 (node1) Hostname: 31-24-168.neu.massopencloud.org Decommission Status : Normal Configured Capacity: 105555197952 (98.31 GB) DFS Used: 331302 (323.54 KB) Non DFS Used: 62558682 (59.66 MB) DFS Remaining: 100106821632 (93.23 GB) DFS Used%: 0.00% DFS Remaining%: 94.84% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Feb 21 04:04:30 UTC 2018 Last Block Report: Wed Feb 21 03:59:45 UTC 2018

Name: 10.10.10.11:50010 (node2) Hostname: 31-24-168.neu.massopencloud.org Decommission Status : Normal Configured Capacity: 105555197952 (98.31 GB) DFS Used: 331302 (323.54 KB) Non DFS Used: 62554586 (59.66 MB) DFS Remaining: 100106825728 (93.23 GB) DFS Used%: 0.00% DFS Remaining%: 94.84% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Feb 21 04:04:30 UTC 2018 Last Block Report: Wed Feb 21 03:59:45 UTC 2018

Name: 10.10.10.7:50010 (localhost) Hostname: 31-24-168.neu.massopencloud.org Decommission Status : Normal Configured Capacity: 105555197952 (98.31 GB) DFS Used: 504500 (492.68 KB) Non DFS Used: 63659340 (60.71 MB) DFS Remaining: 100105547776 (93.23 GB) DFS Used%: 0.00% DFS Remaining%: 94.84% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Wed Feb 21 04:04:30 UTC 2018 Last Block Report: Wed Feb 21 03:59:45 UTC 2018

My yarn-site.xml is as below, obviously wrapped in \\

yarn.acl.enable 0 yarn.resourcemanager.hostname node-master yarn.nodemanager.resource.memory-mb 14392 yarn.nodemanager.resource.cpu-vcores 8 yarn.scheduler.minimum-allocation-mb 1024 yarn.scheduler.minimum-allocation-vcores 1 yarn.scheduler.maximum-allocation-mb 2048 yarn.scheduler.maximum-allocation-vcores 2 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.vmem-check-enabled false

and my mapred-site.xml is:

yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.command-opts -Xmx768m mapreduce.framework.name yarn Execution framework. mapreduce.map.cpu.vcores 1 The number of virtual cores required for each map task. mapreduce.reduce.cpu.vcores 1 The number of virtual cores required for each reduce task. mapreduce.map.memory.mb 1024 Larger resource limit for maps. mapreduce.map.java.opts -Xmx768m Heap-size for child jvms of maps. mapreduce.reduce.memory.mb 1024 Larger resource limit for reduces. mapreduce.reduce.java.opts -Xmx768m Heap-size for child jvms of reduces.

Aamer012 commented 6 years ago

Facing same issue in Hibench 6.0 and 7.0 from 3 weeks.

anchorbob commented 9 months ago

did you resolve the issue? I am facing the same

Intel-bigdata / HiBench