HiBench can't run without Yarn

at15 commented 7 years ago

[x] Hadoop support single node with Yarn setup in YARN on a Single Node

[x] it seems single node Yarn won't work

https://github.com/intel-hadoop/HiBench/issues/138

https://github.com/intel-hadoop/HiBench/blob/master/bin/functions/load_config.py#L515

File "/usr/lib/hibench/bin/functions/load_config.py", line 528, in probe_masters_slaves_by_Yarn
assert 0, "Get workers from yarn-site.xml page failed, reason:%s\nplease set `hibench.masters.hostnames` and `hibench.slaves.hostnames` manually" % e
AssertionError: Get workers from yarn-site.xml page failed, reason:Unknown resourcemanager, please check `hibench.hadoop.configure.dir` and "yarn-site.xml" file
please set `hibench.masters.hostnames` and `hibench.slaves.hostnames` manually

at15 commented 7 years ago

Add the following to yarn-site.xml works

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-master</value>
    </property>

at15 commented 7 years ago

Can't start datanode today, got issue

End of File Exception between local host is: "vagrant.vm/127.0.1.1"; destination host is: "0.0.0.0":9000

at15 commented 7 years ago

Found something about the host file in https://github.com/apache/bigtop/blob/master/provisioner/vagrant/Vagrantfile#L72

# Remove 127.0.0.1 entry since vagrant's hostname setting will map it to FQDN,
# which miss leads some daemons to bind on 127.0.0.1 instead of public or private IP
sed -i "/127.0.0.1/d" /etc/hosts

at15 commented 7 years ago

so all hosts file are overwritten now, this problem is gone, also host name is changed for every vm

at15 / hadoop-spark-perf

HiBench can't run without Yarn #5