xkrogen
commented
5 years ago Hi @wangzhe330 , thanks for trying out Dynamometer and sorry to hear that you are experiencing an issue. It looks like the NameNode is receiving the parameters just fine; if you look at the log line including the createNameNode statement, it shows all of the correct parameters listed there as expected.
Your startup command seems a little odd -- it contains --conf_path twice, and --block_list_path without any argument. Did you make a mistake when pasting it in?
Can you additionally take a look at the logs on the ApplicationMaster? Please note that the NameNode does not run in the same container as the AM; there is a distinct container which runs the AM only. You should be able to find the link to this in the driver logs or from the ResourceManager UI.
wangzhe330
hexiangheng
start-dynamometer-cluster.sh command: ./start-dynamometer-cluster.sh --hadoop_binary_path hadoop-2.7.2.tar.gz --conf_path /opt/hadoop/wz/dynamome --conf_path /opt/hadoop/wz/dynamometer/bin/conf/ --fs_image_dir hdfs:///dyno/fsimage --block_list_path
check the NameNode's starting log on the AM node : 2019-01-08 16:32:38,311 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2019-01-08 16:32:38,315 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode [-D, fs.defaultFS=hdfs://host-xx-xx:9002, -D, dfs.namenode.rpc-address=host-xx-xx:9002, -D, dfs.namenode.servicerpc-address=host-xx-xx:9022, -D, dfs.namenode.http-address=host-xx-xx:50077, -D, dfs.namenode.https-address=host-xx-xx:0, -D, dfs.namenode.name.dir=file:///opt/huawei/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1546852874867_0024/container_1546852874867_0024_01_000002/dyno-node/name-data, -D, dfs.namenode.edits.dir=file:///opt/huawei/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1546852874867_0024/container_1546852874867_0024_01_000002/dyno-node/name-data, -D, dfs.namenode.checkpoint.dir=file:///opt/huawei/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1546852874867_0024/container_1546852874867_0024_01_000002/dyno-node/checkpoint, -D, dfs.namenode.safemode.threshold-pct=0.0f, -D, dfs.permissions.enabled=true, -D, dfs.cluster.administrators="*", -D, dfs.block.replicator.classname=com.linkedin.dynamometer.BlockPlacementPolicyAlwaysSatisfied, -D, hadoop.security.impersonation.provider.class=com.linkedin.dynamometer.AllowAllImpersonationProvider, -D, hadoop.tmp.dir=/opt/huawei/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1546852874867_0024/container_1546852874867_0024_01_000002/dyno-node, -D, hadoop.security.authentication=simple, -D, hadoop.security.authorization=false, -D, dfs.http.policy=HTTP_ONLY, -D, dfs.client.read.shortcircuit=false] 2019-01-08 16:32:38,318 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /**** SHUTDOWN_MSG: Shutting down NameNode at host-xx-xx ****/
In start-component.sh , line 277: ${HADOOP_HOME}/sbin/hadoop-daemon.sh start namenode $namenodeConfigs $NN_ADDITIONAL_ARGS;
It seems that NameNode can't recognize the parameters ( $namenodeConfigs ) The $namenodeConfigs is like this : read -r -d '' namenodeConfigs <<EOF -D fs.defaultFS=hdfs://${nnHostname}:${nnRpcPort} -D dfs.namenode.rpc-address=${nnHostname}:${nnRpcPort} -D dfs.namenode.servicerpc-address=${nnHostname}:${nnServiceRpcPort} -D dfs.namenode.http-address=${nnHostname}:${nnHttpPort} -D dfs.namenode.https-address=${nnHostname}:0 -D dfs.namenode.name.dir=file://${nameDir} -D dfs.namenode.edits.dir=file://${editsDir}
If I has usage error for start-dynamometer-cluster.sh ?