search

liangguohun / HadoopSpark

Haddoop and Spark config
Apache License 2.0
0 stars 0 forks source link

hadoop yarn 运行 #11

Open liangguohun opened 7 years ago

liangguohun commented 7 years ago

~/.bashrc 补上下面 export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

spark-submit --driver-memory 2g --class RunWordCount --master yarn-client bin/WordCount.jar 运行后存在hdfs上

2、配置spark standalone环境变量 cp /usr/local/spark/conf/spark-env.sh.template /usr/local/spark/conf/spark-env.sh export SPARK_MASTER_IP=master export SPARK_WORKER_CORES=1 export SPARK_WORKER_MEMORY=400M export SPARK_WORKER_INSTANCES=2 //相应内容看文档里头的注释

ssh data1 sudo mkdir /usr/local/spark sudo chown hduser:hduser /usr/local/spark exit sudo scp -r /usr/local/spark hduser@data2:/usr/local 如果没有改变所有者 The authenticity of host 'data2 (192.168.56.102)' can't be established. ECDSA key fingerprint is 83:b3:ca:a3:01:b6:d6:6e:ea:d7:e4:3a:e7:d9:02:28. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'data2,192.168.56.102' (ECDSA) to the list of known hosts. hduser@data2's password: 即使输入也没有权限，不改变所有者尝试

liangguohun commented 7 years ago

/usr/local/spark/sbin/start-all.sh 运行 浏览器http://master:8080 查看运行情况

spark-submit --driver-memory 2g --class RunWordCount --master spark://master:7077 bin/WordCount.jar