搭建Spark集群

[wittyResry]

1. SSH配置

• 打通master和slave集群，把master上spark用户的公钥scp到包括master本身的所有集群上.ssh/authorized_keys
• 注意mac还需要在: 系统偏好-》共享-》远程登录 钩上才能用ssh的22端口

starting org.apache.spark.deploy.master.Master, logging to /Users/resry/sparkTest/spark-2.0.0-bin-hadoop2.3/logs/spark-resry-org.apache.spark.deploy.master.Master-1-resry-2.local.out
localhost: ssh: connect to host localhost port 22: Connection refused

2. Spark配置文件

$ cd conf
$ cp slaves.template slaves
$ cp spark-env.sh.template spark-env.sh

• 编辑spark-evn.sh，加入 export JAVA_HOME=$(/usr/libexec/java_home -v "1.8*")
• 编辑slaves文件，加入所有工作节点的主机名，每个节点占据一行。

  3. 复制Spark目录

• 接着将配置好的Spark目录复制到所有节点上，目录所存放的位置在所有节点上都必须是一致的，如/Users/resry/sparkTest/spark-2.0.0-bin-hadoop2.3

  4. 启动spark集群

• sbin/start-all.sh

  5. 运行一个用例

• 运行example $ ./bin/spark-submit --master spark://resry-2.local:7077 examples/src/main/python/pi.py

  6. 查看运行结果

[wittyResry]

本地执行Spark计算Pi值

$ ./bin/run-example SparkPi 10

• 查看UI:http://10.180.21.26:4040
• 加大Stage试一下 $ ./bin/run-example SparkPi 10000

[wittyResry]

交互式Shell: pyspark

$ ./bin/pyspark
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/10/06 16:41:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/06 16:41:23 WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --num-executors to specify the number of executors
 - Or set SPARK_EXECUTOR_INSTANCES
 - spark.executor.instances to configure the number of instances in the spark config.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.0
      /_/

Using Python version 2.7.10 (default, Oct 23 2015 19:19:21)
SparkSession available as 'spark'.
>>> lines = sc.textFile('/etc/passwd')
>>> lines.count()
96

[wittyResry]

本地提交scala jar任务

• https://github.com/wittyResry/myIssue/tree/sbtTest 首先执行 $ sbt clean package打出jar包

$ pwd
/Users/resry/sparkTest/sbtTest
$ sbt clean package
$ spark-submit --class LineCount --master local ./target/scala-2.11/line-count_2.11-1.0.jar

[wittyResry]

本地提交python任务

• https://github.com/wittyResry/myIssue/tree/sbtTest

$ spark-submit --master local LineCount.py
Test/pythonSparkTest/LineCount.py:6, took 0.726458 s
96
16/10/06 18:32:49 INFO SparkContext: Invoking stop() from shutdown hook