前情提要（城管委）

机器申请7台,5台hadoop集群,2台应用

工具安装(可选)(所有主机):

 yum install -y screen git fish psmisc #(screen避免vpn不稳定断连,更新git方便拉项目,fish偶尔代替bash使用,)

 curl https://raw.githubusercontent.com/Leptune/vim-for-coding/master/install.sh|sh #可选:vim升级,方便使用

 vim /etc/screenrc #取消caption注释,显示screen尾部信息

 screen -S hdpsetup

磁盘挂载

申请机器中Disk /dev/vdb: 107.4 GB 未挂载 (所有主机)

 df -h #查看磁盘使用情况

 root@cgwtest2 ~# df -h
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/vda3        18G  1.7G   17G  10% /
 devtmpfs        7.8G     0  7.8G   0% /dev
 tmpfs           7.8G     0  7.8G   0% /dev/shm
 tmpfs           7.8G  8.4M  7.8G   1% /run
 tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
 /dev/vda1       253M  184M   69M  73% /boot
 tmpfs           1.6G     0  1.6G   0% /run/user/0

 fdisk -l #检查磁盘

 Disk /dev/vda: 107.4 GB, 107374182400 bytes, 209715200 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk label type: dos
 Disk identifier: 0x00045d20

    Device Boot      Start         End      Blocks   Id  System
 /dev/vda1   *        2048      526335      262144   83  Linux
 /dev/vda2          526336     4720639     2097152   82  Linux swap / Solaris
 /dev/vda3         4720640    41943039    18611200   83  Linux

 Disk /dev/vdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes

发现所有机器vdb未被挂载，遂新增两个主分区分别vdb1 20G和vdb2 80G

 fdisk /dev/vdb #根据提示操作 vdb1的20G cgwtest1 挂载HDP安装包路径,其余机器暂未使用vdb1,而vdb2的80G根据往常经验用于HDP各组件安装路径

 mkfs.ext4 /dev/vdb1 #格式化为ext4文件系统
 mkfs.ext4 /dev/vdb2 #格式化为ext4文件系统

 mkdir /data #创建路径为hadoop组件存放准备
 mount /dev/vdb2 /data #挂载80G硬盘分区

 mkdir -p /var/www/html/
 umount /var/www/html

 mount /dev/vdb1 /var/www/html #挂载20G为安装包存放
 vim /etc/fstab # 添加  /dev/vdb2 /data ext4 defaults 0 0

离线安装HDP(避免网络波动),需提前下载安装包到服务器(cgwtest1.xxx.com)

 mkdir /data/packagebackup20180918/

 cd /data/packagebackup20180918/

 wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/ambari-2.5.1.0-centos7.tar.gz

 wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/HDP-2.6.1.0-centos7-rpm.tar.gz

 wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

开始部署

软件环境要求:

 yum and rpm (RHEL/CentOS/Oracle/Amazon Linux)

 scp, curl, unzip, tar, and wget

 OpenSSL (v1.01, build 16 or later)

 # For CentOS 7, Ubuntu 14, Ubuntu 16, and Debian 9:  
 # Python 2.7.x

最大打开文件数设置(所有主机)

ulimit -n 10000

设置FQDN主机名,另如需更改组件默认路径需提前创建(所有主机)

hostname:

检查:hostname -f (要求FQDN)

修改hostname文件

 vim /etc/hostname 

 cgwtest1.xxx.com #所有机器按序号

刷新hostname

 hostname -F /etc/hostname

设置hosts文件(所有主机)

 vim /etc/hosts 添加
  192.168.0.186  cgwtest1.xxx.com   cgwtest1
  192.168.0.187  cgwtest2.xxx.com   cgwtest2
  192.168.0.188  cgwtest3.xxx.com   cgwtest3
  192.168.0.189  cgwtest4.xxx.com   cgwtest4
  192.168.0.190  cgwtest5.xxx.com   cgwtest5

注意勿删:

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

SSH互信(所有主机)

 ssh-keygen #（一路回车）

 ssh-copy-id  cgwtest1 # (输入yes,输入cgwtest1密码)

 ssh-copy-id  cgwtest2 3 4 5 #重复

测试免密登陆:ssh 节点

ssh cgwtest1 2 3 4 5 重复

ssh cgwtest1.xxx.com 2 3 4 5重复

安装NTP同步时钟(所有主机)

 yum install -y ntp

 systemctl enable ntpd

设置网络配置(所有主机)

 vim /etc/sysconfig/network

 NETWORKING=yes

 HOSTNAME=cgwtest1.xxx.com #每个机器按序号

设置防火墙(所有主机)

 systemctl disable firewalld

 service firewalld stop

关闭SELinux(所有主机)

 setenforce 0

关闭iptables(所有主机)

 chkconfig iptables off #孝鹏配置,命令报错,暂时先忽略

设置UMASK(所有主机)****

 umask 0022

 echo umask 0022 >> /etc/profile

孝鹏脚本配置(官网手册没有)(所有主机)

echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag
echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled

#install jdk
mkdir -p /opt/hdp
chmod 777 /opt/hdp/
wget http://10.36.8.17/web/jdk.tar -O /root/jdk.tar
cd /usr/local/
tar -xvf /root/jdk.tar
rm -rf /usr/bin/java
ln -s /usr/local/jdk/bin/java /usr/bin/java
ln -s /usr/local/jdk/bin/jps /usr/bin/jps
ln -s /usr/local/jdk/bin/jstat /usr/bin/jstat
ln -s /usr/local/jdk/bin/jstack /usr/bin/jstack
ln -s /usr/local/jdk/bin/jmap /usr/bin/jmap

java

mv /usr/local/usr/local/jdk /usr/local/

## 查看丽水MYSQL(root/xxx2017_)版本信息(10.1.4.147)

 mysql -v #shell环境查看

# 丽水mysql版本:mysql  Ver 14.14 Distrib 5.7.20, for Linux (x86_64) using  EditLine wrapper

 mysql -h localhost -u root -p #登录

 status #登录后查看

 mysql  Ver 14.14 Distrib 5.7.20, for Linux (x86_64) using  EditLine wrapper

 Connection id:          2803
 Current database:
 Current user:           root@localhost
 SSL:                    Not in use
 Current pager:          stdout
 Using outfile:          ''
 Using delimiter:        ;
 Server version:         5.7.20 MySQL Community Server (GPL)
 Protocol version:       10
 Connection:             Localhost via UNIX socket
 Server characterset:    utf8
 Db     characterset:    utf8
 Client characterset:    utf8
 Conn.  characterset:    utf8
 UNIX socket:            /var/lib/mysql/mysql.sock
 Uptime:                 3 days 18 hours 3 min 5 sec

 Threads: 29  Questions: 11280977  Slow queries: 0  Opens: 1146  Flush tables: 1  Open tables: 1094  Queries per second avg: 34.797

安装MYSQL5.7(cgwtest1.xxx.com)

 wget https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm

 rpm -Uvh mysql80-community-release-el7-1.noarch.rpm

 yum repolist all | grep mysql #查看可选安装版本

 yum install -y yum-utils 

 yum-config-manager --disable mysql80-community #disable最新的8.0版本

 yum-config-manager --enable mysql57-community #enable 5.7 与丽水一样

 yum repolist all | grep mysql #再次确认接下来安装的版本

 yum install -y mysql-community-server  #开始安装

 systemctl start mysqld.service #启动mysql

 systemctl status mysqld.service #查看启动状态

    # Redirecting to /bin/systemctl status  mysqld.service
    # ● mysqld.service - MySQL Server
    #    Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
    #    Active: active (running) since Sun 2018-09-16 11:29:47 CST; 2min 0s ago
    #      Docs: man:mysqld(8)
    #            http://dev.mysql.com/doc/refman/en/using-systemd.html
    #   Process: 22937 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
    #   Process: 22861 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
    #  Main PID: 22939 (mysqld)
    #    CGroup: /system.slice/mysqld.service
    #            └─22939 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
    #
    # Sep 16 11:29:34 cgwtest1.xxx.com systemd[1]: Starting MySQL Server...
    # Sep 16 11:29:47 cgwtest1.xxx.com systemd[1]: Started MySQL Server.

grep 'temporary password' /var/log/mysqld.log #查看初始密码

# 2018-09-16T03:29:37.382134Z 1 [Note] A temporary password is generated for root@localhost: **z2cqJDr5N0&h**

mysql -uroot -p #登录 输入密码

alter user 'root'@'localhost' identified by 'xxx2017_'; #修改与丽水一样密码

status 

    # mysql  Ver 14.14 Distrib 5.7.23, for Linux (x86_64) using  EditLine wrapper
    #
    # Connection id:    5
    # Current database:
    # Current user:     root@localhost
    # SSL:            Not in use
    # Current pager:    stdout
    # Using outfile:    ''
    # Using delimiter:  ;
    # Server version:   5.7.23 MySQL Community Server (GPL)
    # Protocol version: 10
    # Connection:       Localhost via UNIX socket
    # Server characterset:   latin1
    # Db     characterset:   latin1
    # Client characterset:   utf8
    # Conn.  characterset:   utf8
    # UNIX socket:      /var/lib/mysql/mysql.sock
    # Uptime:           30 min 54 sec
    #
    # Threads: 1  Questions: 11  Slow queries: 0  Opens: 106  Flush tables: 1  Open tables: 99  Queries per second avg: 0.005

如果需要简单密码需要修改认证配置:

 show variables like 'validate%';

 set xxxx = xxxx

将字符集修改与丽水一样:

 show variables like '%char%';

 set character_set_database=utf8;

 set character_set_server=utf8;

创建ambari、hive、oozie数据库备用


 create database ambari;

 create database hive;

 create database oozie;

庭桂有道云MYSQL配置(cgwtest1.xxx.com)

 grant all privileges on *.* to 'root'@'%' identified by 'xxx2017_' with grant option;

 FLUSH PRIVILEGES;

 yum install -y mysql-connector-java #为之后ambari使用mysql安装JDBC

JDBC地址(备忘):/usr/share/java/mysql-connector-java.jar

离线安装HDP(cgwtest1.xxx.com)

安装httpd服务(cgwtest1.xxx.com)

 yum install -y httpd #安装httpd

 如果之前没有挂载这里可以挂:mount /dev/vdb1 /var/www/html/

 service httpd restart #启动

解压提前下载的3个安装包(cgwtest1.xxx.com)

 cd /var/www/html/

 mkdir ambari
 tar -xzvf /data/packagebackup20180918/ambari-2.5.1.0-centos7.tar.gz -C /var/www/html

 mkdir  -p /var/www/html/hdp/
 tar -xzvf /data/packagebackup20180918/HDP-2.6.1.0-centos7-rpm.tar.gz -C /var/www/html/hdp

 mkdir -p /var/www/html/hdp/HDP-UTILS
 tar -xzvf /data/packagebackup20180918/HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS

安装yum-plugin-priorities (所有机器)

 yum install -y yum-plugin-priorities **(所有机器)**

 vim /etc/yum/pluginconf.d/priorities.conf #添加以下配置**(所有机器)**

  enabled=1
  gpgcheck=0

下载ambari公共仓库并修改参数(baseurl gpgcheck)后拷贝至yum仓库地址(cgwtest1.xxx.com)

 cd ~

 wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/ambari.repo #(cgwtest1)

 vim ambari.repo

    #VERSION_NUMBER=2.5.1.0-159  2 [ambari-2.5.1.0]
    name=ambari Version - ambari-2.5.1.0
    baseurl=http://cgwtest1.xxx.com/ambari/centos7/
    gpgcheck=0
    gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/RPM-GPG-KEY/RPM-G
    enabled=1
    priority=1

 cp ambari.repo /etc/yum.repos.d/

建立ambari-server(cgwtest1.xxx.com)

 yum install -y ambari-server

 # 手动修改参数(二选一):vim /etc/ambari-server/conf/ambari.properties

  添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

  修改server.jdbc.rca.url=jdbc:mysql://localhost:3306/ambari

 # 或者直接执行命令修改(二选一,推荐)

  ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar #设置mysql jdbc驱动 

  ambari-server setup

  Customize user account for ambari-server daemon [y/n] (n)?

 # Checking JDK...
 # [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
 #
 # [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
 #
 # [3] Custom JDK
 #
 # Enter choice (1): **3**

  输入jdk路径:**/usr/local/jdk**

  Enter advanced database configuration [y/n] (n)? **y**

  Choose one of the following options:
  [1] - PostgreSQL (Embedded)
  [2] - Oracle
  [3] - MySQL / MariaDB
  [4] - PostgreSQL
  [5] - Microsoft SQL Server (Tech Preview)
  [6] - SQL Anywhere

  [7] - BDB

  Enter choice (1): **3**

  Hostname (localhost): 
  Port (3306):
  Database name (ambari):**ambari**
  Username (ambari): **root**
  Enter Database Password (bigdata):**xxx2017_**
  Re-enter password:**xxx2017_**
  Configuring ambari database...
  WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.

  Press <enter> to continue.

  Configuring remote database connection properties...
  WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

  > **mysql -uroot -p #登录mysql**
  >
  > **use ambari; #进入ambari数据库**
  >
  > **source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql**
  >
  > **ctrl+D #退出mysql**

  Proceed with configuring remote database connection properties [y/n] (y)?
  Extracting system views...
  ............
  Adjusting ambari-server permissions and ownership...
  Ambari Server 'setup' completed successfully.

开启ambari-server(cgwtest1.xxx.com)

 ambari-server start
 ambari-server status

登录Apache Ambari(成都VPN)

http://cgwtest1.xxx.com:8080 (admin/admin)

启动安装向导
创建集群
输入集群名字:
cgwtest：选中HDP2.6.1.0 #与丽水一样，使用本地仓库local repository

redhat7

HDP-2.6

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0

修改为

http://cgwtest1.xxx.com/hdp/HDP/centos7/

HDP-UTILS-1.1.0.21

http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7

修改为

http://cgwtest1.xxx.com/hdp/HDP-UTILS/

输入目标主机:

hostname:

 cgwtest[1-5].xxx.com

从服务器SFTP下载私钥再上传ambari安装向导

确认集群主机
选择服务，参照丽水
根据孝鹏、庭桂部署经验,所有参数以/var、/hadoop、/hdp开头的path都改为/data目录打头

hive & oozie 数据库配置需注意:
#  选择**Existing MySQL / MariaDB Database**  
# 
#  Database Name:**hive** & **oozie**
# 
#  Database Username:**root**
# 
#  Database Password:**xxxxxxx**
# 
#  Database URL改为**jdbc:mysql://cgwtest1.xxx.com/hive**
# 
#  **Test Connection**  Connection OK即可
[Ambari Metrics]&[SmartSense]密码:******

Ranger Requirements

You must have an MySQL/Oracle/Postgres/MSSQL/SQL Anywhere Serverdatabase instance running to be used by Ranger.In Assign Masters step of this wizard, you will be prompted to specify which host for the Ranger Admin. On that host, you must have DB Client installed for Ranger to access to the database. (Note: This is applicable for only Ranger 0.4.0)Ensure that the access for the DB Admin user is enabled in DB server from any host.Execute the following command on the Ambari Server host. Replace database-type with mysql|oracle|postgres|mssql|sqlanywhere and /jdbc/driver/pathbased on the location of corresponding JDBC driver:ambari-server setup --jdbc-db={database-type} --jdbc-driver={/jdbc/driver/path}I have met all the requirements above.

分配Masters
分配Slaves and Clients
修改全部服务路径为data,已提前做好磁盘挂载

修改Misc的所有services accounts为hadoop

Configurations

Some service configurations are not configured properly. We recommend you review and change the highlighted configuration values. Are you sure you want to proceed without correcting configurations?

| Type    | Service | Property                     | Value                                                       | Description                                                  |
| ------- | ------- | ---------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
| Warning | Atlas   | atlas.graph.storage.hostname | cgwtest4.xxx.com,cgwtest3.xxx.com,cgwtest5.xxx.com | Atlas is configured to use the HBase installed in this cluster. If you would like Atlas to use another HBase instance, please configure this property and HBASE_CONF_DIR variable in atlas-env appropriately. |

预览Review

Admin Name : admin
Cluster Name : cgwtest

Total Hosts : 5 (5 new)

Repositories:

redhat7 (HDP-2.6): http://cgwtest1.xxx.com/hdp/HDP/centos7/
redhat7 (HDP-UTILS-1.1.0.21): http://cgwtest1.xxx.com/hdp/HDP-UTILS/

Services:

HDFS
- DataNode : 5 hosts
- NameNode : cgwtest1.xxx.com
- NFSGateway : 0 host
- SNameNode : cgwtest2.xxx.com
YARN + MapReduce2
- App Timeline Server : cgwtest1.xxx.com
- NodeManager : 2 hosts
- ResourceManager : cgwtest2.xxx.com
Tez
- Clients : 1 host
Hive
- Metastore : cgwtest3.xxx.com
- HiveServer2 : cgwtest3.xxx.com
- WebHCat Server : cgwtest3.xxx.com
- Database : Existing MySQL / MariaDB Database
HBase
- Master : cgwtest3.xxx.com
- RegionServer : 2 hosts
- Phoenix Query Server : 0 host
Pig
- Clients : 1 host
Sqoop
- Clients : 1 host
Oozie
- Server : cgwtest4.xxx.com
- Database : Existing MySQL / MariaDB Database
ZooKeeper
- Server : 3 hosts
Falcon
- Server : cgwtest5.xxx.com
Ambari Infra
- Infra Solr Instance : cgwtest5.xxx.com
Ambari Metrics
- Metrics Collector : cgwtest4.xxx.com
- Grafana : cgwtest5.xxx.com
Atlas
- Metadata Server : cgwtest3.xxx.com
Kafka
- Broker : cgwtest4.xxx.com
SmartSense
- Activity Analyzer : cgwtest3.xxx.com
- Activity Explorer : cgwtest3.xxx.com
- HST Server : cgwtest5.xxx.com
Zeppelin Notebook
- Notebook : cgwtest5.xxx.com
Slider
- Clients : 1 host

Summary

Here is the summary of the install process.

The cluster consists of 5 hosts
- Installed and started services successfully on 4 new hosts
- 1 warnings
Master services installed
- NameNode installed on cgwtest1.xxx.com
- SNameNode installed on cgwtest2.xxx.com
- ResourceManager installed on cgwtest2.xxx.com
- History Server installed on cgwtest2.xxx.com
- HiveServer2 installed on cgwtest3.xxx.com
- HBase Master installed on cgwtest3.xxx.com
- Oozie Server installed on cgwtest4.xxx.com
Starting services failed

启动失败的服务手动启动

dfs.permissions.superusergroup=hdfs,hadoop

groupadd supergroup

usermod -a -G supergroup hadoop,hdfs

su - hdfs -s /bin/bash -c "hdfs dfsadmin -refreshUserToGroupsMappings"

cgwtest1.xxx.com:8080

HDFS

configs

custom core-site

add :

hadoop.proxyuser.ambari.groups= hadoop.proxyuser.ambari.hosts=

remove

other proxyuser

hdfs dfs -chown -R hadoopo /

hdfs dfs -chgrp -R hadoop /

hadoop用户取消密码登陆

 passwd -d hadoop #免密码

完成

change  hive.metastore.warehouse.dir from /data/apps/hive/warehouse to /apps/hive/warehouse

change fs.defaultFS from hdfs://cgwtest1.xxx.com:8020 to hdfs://cgwtest1.xxx.com:8082

Zakariyya / blog

集群HDP离线部署文档 #136