moonyaero commented 6 years ago

Steps of Installation

1. CM Install Labs - Before You Start

2. CM Install Lab - Prepare your instances

3. Create five m4.xlarge nodes

4. CM Install Lab

5. MySQL/MariaDB Installation Lab

6. Configure MySQL with a replica server

7. MySQL installation - Plan Two Detail

8. Cloudera Manager Install Lab

9. Cloudera Manager Install Lab

Appendix i. Cluster install: Bonus Lab

moonyaero commented 6 years ago

CM Install Labs - Before You Start

moonyaero commented 6 years ago

Creation of 5 m4.xlarge nodes
AMI set to CentOS 6.8 Clean
Storage of each instance is 40GB

moonyaero commented 6 years ago

Downloaded AWS Private Keys and store it safely. Able to SSH into all of the instances using PuTTY.

moonyaero commented 6 years ago

Run Cloudera Pre-requisite Task on all the nodes.

1) Disable SELinux

2) Set vm.swappiness

vm.swappiness = 1

3) Show mount attributes and remove reserved block counts on ext4, verify with the second command

sudo tune2fs -m 0 /dev/xvda1 sudo tune2fs -l /dev/xvda1 | grep 'Reserved block count

4) Disable Transparent Huge Pages by appending the following into /etc/rc.d/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag

5) Set correct host configuration in /etc/hosts by appending the following:

ec2-13-250-104-43.ap-southeast-1.compute.amazonaws.com namenode 
ec2-52-221-200-239.ap-southeast-1.compute.amazonaws.com cm 
ec2-54-169-75-184.ap-southeast-1.compute.amazonaws.com datanode1 
ec2-54-251-176-98.ap-southeast-1.compute.amazonaws.com datanode2 
ec2-54-255-205-156.ap-southeast-1.compute.amazonaws.com datanode3

6) Show that forward and reverse host lookup are correctly resolved

Example Command: nslookup ec2-54-255-205-156.ap-southeast-1.compute.amazonaws.com

Ouput:

Server: 172.31.0.2 
Address: 172.31.0.2#53 

Non-authoritative answer: 
Name: ec2-54-255-205-156.ap-southeast-1.compute.amazonaws.com 
Address: 172.31.29.155

7)Show the nscd and ntpd service is running (installed nscd)

Output: NSCD

[root@ip-172-31-16-48 ec2-user]# service nscd status 
nscd (pid 5948) is running...

NTPD

[root@ip-172-31-16-48 ec2-user]# service ntpd status 
ntpd (pid 1324) is running...

moonyaero commented 6 years ago

Download Cloudera Prequisite Components and Cloudera Manager

i) Make custom repositories mkdir /repofiles cd repofiles

ii) Download MySQL rpm by downloading mysql-community rpm from mysql website. Only on MySQL Master and MySQL Replica Node 'wget http://repo.mysql.com/mysql80-community-release-el6-1.noarch.rpm sudo yum install mysql-server`

iii) Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html. This should be done on all the other nodes. Unzip it into path, copy and rename them into target directory (/usr/share/java) tar zxvf mysql-connector-java-5.1.31.tar.gz '$ sudo cp mysql-connector-java-5.1.31/mysql-connector-java-5.1.31-bin.jar /usr/share/java/mysql-connector-java.jar'

iii) Download Oracle JDK Should be done on all the nodes wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.14.3/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

moonyaero commented 6 years ago

Installation and Configuration of Master MySQL

Note: Same Installation of MySQL will need to be done on your Replica Node

Installation of MySQL community server

yum install mysql80-community-release-el6-1.noarch.rpm yum -y install mysql-server

Start your MySQL service

service mysqld start

Run MySQL secure installation script

/usr/bin/mysql_secure_installation

OUTPUT:

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!

Login as root into MySQL to verify it is successful

mysql -u root -p

Create scm database grant privileges

mysql > create database scm DEFAULT CHARACTER SET utf8;
mysql >grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'scm_password';

Run scm_prepare_database.sh script to prepare your scm database

sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -h ip-172-31-30-109.ap-southeast-1.compute.internal scm scm

Create necessary databases, users, and set privileges

mysql > create database amon DEFAULT CHARACTER SET utf8;
mysql > create database rman DEFAULT CHARACTER SET utf8;
mysql > create database metastore DEFAULT CHARACTER SET utf8;
mysql > create database sentry DEFAULT CHARACTER SET utf8;
mysql > create database nav DEFAULT CHARACTER SET utf8;
mysql > create database navms DEFAULT CHARACTER SET utf8;
mysql > create database oozie DEFAULT CHARACTER SET utf8;
mysql > create database hue default character set utf8 default collate utf8_general_ci;

mysql > grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
mysql > grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
mysql > grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
mysql > grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';
mysql > grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';
mysql > grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';
mysql > grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
mysql > grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
mysql > grant all on hue.* to 'hue'@'%' identified by 'huepassword';

mysql > flush privileges;

Ensure you are able to login into as a user

e.g. mysql -u amon -p

moonyaero commented 6 years ago

Configuration of MySQL Replica on Datanode1

Terminology: Master MySQL - CM Node Replica MySQL - Datanode1

On the master MySQL node, grant replication privileges for your replica node:

a. Log in with mysql -u ... -p 
b. Note the FQDN of your replica host.
c. mysql> `GRANT REPLICATION SLAVE ON *.* TO 'user'@'FQDN' IDENTIFIED BY 'password';`
d. mysql> `SET GLOBAL binlog_format = 'ROW';` 
e. mysql> `FLUSH TABLES WITH READ LOCK;`

In a second terminal session, log into the MySQL master and show its status:

a. mysql> `SHOW MASTER STATUS;`
b. Make note of the file name and byte offset. The replica needs this info to sync to the master.
c. Logout of the second session; remove the lock on the first with mysql> UNLOCK TABLES;

On Master node, do a SQL dump, gunzip it,

a. `mysqldump -A -u root -p --flush-privileges --master-data=1 | gzip -1 > master.sql.gz`
b. `zgrep -m 1 -P 'CHANGE MASTER' master.sql.gz`

Copy the zip file into /var/www/html where you can download it locally. Upload locally from your laptop to the Replica Node

Login to the replica server (via your unzip master.sql file) and configure a connection to the master:

a. `mysql -u root -p < master.sql`
b. `CHANGE MASTER TO MASTER_HOST='ec2-52-221-200-239.ap-southeast-1.compute.amazonaws.com', MASTER_USER='root', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql_binary_log.000002', MASTER_LOG_POS=11462134;`

Initiate slave operations on the replica

a. mysql> `START SLAVE;`
b. mysql> `SHOW SLAVE STATUS \G`
c. If successful, the Slave_IO_State field will read Waiting for master to send event
d. Once successful, capture this output and store it in labs/2_replica_working.md

Create a database called test on Master Node, verified that it appeared on Replica Node

moonyaero commented 6 years ago

Cloudera Manager Install Lab

Ensure that the repo for Cloudera Manager is the correct version

Change your baseurl to the version that you are using
You can find the path under http://archive.cloudera.com/cm5/
In this installation, we are installing the latest version so we do not need to modify the repo baseurl.

Path B install using CM 5.12.x

1) Install a supported Oracle JDK on your CM Node

i) Do a yum installation

yum installed oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

Note: If you are manually installing JDK on all the nodes, ensure this download and installation is done first

ii) Set JAVA PATH (do the same for all the nodes after installation

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/ export PATH=$JAVA_HOME/bin:$PATH

2) Install a supported JDBC connector on all nodes

i) This step is already done on in Cloudera Prequisite Components

3) Install cloudera-manager-server and cloudera-manager-agent

yum install cloudera-manager-server yum install cloudera-manager-daemons yum install cloudera-manager-agent

4) Create the databases and access grants you will need

i) You should have already created the databases and access grant for amon, rman, metastore, sentry, nav, navms, hue, oozie while setting up MySQL database.

5) Configure Cloudera Manager to connect to the database

i) Likewise, this is already done when you ran the scm_prepare_database.sh script to prepare your scm database.

6) After you're done with the above. we then try to connect to cloudera manager UI via your Public DNS IP with port 7180

7) If you are able to see the UI, login with admin, admin credentials.

8) Run through with the installation wizard

Search for your AWS private IP hostname. (if you are unable to verify host health, check your AWS security groups, incoming ports)
Assign roles for your nodes.
- 1 Master Node, 1 CM, 3 Worker Node
- Make sure there is at least 3 nodes running zookeeper.
Provide databases credentials and test connection
Allow for distribution of parcels and activation.

moonyaero / SEBC

1) Installation Lab #1

Run Cloudera Pre-requisite Task on all the nodes.

1) Disable SELinux

2) Set vm.swappiness

3) Show mount attributes and remove reserved block counts on ext4, verify with the second command

4) Disable Transparent Huge Pages by appending the following into /etc/rc.d/rc.local

5) Set correct host configuration in /etc/hosts by appending the following:

6) Show that forward and reverse host lookup are correctly resolved

7)Show the nscd and ntpd service is running (installed nscd)

Download Cloudera Prequisite Components and Cloudera Manager

Installation and Configuration of Master MySQL

Installation of MySQL community server

Start your MySQL service

Run MySQL secure installation script

Login as root into MySQL to verify it is successful

Create scm database grant privileges

Run scm_prepare_database.sh script to prepare your scm database

Create necessary databases, users, and set privileges

Ensure you are able to login into as a user

Configuration of MySQL Replica on Datanode1

On the master MySQL node, grant replication privileges for your replica node:

In a second terminal session, log into the MySQL master and show its status:

On Master node, do a SQL dump, gunzip it,

Copy the zip file into /var/www/html where you can download it locally. Upload locally from your laptop to the Replica Node

Login to the replica server (via your unzip master.sql file) and configure a connection to the master:

Initiate slave operations on the replica

Create a database called test on Master Node, verified that it appeared on Replica Node

Cloudera Manager Install Lab

Ensure that the repo for Cloudera Manager is the correct version

Path B install using CM 5.12.x

1) Install a supported Oracle JDK on your CM Node

2) Install a supported JDBC connector on all nodes

3) Install cloudera-manager-server and cloudera-manager-agent

4) Create the databases and access grants you will need

5) Configure Cloudera Manager to connect to the database

6) After you're done with the above. we then try to connect to cloudera manager UI via your Public DNS IP with port 7180

7) If you are able to see the UI, login with admin, admin credentials.

8) Run through with the installation wizard

9) After Cluster restart, verify that the cluster is up and running