search

ibrahimsaf / SEBC

0 stars 0 forks source link

Installation Lab #1

Closed ibrahimsaf closed 6 years ago

ibrahimsaf commented 6 years ago

Change AWS location to francfurt Use 100Gohard disks, The Centos7 image si a 7.4, not compatible with cdh 5.8.3 So I tooked the centos6 image. It's a 6.9, so compatible with cloudera 5.8.3. => A bug while expanding disk. So I took an older version of centos7 template which is a centos 7.2.

Configure the swappiness on all machines : sysctl vm.swappiness=1 Edit the /etc/sysctl.conf file and add at the end: "vm.swappiness=1"

df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.8G 666M 6.7G 9% / tmpfs 7.8G 0 7.8G 0% /dev/shm

Disable transparent hugepage support on all nodes: echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

ibrahimsaf commented 6 years ago

Network configuration:

ifconfig

eth0 Link encap:Ethernet HWaddr 06:97:48:8F:A8:62 inet addr:172.31.38.98 Bcast:172.31.47.255 Mask:255.255.240.0 inet6 addr: fe80::497:48ff:fe8f:a862/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:668 errors:0 dropped:0 overruns:0 frame:0 TX packets:577 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:65049 (63.5 KiB) TX bytes:72749 (71.0 KiB) Interrupt:145

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

ibrahimsaf commented 6 years ago

NSLOOKUP: Install the dns utils: sudo yum install bind-utils then: nslookup ip-172-31-43-182 Server: 172.31.0.2 Address: 172.31.0.2#53

Non-authoritative answer: Name: ip-172-31-43-182.eu-central-1.compute.internal Address: 172.31.43.182

service nscd status service ntpd status sudo yum install nscd sudo yum install ntp

[centos@ip-172-31-43-182 ~]$ service nscd status nscd is stopped [centos@ip-172-31-43-182 ~]$ sudo service nscd start Starting nscd: [ OK ] [centos@ip-172-31-43-182 ~]$ sudo service ntpd start Starting ntpd: [ OK ] [centos@ip-172-31-43-182 ~]$ sudo service ntpd status ntpd (pid 1741) is running... [centos@ip-172-31-43-182 ~]$ sudo service nscd status nscd (pid 1712) is running...

ibrahimsaf commented 6 years ago

COnfigure the MySQL repo: yum -y install wget wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm rpm -Uvh mysql57-community-release-el7-11.noarch.rpm

yum clean all yum check-update

Activate the Mysql 5.5 repo only. then install the mysql-server on the master and replica: vim /etc/yum.repos.d/mysql-community.repo Enable to use MySQL 5.5 [mysql55-community] name=MySQL 5.5 Community Server baseurl=http://repo.mysql.com/yum/mysql-5.5-community/el/7/$basearch/ enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

Install mysql on all nodes: yum install mysql-community-client

Install the mysql server on master and replica node: yum install mysql-community-server Verify the installation: [root@ip-172-31-4-12 ~]# yum list installed mysql-community-server Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile

Use the configuration file recommended by cloudera: [mysqld] transaction-isolation = READ-COMMITTED

Disabling symbolic-links is recommended to prevent assorted security risks;

to do so, uncomment this line:

symbolic-links = 0

key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1

max_connections = 550

expire_logs_days = 10

max_binlog_size = 100M

log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system

and chown the specified folder to the mysql user.

log_bin=/var/lib/mysql/mysql_binary_log

For MySQL version 5.1.8 or later. For older versions, reference MySQL documentation for configuration help.

binlog_format = mixed

read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M

InnoDB settings

innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M

[mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

ibrahimsaf commented 6 years ago

Download mysql JDBC: wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.44.zip yum install unzip unzip mysql-connector-java-5.1.44.zip Copy this driver to all nodes: mysql-connector-java-5.1.44/mysql-connector-java-5.1.44-bin.jar cp mysql-connector-java-5.1.44/mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar

Configure the master:

Start MySQL server: /bin/systemctl start mysqld.service

Configure MySQL: [root@ip-172-31-43-182 ~]# /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current password for the root user. If you've just installed MySQL, and you haven't set the root password yet, the password will be blank, so you should just press enter here.

Enter current password for root (enter for none): OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL root user without the proper authorisation.

Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success!

By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment.

Remove anonymous users? [Y/n] Y ... Success!

Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y ... Success!

By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment.

Remove test database and access to it? [Y/n] Y

Reloading the privilege tables will ensure that all changes made so far will take effect immediately.

Reload privilege tables now? [Y/n] Y ... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MySQL installation should now be secure.

Thanks for using MySQL!

On the master

mysql> GRANT REPLICATION SLAVE ON . TO 'replica'@'ip-172-31-7-61.eu-central-1.compute.internal.' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

mysql> SET GLOBAL binlog_format = 'ROW'; Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH TABLES WITH READ LOCK; Query OK, 0 rows affected (0.00 sec)

mysql> show master status; +-------------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +-------------------------+----------+--------------+------------------+ | mysql_binary_log.000003 | 1401 | | | +-------------------------+----------+--------------+------------------+ 1 row in set (0.00 sec)

On the slave server: Add in the /etc/my.cfg file: [mysqld] server-id=2

Start mysql service

mysql> CHANGE MASTER TO MASTER_HOST='ip-172-31-4-12.eu-central-1.compute.internal.', MASTER_USER='replica', MASTER_PASSWORD='azerty', MASTER_LOG_FILE='mysql_binary_log.000003', MASTER_LOG_POS=1401; Query OK, 0 rows affected (0.00 sec)

mysql> start slave; Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G 1. row Slave_IO_State: Waiting for master to send event Master_Host: ip-172-31-4-12.eu-central-1.compute.internal. Master_User: replica Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql_binary_log.000003 Read_Master_Log_Pos: 1401 Relay_Log_File: mysqld-relay-bin.000002 Relay_Log_Pos: 260 Relay_Master_Log_File: mysql_binary_log.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1401 Relay_Log_Space: 417 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 1 row in set (0.00 sec)

The replication is working

ibrahimsaf commented 6 years ago

Installation of the cluster: get the repo file here: https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo Change it to the right version: [cloudera-manager]

Packages for Cloudera Manager, Version 5, on RedHat or CentOS 7 x86_64

name=Cloudera Manager baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.8.3/ gpgkey =https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1

Install cloudera manager: yum install cloudera-manager-daemons cloudera-manager-server

Create all needed databases: mysql> create database scm DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec)

mysql> grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

mysql> create database amon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec)

mysql> create database hive DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec)

mysql> create database hue DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec)

mysql> create database oozie DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec)

mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'azerty'; Query OK, 0 rows affected (0.00 sec)

Configure cloudera manager to access to the database; /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm azerty

Démarrer cloudera manager: service cloudera-scm-server start

Change the parcel link: https://archive.cloudera.com/cdh5/parcels/5.8.3/

ibrahimsaf commented 6 years ago

The inspector says that every thing is ok, perfect :) image

Configure the services on the 5 machines:

3 zookeeper instannces (odd number)

ibrahimsaf commented 6 years ago

Install parcel repo: Install httpd server and start it: yum install httpd service httpd start

Change the cloudera manager config to: http://18.194.214.188/cdh5.8.3/

mfernest commented 6 years ago

FWIW, this is far more detail than necessary for commenting an Issue. If the comments on the work are as detailed as the work product itself, you're not really summarizing things.

mfernest commented 6 years ago

See commit d4c6361

mfernest commented 6 years ago

🏆