create cluster with Ambari without Docker container successfully

[jentle]

Hi, guys. It is able to use ambari web UI or REST API to build a cluster in AWS. You can view the template at http://54.86.210.191:8080

The instance is CentOS 6.3 64 bits, 1GB RAM.

1. Download repo and setup Ambari server in one of the instance. http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.1.0/bk_Installing_HDP_AMB/content/_download_the_ambari_repo.html
2. Create other instances as slave nodes, enable ssh login as root
3. When create the cluster with ambari, register hosts with AWS instance private DNS, paste the public key to enable SSH. Ambari will handle the installation and register all the host instances. or you can choose install the ambari-agent in each instance and register ambari-server hostname, then start.
4. Select componets and services , launch the cluster.

TIPS: It would be much easier to find a way to hide the ambari-agent docker under a hostname known to agent-server. Simply register in Ambari-server.

Some point to care : Ambari 1.7.0 version seems stable at present, 2.1.0 version would cause some problem in different OS. Currently support and limit to CentOS 6, SLES11, Ubuntu 12. The RAM size of Ambari server would not sufficient to handle the operation of Cluster. It is better to use a enhanced instance. To running the Hadoop job, we have to attach storage to the instance.

[dragonfly90]

Where is the centos image ?

[jentle]

It is the instance OS，not docker image

在 2015年10月10日，上午10:47，Liang Dong notifications@github.com 写道：

Where is the centos image ?

― Reply to this email directly or view it on GitHub.

[jentle]

Hi, guys. I have partially resolved the problem and wish you could contribute further work.

You can follow the script here to reproduce or view the cluster at 54.84.28.157:8080.

Ambari Server In the server node, we will deploy the ambari server container and consul agent container (store all the cluster nodes). You can also deploy the consul in separate instance .

The consul and Ambari server is hosted at 172.31.55.213, public address 54.84.28.157

_launch the consul agent_

docker run -d -p 8500:8500 -p 8400:8400 -p 53:53/udp --name amb-consul -h amb-consul.service.consul  sequenceiq/consul -server -bootstrap -ui-dir /ui

-p 8500 , registering node using HTTP -P 8400, RPC -p 53, Expose DNS for other hosts -ui-dir, Enable the Web UI, you can visit using http://54.84.28.157:8500

_Setup the Ambari server_

docker run -d -p 8080:8080 -p 8441:8441 -p 8440:8440 -e BRIDGE_IP=172.31.55.213 --name amb-server -h amb-server.service.consul sequenceiq/ambari /start-server

-p 8080, Web UI -p 8440, 8441 for Ambari agent register and sent heatbeat BRIDGE_IP, config as the DNS server address for all the node, here is the consul agent IP address. here we only start the Ambari server

_Register Ambari server hostname to consul_

curl -X PUT -d "{\"Node\":\"amb-server\",\"Address\":\"172.31.55.213\", \"Service\":{\"Service\":\"amb-server\"}}" http://54.84.28.157:8500/v1/catalog/register

here we use the consul API to register the server node. We used the private IP address since the cluster is running inside my network.

next we create a duplicate for the server host named "ambari-8080", which we will used for ambari agent regisetering.

curl -X PUT -d "{\"Node\":\"ambari-8080\",\"Address\":\"172.31.55.213\", \"Service\":{\"Service\":\"ambari-8080\"}}" http://54.84.28.157:8500/v1/catalog/register

Ambari Agent We launch a docker container as Ambari agent named amb-client1. This is much easier.

docker run -d -p 8670 -e BRIDGE_IP=172.31.55.213 --name amb-client1 -h amb-client1.service.consul sequenceiq/ambari /start-agent

-p 8670, used for Ambari server send configuration to ambari agent BRIDGE_IP, config as the DNS server address for all the node, here is the consul agent IP address. here we only start the Ambari server.

Let's take a look at how it works.

#ambari agent initialization
$ vi /etc/ambari-agent/conf/ambari-agent.ini

[server]
hostname=ambari-8080.service.consul
url_port=8440
secured_url_port=8441

When we launch the container

ambari-agent start

the container resolve the hostname "ambari-8080.service.consul". Obviously, it is blind to this hostname. Have no choice, it turn to 172.31.55.213 via interface 53 for help. Amazingly, that is where consul located and we just registered ambari-8080 before. Which could be found in

$ vi /etc/resolv.conf
nameserver 172.31.55.213
search service.consul node.dc1.consul                                       

Next step is similar,register to consul of our client node

curl -X PUT -d "{\"Node\":\"amb-client1\",\"Address\":\"172.31.55.215\", \"Service\":{\"Service\":\"amb-client1\"}}" http://54.84.28.157:8500/v1/catalog/register

Create the cluster For testing, I just choose the web to install the cluster. In "install options", type in the client node "amb-client1.service.consul", it will be resolved by the server. Since we have ambari agent installed, do not choose SSH, and move next. We can see the node is successfully confirmed. Then you could install other service inside. You might receive many warnings with only 1 node and some ports not exposed. You can view or add more nodes just follow above steps. Quite easy.

[jessexu20]

Okay... will look into it some time tmr ...

[jessexu20]

So I suppose we are using the latest version with consul instead of the 1.7 ? correct? @jentle

[maxlpy]

I remember he used 1.7.

[jentle]

the version of doesnt matter， the consul and Ambai are in different docker and independent. The key point is you have to start server and agent separately. but notice that in /etc/Ambari-agent/conf/Ambari-agent.init, configure the host name of server correctly. Well， you can use the lattest version without causing other problem.

在 2015年10月11日，上午11:24，Jesse Xu notifications@github.com 写道：

So I suppose we are using the latest version with consul instead of the 1.7 ? correct?

― Reply to this email directly or view it on GitHub.

[maxlpy]

I can use my private IP to register to your consul.

[maxlpy]

1. Ambari server and agent docker image. The following command will pull the latest version, which includes Ambari server and Ambari agent. You can find the docker file from this Url: https://hub.docker.com/r/sequenceiq/ambari/ docker pull sequenceiq/ambari
2. Ambari consul's docker image. Url of this image is https://hub.docker.com/r/sequenceiq/consul/. docker pull sequenceiq/consul, or you can use this version v0.5.0-v6 You can try above command to get the latest version. But in my environment, it reports an error "can't find latest version's". In this situation, you can click on Tags button from URL page and select one version to specify it.

[dragonfly90]

@maxlpy I can use my private IP to register to your consul-- You mean you can add another client in your private ip? Is the client 3 added by you?

[jentle]

@maxlpy

Did you register your ambari agent successfully to the server? I cannot confirm you via the server. You can check the log inside your docker, did it received the heartbeat response. vi /var/log/ambari-agent/ambari-agent.log

[maxlpy]

I can register to consul. But I can't let server to create cluster on agent.

[maxlpy]

@dragonfly90 Yes, you can just set up a ambari agent, and then CURL to Jiang's consul.

[jentle]

@maxlpy The workflow should be that agent connects to server first. Then you register to consul of your agent. The second step always succeed because it uses a public address and just stores the address of your agent. You should check the agent log for details. I doubt the private IP address of the server is not visible to your agent. If it cannot connecting to the "amb-server.service.consul", you can configure the "/etc/ambari-agent/conf/ambari-agent.ini", set the server hostname as "54.84.28.157" which is the server public address. Restart your agent. You can setup the server in your network for testing.

[maxlpy]

@jentle My agent can't find ambari server. It has following warnings: INFO 2015-10-12 03:12:29,952 main.py:70 - loglevel=logging.DEBUG INFO 2015-10-12 03:12:29,952 DataCleaner.py:39 - Data cleanup thread started INFO 2015-10-12 03:12:29,954 DataCleaner.py:120 - Data cleanup started DEBUG 2015-10-12 03:12:29,954 DataCleaner.py:71 - Cleaning up inside directory /var/lib/ambari-agent/data INFO 2015-10-12 03:12:29,954 DataCleaner.py:122 - Data cleanup finished INFO 2015-10-12 03:12:29,960 PingPortListener.py:50 - Ping port listener started on port: 8670 WARNING 2015-10-12 03:12:49,983 main.py:291 - Unable to determine the IP address of the Ambari server 'ambari-8080.service.consul' INFO 2015-10-12 03:12:49,984 NetUtil.py:59 - Connecting to https://ambari-8080.service.consul:8440/ca WARNING 2015-10-12 03:13:10,001 NetUtil.py:82 - Failed to connect to https://ambari-8080.service.consul:8440/ca due to [Errno -3] Temporary failure in name resolution WARNING 2015-10-12 03:13:10,001 NetUtil.py:105 - Server at https://ambari-8080.service.consul:8440 is not reachable, sleeping for 10 seconds... INFO 2015-10-12 03:13:20,002 NetUtil.py:59 - Connecting to https://ambari-8080.service.consul:8440/ca WARNING 2015-10-12 03:13:40,019 NetUtil.py:82 - Failed to connect to https://ambari-8080.service.consul:8440/ca due to [Errno -3] Temporary failure in name resolution WARNING 2015-10-12 03:13:40,019 NetUtil.py:105 - Server at https://ambari-8080.service.consul:8440 is not reachable, sleeping for 10 seconds... INFO 2015-10-12 03:13:50,020 NetUtil.py:59 - Connecting to https://ambari-8080.service.consul:8440/ca WARNING 2015-10-12 03:14:10,036 NetUtil.py:82 - Failed to connect to https://ambari-8080.service.consul:8440/ca due to [Errno -3] Temporary failure in name resolution WARNING 2015-10-12 03:14:10,036 NetUtil.py:105 - Server at https://ambari-8080.service.consul:8440 is not reachable, sleeping for 10 seconds... INFO 2015-10-12 03:14:20,036 NetUtil.py:59 - Connecting to https://ambari-8080.service.consul:8440/ca WARNING 2015-10-12 03:14:40,051 NetUtil.py:82 - Failed to connect to https://ambari-8080.service.consul:8440/ca due to [Errno -3] Temporary failure in name resolution

[jentle]

That's make sense. For security, I only use private ip to group cluster. Your have to two options

1. register server with public IP, overwrite the old.
2. change the server hostname using public IP in /etc/ambari-agent/conf/ambari-agent.ini

[jentle]

@dragonfly90 @maxlpy on Client Server

change the rsa with your own pem

sudo -i sudo sed -i -e 's/#PermitRootLogin/PermitRootLogin/g' /etc/ssh/sshd_config

Replace the following middle part in "" with your machine's pem file in "/root/.ssh/authorized_keys"

sudo echo 'ssh-rsa "AAAAB3NzaC1yc2EAAAADAQABAAABAQDGOQUCMQqUCNT2xdFSJM9pr6dJWuT5oAvu+PrtqyUHxKKQ6GbwOusyt0XnmahpW4PND8YQCSHCgATOPMNGOw3JtTNjWdG4ApmWRLh4zR84vrI3FjUYE5JU3yZYe8j33gkV3iktrjZC4bb+QhsPB8ABbC+MMLUQgrCyEqBdrq305Fd313o7G6Ln1RBTuA9PzyoMYJwCkkYgpKNyeIiOxo/dX8f+Gi9ZxI2j/x3F2aw2Cy9FNSf01LPPVhNXz6fCySleiDHyoPwMxZde8BJ3NpWNZCBzAinc1ZuZWuuhjJQfdCgzierJbRqJ9ckDJw2xXyf5ooZ1G5DR18z/6gGxWfr5" pem-file-name > /root/.ssh/authorized_keys