search

abajwa-hw / ambari-flink-service

Ambari service for Apache Flink
125 stars 111 forks source link

readme

An Ambari Service for Flink

Ambari service for easily installing and managing Flink on HDP clusters. Apache Flink is an open source platform for distributed stream and batch data processing More details on Flink and how it is being used in the industry today available here: http://flink-forward.org/?post_type=session

The Ambari service lets you easily install/compile Flink on HDP 2.6.5

Limitations:

Author: Ali Bajwa

Setup (中文访问 https://blog.csdn.net/jzy3711/article/details/104043860)

non sandbox

sudo service ambari-server restart


- Then you can click on 'Add Service' from the 'Actions' dropdown menu in the bottom left of the Ambari dashboard:

On bottom left -> Actions -> Add service -> check Flink server -> Next -> Next -> Change any config you like (e.g. install dir, memory sizes, num containers or values in flink-conf.yaml) -> Next -> Deploy

  - By default:
    - Container memory is 1024 MB
    - Job manager memory of 768 MB
    - Number of YARN container is 1

- On successful deployment you will see the Flink service as part of Ambari stack and will be able to start/stop the service from here:
![Image](../master/screenshots/Installed-service-stop.png?raw=true)

- You can see the parameters you configured under 'Configs' tab
![Image](../master/screenshots/Installed-service-config.png?raw=true)

- One benefit to wrapping the component in Ambari service is that you can now monitor/manage this service remotely via REST API

export SERVICE=FLINK export PASSWORD=admin export AMBARI_HOST=localhost

detect name of cluster

output=curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' http://$AMBARI_HOST:8080/api/v1/clusters CLUSTER=echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'

get service status

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X GET http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE

start service

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Start $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE

stop service

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE


- ...and also install via Blueprint. See example [here](https://github.com/abajwa-hw/ambari-workshops/blob/master/blueprints-demo-security.md) on how to deploy custom services via Blueprints

#### set Flink version
- configuration/flink-ambari-config.xml
flink_download_url http://X.X.151.15/Package/flink-1.9.0-bin-scala_2.11.tgz Snapshot download location. Downloaded when setup_prebuilt is true 
value from [http://apachemirror.wuchna.com/flink/](http://apachemirror.wuchna.com/flink/) or [http://www.us.apache.org/dist/flink/](http://www.us.apache.org/dist/flink/)  or [https://archive.apache.org/dist/](https://archive.apache.org/dist/)or customize repo

- metainfo.xml
FLINK 
        <displayName>Flink</displayName>
        <comment>Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.</comment>
        <version>1.9.0</version>
 vsersion = your flink version

#### Flink on Yarn
- metainfo.xml
yarn.client.failover-proxy-provider org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 
 restart yarn

 #### Flink Configuration
 ![Image](../master/screenshots/Flink-conf.png?raw=true)
- java_home is consistent with / etc / profile

hdp-select status hadoop-client hadoop-client - 

- hadodp_conf_dir = /etc/hadoop/<version>/0

#### Use Flink

- Run word count job

su flink export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_CLASSPATH=hadoop classpath cd /opt/flink ./bin/flink run --jobmanager yarn-cluster -yn 1 -ytm 768 -yjm 768 ./examples/batch/WordCount.jar

- This should generate a series of word counts
![Image](../master/screenshots/Flink-wordcount.png?raw=true)

- Open the [YARN ResourceManager UI](http://sandbox.hortonworks.com:8088/cluster). Notice Flink is running on YARN
![Image](../master/screenshots/YARN-UI.png?raw=true)

- Click the ApplicationMaster link to access Flink webUI
![Image](../master/screenshots/Flink-UI-1.png?raw=true)

- Use the History tab to review details of the job that ran:
![Image](../master/screenshots/Flink-UI-2.png?raw=true)

- View metrics in the Task Manager tab:
![Image](../master/screenshots/Flink-UI-3.png?raw=true)

#### Other things to try

- [Apache Zeppelin](https://zeppelin.incubator.apache.org/) now also supports Flink. You can also install it via [Zeppelin Ambari service](https://github.com/hortonworks-gallery/ambari-zeppelin-service) for vizualization

More details on Flink and how it is being used in the industry today available here: [http://flink-forward.org/?post_type=session](http://flink-forward.org/?post_type=session)

#### Remove service

- To remove the Flink service: 
  - Stop the service via Ambari
  - Unregister the service

export SERVICE=FLINK export PASSWORD=admin export AMBARI_HOST=localhost

detect name of cluster

output=curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' http://$AMBARI_HOST:8080/api/v1/clusters CLUSTER=echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X DELETE http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE


If above errors out, run below first to fully stop the service

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop $SERVICE via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE


- Remove artifacts

rm -rf /opt/flink* rm /tmp/flink.tgz