snowch / biginsight-examples

Example projects to help you quickly get started with BigInsights
Apache License 2.0
7 stars 4 forks source link

README


NOTE: the examples in this repository are being migrated to https://github.com/IBM-Bluemix/BigInsights-on-Apache-Hadoop. Please head over to that URL to see if the example you wish to try has been migrated.


This repository contains example projects to help you quickly get started with BigInsights. Following the steps below on your client machine, it should take you less than 5 minutes to run any of the example projects against a BigInsights cluster. The projects are tested on BigInsights on Cloud (bluemix) but they should also work for BigInsights on-premise.

The core idea is that you can run an example project to see it working against your BigInsights cluster. You can then copy the project and adapt it to add your own custom logic. Think of the example projects as working blueprints. A design decision was taken that the build scripts should be as independent as possible to allow developers to take a standalone example script project and reuse it with minimal effort.

See the 5 minute demo at the end of the README to watch an example of setting up the biginsight-examples project and then run 'ls /' on the cluster using the Knox webHdfs API.

Also see this repository for example projects for Apache Spark on Bluemix.


Pre-requisites

Mandatory

Optional

NOTE: You do NOT need to install gradle, the gradlew scripts mentioned below will install gradle for you

Setup Instructions

See the FAQ for common issues and their resolution.

See the 5 minute demo at the end of the README to see these steps in action.

Now run the WebHdfsGroovy example to list the folders in hdfs:

Optional: run tests (this takes approximately 40 mins on my cluster)

Next, see the section below for details of running an example script.


Running the scripts

After performing the setup steps, here is the output from running the WebHdfsGroovy Ls example to list the files and directories in the hdfs root directory:

biginsight-examples snowch$ ./gradlew -p examples/WebHdfsGroovy Ls
:compileJava UP-TO-DATE
...

>> [app-logs, apps, biginsights, ibmpacks, iop, mapred, mr-history, secureDir, securedir, tmp, user]

>> Ls test was successful.

BUILD SUCCESSFUL

Total time: 4.899 secs

Each example project has a README.md file describing how to run the project. All of the example projects are available in the [examples] folder.


5 Minute Demo

To watch a 5 Minute Demo setting up the project and running an example, see here.


Smoke testing your cluster

The examples can be used to smoke test your cluster. For more information, see here.


Tips

For some tips using this project, for example with multiple clusters, see here.


Contributing

Contributions are welcome. Please send a pull request or contact me at chris.snow@uk.ibm.com to get involved.