search

IBMStreams / streamsx.dps

The IBMStreams/streamsx.dps GitHub repository is home to the Streams toolkit named DPS (Distributed Process Store)
http://ibmstreams.github.io/streamsx.dps
Other
4 stars 9 forks source link
distributed-computing ibm-streams stream-processing toolkit

readme

Overview

The IBMStreams/streamsx.dps GitHub repository is home to the Streams toolkit named DPS (Distributed Process Store). This Streams toolkit allows a simple way for the SPL, C++ and Java operators belonging to a single or multiple applications to share the application specific state information via an external K/V store. It does this via a collection of APIs that can be called from any part of the SPL, C++ and Java operator code.

Following are the external NoSQL K/V stores that can be configured to work with the DPS toolkit for the purpose of sharing application state in a distributed manner.

  1. Memcached
  2. Redis [version 2.x that doesn't have a built-in cluster feature]
  3. Cassandra
  4. IBM Cloudant
  5. HBase
  6. Mongo
  7. Couchbase
  8. Redis-Cluster [New cluster feature is available in Redis version 3 and above]

This toolkit evolved from the early research work done at the IBM.T.J.Watson Research Center, Yorktown Heights, New York. The links below highlight the origins of this asset.

Technical Positioning

Podcast

Documentation

For an API description see the GitHub pages for this toolkit. The documentation is also available locally in the doc folder, after building the toolkit. For configuration hints and other information, see the Wiki pages

Other things to consider when working with this toolkit:

Building the toolkit

To build the toolkit perform the following steps:

  1. Clone the repository
    git clone https://github.com/IBMStreams/streamsx.dps.git
  2. Build the toolkit
    make clean all

Prerequisuites
you need to have the following rpm packages installed on the build machine:

curl
curl-devel
lua
lua-devel
openldap-devel
openssl-devel
cyrus-sasl
cyrus-sasl-devel

Learn more about Streams