search

IBMStreams / administration

Umbrella project for the IBMStreams organization. This project will be used for the management of the individual projects within the IBMStreams organization.
Other
19 stars 10 forks source link

streamsx.visualization: Proposal for rapid prototyping of advanced real-time visualizations from Streams #94

Closed sriumcp closed 8 years ago

sriumcp commented 8 years ago

The Problem

Javascript based real-time visualizations have undergone a revolution in recent years. D3 and its derivatives dc.js, NVD3, n3-charts, Cubism and Vega, and libraries like leaflet.js, sigma.js, cytoscape have made it possible to create stunningly beautiful and reactive real-time visualizations with dynamic data.

In theory, Streams and real-time visualizations are a perfect fit. In practice, creating beautiful visualizations like this and this with data backed by Streams continues to be a difficult task because:

  1. There is a steep learning curve associated with many of these javascript visualization libraries
  2. Even a developer skilled in Streams and web development faces significant effort in terms of:
    • Setting up the html / css needed by the dashboard
    • Setting up the javascript logic for continuously acquiring data from Streams, reactively transforming this data to fit the visualization format, and reflowing the visualizations reactively in real-time
    • Setting up the javascript logic for handling user interactions like clicks, mouse-overs, and selections

      The Proposal

I propose a web application called READ that will make it possible to create beautiful advanced real-time visualizations and dashboards backed by data from Streams in minutes. I propose streamsx.visualization as the name of the repository that will host READ.

Target Users

READ is a web application for Streams application developers who want to communicate real-time insights from their Streams applications using advanced real-time visualizations and dashboards.

Business Value

Streams application developers will be able to rapidly prototype real-time dashboards without the pain involved in real-time dashboard development. READ is especially useful in Streams PoCs: with READ, the PoC developer can focus almost exclusively on the Streams application logic and performance and with minimal effort, easily and effectively convey Proof-of-Value of Streams applications through real-time visualizations and dashboards. I fully anticipate READ becoming a standard weapon in every Streams application developer’s arsenal.

Features

The checked features will be available as part of the initial release. The unchecked features are planned for future releases.

Screenshot of a READ Dashboard

alt text

Related Projects

  1. Active Sheets: Connect Streams to Microsoft Excel for real-time visualization of Streams data using Excel’s visualization capabilities.
  2. A proposal #59 by @chanskw for a repo called streams.visualization which has since been abandoned due to lack of resources. The scope of READ is different from #59.
ddebrunner commented 8 years ago

+1000

Just to note that HTTPTupleView has a easy to use default live map visualization if a stream contains lat/long positions. Also has the ability to set the icon to display and provide a pop-up note. Could READ include the option to display these maps, or support live maps itself?

@sriumcp Also, anything improvements needed for HTTPTupleView and its associated operators, please enter issues at streamsx.inet.

sriumcp commented 8 years ago

Could READ include the option to display these maps, or support live maps itself?

Absolutely. The proposed leaflet.js integration is all about maps and geo visualizations (e.g., using topo json / geo json / lat-long data)

rzbhatti commented 8 years ago

I agree with Sri. I have personally used the HTTPTupleView operator and integrated it with the READ visualizations for realtime system monitoring. It so easy and agile. It would be great to have more tight coupling of READ to Streams - through so READ visualization toolkit for Streams.

One-More thing I did not have any javascript experience I just learned it on the fly...

ddebrunner commented 8 years ago

is READ a third party component or original code for this toolkit?

sriumcp commented 8 years ago

is READ a third party component or original code for this toolkit?

Everything that will be hosted in the streamsx.visualization repository is original code. READ has multiple dependencies, but none of the third-party READ dependencies will be included as part of the streamsx.visualization repo; the user will install them from their respective repos as part of the process of getting started with READ by using the npm and bower package managers.

Here is the exact set of steps / commands the user will follow to install READ.

  1. Download and install node and npm: https://nodejs.org/
  2. Use npm to download and install bower: npm install -g bower
  3. Use git to download original READ code from the IBMStreams/streamsx.visualization repo: git clone https://github.com/IBMStreams/streamsx.visualization.git ()
  4. Install dependencies
    • cd streamsx.visualization
    • Use npm to download and install npm dependencies for READ from their respective repos - not from the IBMStreams/streamsx.visualization repo: npm install
  5. Install more dependencies
    • cd streamsx.visualization/frontend/flowboards
    • Use bower to download and install bower dependencies for READ from their respective repos - not from the IBMStreams/streamsx.visualization repo: bower install

READ is ready to be launched at this point.

chanskw commented 8 years ago

Can you please give me the list of the third-party dependencies?

sriumcp commented 8 years ago

https://gist.github.com/sriumcp/c438cd915482a9825e8e51690677f6aa

This gist has two files.

  1. package.json lists all the npm dependencies under the dependencies property (devDependencies are optional)
  2. bower.json lists all the bower dependencies under the dependencies property (devDependencies are optional)

None of the dependencies will be included within the streamsx.visualization repository.

sriumcp commented 8 years ago

I checked the licenses of all the READ dependencies (both npm and bower dependencies). The dependencies have one of the following five licenses: MIT, Apache 2, BSD 2, ISC or OFL. The last one is the license for font-awesome - a font that READ depends on.

leongor commented 8 years ago

+1

deepakturaga commented 8 years ago

+1 We have already started to use this with some of our customer PoCs - and it can really accelerate the PoC

cancilla commented 8 years ago

+1 Is the platform capable of visualizing time series data such that there is no data loss? Can you elaborate on how this works?

sriumcp commented 8 years ago

visualizing time series data such that there is no data loss

@cancilla Could you please describe the scenario you have in mind with an example so that I can describe how READ might work for that scenario?

We have so far used two ways to connect READ to Streams applications:

  1. Use the HTTPTupleView to set up a GET end-point within Streams. Use READ to access all of the data losslessly that is backed by this end point and plot it. This data access and chart refresh can occur periodically based on a frequency that you set within READ.
  2. Use the HTTPPost operator from Streams to POST data losslessly to a POST end-point within READ. The upstream operator(s) within Streams determines what data needs to be pushed out through HTTPPost. Streams controls the frequency of data refresh and chart refresh in this case.

HTTPTupleView and HTTPPost by themselves are mechanisms for exposing Streams data (e.g., the data within an aggregation window) to the outside world (in this case, READ).

mikespicer commented 8 years ago

+1

chanskw commented 8 years ago

+1

chanskw commented 8 years ago

created streamsx.visualization repository. @sriumcp is the initial committer.