Establish Dashboard Framework

[cing]

Design decisions need to be made about the HPC Dashboard before development can begin.

Namely, we must determine if an existing open-source dashboard framework suits the goals of this web application. The dashboard framework must have:

• Support for real-time access to data through a RESTful API (using something like jQuery?)
• Support for modern javascript plotting libraries (not just restricted to gauges/pie charts/bar graphs)
• Support for authentication (this could be added later?)
• Support for resizable dashboard elements (some elements should be larger than others)

There are numerous dashboard frameworks on Github that should be considered (but are not limited to):

• Dashing.io
• JSPlate
• Dash-Ku
• Angular Dashboard Framework

It's possible that the actual layout/UI of the application will be largely handled by the dashboard framework, but I am not exactly sure about this.

[zyxue]

The NEWT API looks interesting, but still not sure how it works? e.g. for authentication, does it needs coordination from SciNet?

[cing]

Let's discuss the NEWT API in a separate issue about the communications protocol, but yes, the dashboard will require a NEWT server to be running on the supercomputer side, (I've already discussed this with our local supercomputer group, SciNet, but they won't have this running for some time). As I will discuss in the communications issue, the NEWT server can also be run locally for developers so that's not really an issue that a supercomputing group be involved at this point.

[cing]

I'll just clarify that when I said the framework must support authentication, I just meant we need a login prompt to the front-end for administrators so you could add/remove dashboard widgets. I suspect this is pretty universal for the dashboard frameworks, but I'll have to verify.

[zyxue]

Have you looked into D3.js, which seems to be very popular nowadays, but I am not sure if it fits well for dashboard.

[kinow]

Happy Dashing user here. Just set up a server to monitor some services at work, so here's some quick pros and cons:

• +1 due to its scheduler you can write jobs that pull data from any source
• +1 to the widget API, that lets you reuse widgets and create new ones
• +1 for its speed. IIUC, the browser receives notifications from the server, avoiding polling
• -1 exactly to the widget API. Had to write some widgets myself, and it's a PITB since you need to know JS, CoffeeScript, SCSS and HTML
• -1 to the deployment. My favorite distro is Ubuntu, but most of the time I'm working on corporate RHEL or CentOS servers. Ruby is not easy to set up and manage/upgrade in these environments (not as easy as Python, Java, Groovy, etc)
• -1 to testing. My workflow is usually have irb in a terminal, for quick checks of syntax, edit the jobs/.rb, edit dashboards/.erb, stop thin server, start dashing again, see what's broken, start all over again. I think there must be a better way of doing it.

All in all, I've been using it for about 2 years, works very well, has many widgets available, good documentation and if you know a little bit of Ruby and JavaScript/HTML/CSS it is not so hard to maintain an instance running somewhere.

[cing]

I recently had some fun doing a blog post/tutorial on building a front end and back end for data visualization that uses Keen.io/MongoDB/D3.js. Pretty cool. I also talked to Matthew Hanlon at the Texas Advanced Computing Center, and his team also built a dashboard with keen.io/angular.js and connected it with usage of their fancy HPC APIs (called Agave) without much trouble.

Dashing seems like a really active project with some cool modular widget support, but maybe that's too much? Keen.io dashboards seem like a nice framework that I can wrap my head around and doesn't lock us into using Coffeescript or Ruby. If we're using scientific plotting libraries like Bokeh we might not need to worry so much about the details of the widgets yet, not to mention the whole pain of making widgets that @kinow is describing. I talked to Peter Wang, one of the developers at Continuum Analytics, who said: "It's worth pointing out that some dashboard frameworks are somewhat "opinionated" about how they handle data, paginate it, etc. Bokeh is designed to be able to handle all this natively itself, and so I think most of the work is probably in figuring out this data source integration thing.".

I don't think the scheduler is something we're going to need with this project. There are no background processes that need to run, we just need to make some API calls to get data every now and then.

At the upcoming Mozilla Toronto sprint on this project, I'm going to put Keen.io dashboard this forward as the best framework for this application.

[zyxue]

This may not be super relevant.. Hadoop , Spark, ElasticSearch all have respective web interface for monitoring the clusters they're on, Hadoop => Ambari , Spark => port:4040, ElasticSearch => Kibana, which I feel is because they have built good APIs during their development. But I don't know if there are any such APIs for building web applications around HPC setup. This to me appears to be exactly what NEWT is trying to do, but it looks still rough, likely because in the old days, web was still not very popular among HPCers.