search

pomeslab / matplotdash

Web dashboard for monitoring real-time scientific data
MIT License
7 stars 2 forks source link

Establish Backend Communication Protocol with Supercomputer #2

Open cing opened 9 years ago

cing commented 9 years ago

Our dashboard application needs to pull data about running compute jobs from a supercomputer. This messy situation (including all security concerns) is handled by HPC APIs like NEWT, Agave, and OpenLorenz.

I have communicated with the NEWT team about their development timeline and I'd like to go ahead with using their API. NEWT is under active development and is designed to be portable to different supercomputers. It is also very flexible and server-side plugins can be written to emit whatever JSON we want. Revised API docs for NEWT 2.0 will be released before for an upcoming supercomputing conference, but unfortunately they are not ready at the time of the Mozilla Science launch.

Once NEWT 2.0 launches, we will not immediately have a NEWT server running on a supercomputer for testing. Cooperation with my local supercomputing group (SciNet) to install the NEWT server will take time. Developers will have to setup a testing NEWT server. We can setup a NEWT server locally or I could potentially host one on a Heroku account. The NEWT server is written in Python using Django so it isn't difficult to install, but I'm not sure what kind of configuration needs to be done.

Any discussion about the communication between the Dashboard and supercomputer can go here.

cing commented 9 years ago

This project has shifted direction a bit since I first suggested it back in October. Although the primary application of this project for use with data sources on HPC/supercomputers still rings true, I'm hoping this code will be general enough to support scientific plotting from any data source. For instance, data generated by a real-time sensor could be queried just the same as a database within the confines of a supercomputer. Both Agave and NEWT are robust APIs for interacting with supercomputers, but the primary communication will hopefully be more general (like all the other Dashboards).

zyxue commented 8 years ago

It would be interesting if NEWT could adapt the linux authentication system directly (https://docs.djangoproject.com/en/1.9/topics/auth/customizing/#other-authentication-sources), and use that directly for users to login and see their job information.