search

ideonate / cdsdashboards

JupyterHub extension for ContainDS Dashboards
https://cdsdashboards.readthedocs.io/
Other
200 stars 38 forks source link

API Endpoint for Adding/Updating Dashboards #59

Open rcaskey-dtn opened 3 years ago

rcaskey-dtn commented 3 years ago

I've looking into adding a method to bulk create/start dashboards in cdsdashboards.

I took a quick look around hubextension/api and it looks like there are api endpoints for listing and deleting dashboards but nothing pre-existing for creating or updating dashboards - that I could add my own endpoints for adding/starting/stopping dashboards there but that it was not currently implemented.

Is anyone doing anything similar? Are there others interested in this sort of thing? Known gotchas to beware of?

danlester commented 3 years ago

Thank you for sharing your ideas.

There certainly isn't something available in cdsdashboards at present to do this, as you have found. To be honest, I'm not aware of anyone mentioning any need to access dashboards via APIs - other than for the ContainDS JupyterLab extension, and this is fairly lightweight (getting a list of dashboards, and potentially redirecting the user to the edit dashboard form with some values already populated).

It would be useful to be able to extend JupyterHub's core REST API, but that's not possible at the moment as far as I know.

It would be really interesting to understand more about your setup, and why a dashboards create/start API would be useful to you.

rcaskey-dtn commented 3 years ago

We have two issues at play on our end.

Our main issue/characteristic is that we deploy jupyterhub from CICD and manage it in a stateless fashion - the database it uses is temporary and doesn't persist through routine updates. Any state that persists, we closely manage on a carefully curated case-by-case basis. As our operations grow, we anticipate dashboards will run the dev/stg/prd gauntlet and eventually be deployed to multipler jhub instances so that's one reason we put up with the complication that not having a persistent database brings. Exposing the creation of the dashboard via api instead of via a local tool also helps us in that it provides a way for the CICD server to cleanly deploy dashboards directly from git to various target environments.

We have quite a few dashboards and we need a way to start them after jupyterhub reboots (mainly redeployments) since many times the primary consumer is a non-technical decision maker in another department who is not the creater or owner of the dashboard and many times doesn't even know they are using a resource proxied by jupyterhub.

I spent most of yesterday getting my bearings with jupyterhub development and doing some minor re-factoring of the edit handler to get it to follow the same pattern as the handlers that are already been exposed via the API and threw together a working API this morning that I was able to use successfully to create/launch a dashboard. On my branch BasicDashboardEditHandler(BasicDashboardEditMixin) is currently 29 lines.

danlester commented 3 years ago

Thank you for explaining - very interesting.

A transient JupyterHub like this isn't something I've come across before. I don't think it's a use-case I'd expect to support directly, but there's no reason you can't adapt things (as you already have!)

Another possibility could be to run SQL for the dashboards directly into the database before JupyterHub starts.

In general though, I wonder if it would make more sense in your case to develop a CICD pipeline for standalone individual dashboards running in a web server directly (so no JupyterHub). The missing piece then becomes authentication, and that depends on who/where your users are.

That's something I encounter a lot, and often the source control part becomes a burden for one-off temporary dashboards, which is why they move towards ContainDS Dashboards. But in your case, if CICD is still required, Dashboards doesn't really make things easier for your dashboard creators.

Just some thoughts, but of course it's all about the details.

MarcSkovMadsen commented 3 years ago

Listining in. I also would like the dashboards to persist when my jupyterhub images are updated.

MarcSkovMadsen commented 2 years ago

For me this is still the gordian knot to solve. How can i programmatically create or Update dashboards to run on the JupyterHub?

Can it be done from the commandline in jupyterlab? Or from Python code? Or from a rest API call?

Automation is important for robustness. Integration with jupyterhub important to keep it simple and Secure.