plotting GUIs

[mpmdean]

I think it would be very helpful to provide users more tools to explore/plot data in an efficient way?

I'm talking about something that mirrors the capability of PyMCA http://pymca.sourceforge.net/

It is possible to bulid similar functionality from jupyter + ipympl + widgets with a pleasingly small amount of code https://github.com/NSLS-II-SIX/sixtools/blob/master/GUIs/scan_explorer.ipynb

And I'd be prepared to invest more effort in developing this, but first I would like to hear more opinions about whether this is the right way to go.

NSLS-II/qtbroker which aimed to do some of this seems not to under development. @danielballan I assume this is right?

[danielballan]

We have made a couple experimental forays into this, none of which we have pushed very hard. However, we feel we finally have enough infrastructure now to do this now and delivering something for all beamlines by the first 2019 cycle is our top priority for that cycle.

There's a lot of background here, probably easier to discuss on the phone. If you are interested in getting up to speed on our direction and plans, I'd be happy to chat about it.

[mpmdean]

Hi @danielballan

I just wanted to send a nudge regarding this issue. Is 2019-1 still the sort of timescale that we can think about? And is there a repo I can look at?

[danielballan]

Thanks for the ping, @mpmdean. Around December 1, we reluctantly decided that it would take a suicidal amount of work to push all the pieces across the finish line in time for the start next cycle. We do intend to deploy it mid-cycle to interested beamlines.

The path is:

• Rework the blukesy callbacks so that they can be used for the plotting GUI (not just for live plotting). There are several aspects to this, including speed and great flexibility. That work is happening in this PR: https://github.com/NSLS-II/bluesky/pull/1076
• Rework databroker to use a project called intake, which has a similar goal to databroker and is considerably further along in its development. This is what the GUI will use to search and access data. Using intake solves a bunch of problems for us that we would otherwise have to address ourselves, including handing us the client--server we talked about during our last phone conversation. The GUI will be able to seamlessly access data locally or remotely; we get that basically "for free" with intake. You can get a sense of how that looks here: https://github.com/danielballan/intake-bluesky/blob/master/demo.ipynb
• Many users also want to export data from the GUI. This is not a blocking issue for a minimal "plotting GUI" / browser, but we're working on it in parallel because it has started to feel urgent given the unreliabilty of remote access. That work is proceeding in this repo: https://github.com/NSLS-II/suitcase-csv/ Once we are happy with the shape of the CSV exporter, we'll knock out just a couple others and declare victory for now.

It's been difficult to get our heads around all these pieces, but now we have a path and I expect we can get something in (brave) users' hands in weeks, not months.

[mpmdean]

Thanks @danielballan !

[danielballan]

Status update on this:

• The work on exporters landed a couple weeks ago. We have a solid architecture and support for 5 formats with more in progress. See https://nsls-ii.github.io/suitcase/ . As noted above, this is not strictly required for "plotting GUIs" but is often requested at the same time due to the unreliability of remote access.
• The work of rewriting databroker on top of intake is well underway. As mentioned in my previous comment, intake is an external project with similar design goals to databroker that is further developed in many respects. Our prototype integrating intake with bluesky/databroker now supports progressive search, remote access (i.e. an HTTP client--server), and a more natural syntax for accessing data "lazily". The remaining step --- an important one --- is to ensure that the existing databroker API still works the same. We don't want to break working notebooks! Users will be able to choose between the databroker interface they know and an alternative that is essentially intake. Documentation in progress is here: http://nsls-ii.github.io/intake-bluesky . Per @stuartcampbell's direction, we are going to prioritize robust data access (this intake work) as it's important to have a solid foundation before building a GUI on that foundation.
• The work of updating the bluesky plotting callbacks to be sufficiently flexible and fast on saved data is still underway in NSLS-II/bluesky#1076. We made significant progress in the last ~4 weeks, but it won't likely be a main focus until the intake work is 90% landed.

[mpmdean]

Thanks for the update @danielballan

[danielballan]

We have a prototype ready for early feedback: https://github.com/NSLS-II/bluesky-browser#launching-the-demo