Updated plan for best-effort callback rewrite

[danielballan]

The Best-Effort Callback was introduced as an experimental feature. It automatically builds instances of LiveTable, LivePlot, and others to produce a "best effort" visualization of data as it comes it. It can also be used to visualize saved data from databroker.

Known Limitations

• The plotting callbacks aren't sufficiently configurable. For example, you can't ask LivePlot to plot one column divided by another column.
• The plotting produces many matplotlib figure windows, and it gets messy. A tabbed layout in a single application would be nicer.
• We are missing a solid live image and image stack viewer.
• The Best-Effort Callback is not extensible; there is no easy way to add user-defined callbacks or modify the default ones.

Plan

We will release a new package, tentatively bluesky-mpl ("mpl" for "matplotlib") that includes:

• A new take on LivePlot and friends that are more powerful, able to do things like plot derived quantities
• Some optional Qt components to provide a tabbed layout for the Best-Effort Callback plots.
• Additional plot types, most importantly image viewing
• A configuration system for extending the Best-Effort Callback

Much of this is already prototyped in https://github.com/bluesky/bluesky-browser but that project will be archived out of concern that it has a difficult-to-nail-down scope and overlaps to some degree with functionality that Xi-cam endeavors to cover. Some of the pieces of bluesky-browser have already been moved into Xi-cam. The pieces related to plotting and more generally useful and will go in bluesky-mpl.

By developing this "new take" on the Best-Effort Callback and its components in a separate package, we will be able to put it on a separate release schedule than bluesky itself, likely issuing frequent releases at first. We have long discussed the advantages of separating the core of bluesky (RunEngine, plans) from callbacks. Now seems like a good time to make that pivot.

The existing LivePlot and friends will be maintained in bluesky for backward-compatibility and refactored #859 to use components from bluesky-mpl.

[prjemian]

Also, it is very useful to have the PeakStats to be evaluated from a bp.scan(), even when the LivePlot is disabled. Is this possible? Seems that the logic to detect x & y axes is shared with LivePlot.

Take the case from our USAXS instrument where they want to observe a slow trend (such as nucleation and growth in a metal alloy) in a particular scan. Enabling occasional traces (in the LivePlot) is more informative than continuous for such slow trends. They periodically enable LivePlot as desired (say, once out of every 20 scan repetitions), then disable it until the next graph update is selected. That allows an overnight run and an observation of trends without having too much information. They can estimate how many scans will complete overnight, thus at what interval to sample for plots.

[danielballan]

After a couple false starts, including the now-archived bluesky-browser and bluesky-mpl, we have a path forward on this that is sticking. See http://blueskyproject.io/bluesky-widgets/. We have deployed it on a trial basis at a couple NSLS-II beamlines and one APS beamline so far. Feedback has been positive. When it reaches feature parity with the Best-Effort Callback, we will recommend all users migrate.

[stan-dot]

@danielballan where do we track the features to notice feature parity?