Any plans for a visualization tool for databroker?

[prjemian]

This query at APS today from @jphammonds expresses the basic question:

Are there any tools to visually inspect the tables in the databases? For instance, I am setting up to monitor pvs during the run by setting monitors in the preprocessor’s Supplemental Data. Anything that allows browsing the runs (such as db[#]) and see the items available for plotting?

I am also recording some items at certain points in the cycle. Any way to view this?

I would assume a tool like this would simply live outside of bluesky & examine the mongo db independently.

For example, bluesky_snapshot_viewer (a GUI created by: apstools.callbacks.SnapshotReport()) is a tool to visualize a specific type of scan (a bluesky_snapshot)

[prjemian]

@jphammonds:

I have used a tool that allows viewing (and editing!) of the mongodb. It knows nothing about databroker. It's the only raw mongodb viewing I am using now. Certainly there must be others but this one was free and available when I looked. So, I put it on our APS common file server. (There is also a .bat version of the starter for Windows.)

Caution: Since it allows editing, be very judicious in how and when you use this!

You run mViewer by starting a local web server with a port number:

path/to/start_mviewer.sh 8086

access it from a browser session:

/bin/google-chrome http://localhost:8086/index.html

mongodb viewer hints: host: port: 27017 others leave blank"

So far, there is a big opportunity for someone to make a plotting tool. None exists now, so I believe.

You have probably found the databroker tutorial: http://nsls-ii.github.io/databroker/tutorial.html

[danielballan]

I suggest starting with the simple:

db[...].table(stream_name='...').plot()

Separately, we are working on a GUI that sits on top of databroker and lets you search, browse, and visualize Runs. I spent a couple hours on it today, in fact.

We plan to build our tools on top of the databroker API and avoid interacting with Mongo directly so that the tools we write can work equally well on top of any storage backend. We have some prototypes that provide a db object on top of a directory of local files, an HTTP client, etc.

[JPHammonds]

Dan,

I have tried the plot as you have suggested below but I am getting errors with this. A plot window pops up but it is empty.

I am able to list the table:

In [10]: db[-1].table(stream_name="depSys_gun_selector_mps1_voltage_rbv_monitor") Out[10]: time depSys_gun_selector_mps1_voltage_rbv seq_num 1 2019-04-16 16:26:15.623624086 2.8 2 2019-04-16 16:26:15.953511953 2.7 3 2019-04-16 16:26:16.728308678 2.8 4 2019-04-16 16:26:17.556593180 2.7 5 2019-04-16 16:26:19.191735268 379.8 6 2019-04-16 16:26:20.172238111 373.1 7 2019-04-16 16:26:21.005825043 372.8 8 2019-04-16 16:26:21.821944237 372.6 9 2019-04-16 16:26:22.647333145 372.5 10 2019-04-16 16:26:23.454028368 372.3 11 2019-04-16 16:26:24.260576248 372.2 12 2019-04-16 16:26:25.856822968 372.0 13 2019-04-16 16:26:26.659313917 371.7 14 2019-04-16 16:26:27.460757971 371.3 15 2019-04-16 16:26:28.245575190 373.5 16 2019-04-16 16:26:29.042916298 374.9 17 2019-04-16 16:26:29.891757250 370.7 18 2019-04-16 16:26:30.712952137 370.3 19 2019-04-16 16:26:31.478217125 370.2 20 2019-04-16 16:26:32.246970654 370.1 21 2019-04-16 16:26:33.029160023 370.0 22 2019-04-16 16:26:34.624567747 372.3 23 2019-04-16 16:26:35.452236652 374.2 24 2019-04-16 16:26:36.240039587 370.9 25 2019-04-16 16:26:37.073646069 370.8 26 2019-04-16 16:26:37.866647482 371.3 27 2019-04-16 16:26:38.682386875 371.4 28 2019-04-16 16:26:39.429312944 371.0 29 2019-04-16 16:26:40.259035587 370.9 30 2019-04-16 16:26:41.053616285 374.1 ... ... ... 57 2019-04-16 16:27:08.245858669 370.8 58 2019-04-16 16:27:09.029613972 370.0 59 2019-04-16 16:27:09.869656086 370.1 60 2019-04-16 16:27:10.643858671 370.0 61 2019-04-16 16:27:12.232585907 369.8 62 2019-04-16 16:27:13.074648380 370.2 63 2019-04-16 16:27:13.869327068 374.2 64 2019-04-16 16:27:14.632007122 372.8 65 2019-04-16 16:27:15.459666491 370.5 66 2019-04-16 16:27:16.241833925 371.0 67 2019-04-16 16:27:17.102390528 371.3 68 2019-04-16 16:27:18.688259363 371.1 69 2019-04-16 16:27:19.437182426 370.8 70 2019-04-16 16:27:20.284363270 372.7 71 2019-04-16 16:27:21.052206039 374.6 72 2019-04-16 16:27:21.891086102 370.5 73 2019-04-16 16:27:22.680588484 370.0 74 2019-04-16 16:27:23.463406324 370.1 75 2019-04-16 16:27:25.054121017 369.9 76 2019-04-16 16:27:25.880553722 369.8 77 2019-04-16 16:27:26.673506260 370.6 78 2019-04-16 16:27:27.444765806 374.8 79 2019-04-16 16:27:28.281283140 371.9 80 2019-04-16 16:27:29.083446503 370.6 81 2019-04-16 16:27:29.853262186 371.1 82 2019-04-16 16:27:30.626524687 371.3 83 2019-04-16 16:27:31.452813148 371.4 84 2019-04-16 16:27:32.278028011 199.0 85 2019-04-16 16:27:33.114491940 7.8 86 2019-04-16 16:27:33.916901350 2.7

[86 rows x 2 columns]

But when I try to plot I get the following:

In [16]: db[-1].table(stream_name="depSys_gun_selector_mps1_voltage_rbv_monitor").plot()

ValueError Traceback (most recent call last)

in () ----> 1 db[-1].table(stream_name="depSys_gun_selector_mps1_voltage_rbv_monitor").plot() ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in __call__(self, x, y, kind, ax, subplots, sharex, sharey, layout, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, secondary_y, sort_columns, **kwds) 2939 fontsize=fontsize, colormap=colormap, table=table, 2940 yerr=yerr, xerr=xerr, secondary_y=secondary_y, -> 2941 sort_columns=sort_columns, **kwds) 2942 __call__.__doc__ = plot_frame.__doc__ 2943 ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in plot_frame(data, x, y, kind, ax, subplots, sharex, sharey, layout, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, secondary_y, sort_columns, **kwds) 1975 yerr=yerr, xerr=xerr, 1976 secondary_y=secondary_y, sort_columns=sort_columns, -> 1977 **kwds) 1978 1979 ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in _plot(data, x, y, subplots, ax, kind, **kwds) 1802 plot_obj = klass(data, subplots=subplots, ax=ax, kind=kind, **kwds) 1803 -> 1804 plot_obj.generate() 1805 plot_obj.draw() 1806 return plot_obj.result ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in generate(self) 264 265 for ax in self.axes: --> 266 self._post_plot_logic_common(ax, self.data) 267 self._post_plot_logic(ax, self.data) 268 ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in _post_plot_logic_common(self, ax, data) 403 self._apply_axis_properties(ax.xaxis, rot=self.rot, 404 fontsize=self.fontsize) --> 405 self._apply_axis_properties(ax.yaxis, fontsize=self.fontsize) 406 407 if hasattr(ax, 'right_ax'): ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/pandas/plotting/_core.py in _apply_axis_properties(self, axis, rot, fontsize) 476 477 def _apply_axis_properties(self, axis, rot=None, fontsize=None): --> 478 labels = axis.get_majorticklabels() + axis.get_minorticklabels() 479 for label in labels: 480 if rot is not None: ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py in get_majorticklabels(self) 1186 def get_majorticklabels(self): 1187 'Return a list of Text instances for the major ticklabels' -> 1188 ticks = self.get_major_ticks() 1189 labels1 = [tick.label1 for tick in ticks if tick.label1On] 1190 labels2 = [tick.label2 for tick in ticks if tick.label2On] ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py in get_major_ticks(self, numticks) 1337 'get the tick instances; grow as necessary' 1338 if numticks is None: -> 1339 numticks = len(self.get_major_locator()()) 1340 if len(self.majorTicks) < numticks: 1341 # update the new tick label properties from the old ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py in __call__(self) 1059 def __call__(self): 1060 'Return the locations of the ticks' -> 1061 self.refresh() 1062 return self._locator() 1063 ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py in refresh(self) 1079 def refresh(self): 1080 'Refresh internal information based on current limits.' -> 1081 dmin, dmax = self.viewlim_to_dt() 1082 self._locator = self.get_locator(dmin, dmax) 1083 ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py in viewlim_to_dt(self) 837 vmin, vmax = vmax, vmin 838 --> 839 return num2date(vmin, self.tz), num2date(vmax, self.tz) 840 841 def _get_unit(self): ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py in num2date(x, tz) 443 tz = _get_rc_timezone() 444 if not cbook.iterable(x): --> 445 return _from_ordinalf(x, tz) 446 else: 447 x = np.asarray(x) ~/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py in _from_ordinalf(x, tz) 258 259 ix = int(x) --> 260 dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC) 261 262 remainder = float(x) - ix ValueError: ordinal must be >= 1 Traceback (most recent call last): File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 155, in __draw_idle_agg self.draw() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 127, in draw super(FigureCanvasQTAggBase, self).draw() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 430, in draw self.figure.draw(self.renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/figure.py", line 1299, in draw renderer, self, artists, self.suppressComposite) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/image.py", line 138, in _draw_list_compositing_images a.draw(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2437, in draw mimage._draw_list_compositing_images(renderer, self, artists) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/image.py", line 138, in _draw_list_compositing_images a.draw(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 1133, in draw ticks_to_draw = self._update_ticks(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 974, in _update_ticks tick_tups = list(self.iter_ticks()) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 917, in iter_ticks majorLocs = self.major.locator() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 1061, in __call__ self.refresh() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 1081, in refresh dmin, dmax = self.viewlim_to_dt() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 839, in viewlim_to_dt return num2date(vmin, self.tz), num2date(vmax, self.tz) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 445, in num2date return _from_ordinalf(x, tz) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 260, in _from_ordinalf dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC) ValueError: ordinal must be >= 1 Traceback (most recent call last): File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 155, in __draw_idle_agg self.draw() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 127, in draw super(FigureCanvasQTAggBase, self).draw() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 430, in draw self.figure.draw(self.renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/figure.py", line 1299, in draw renderer, self, artists, self.suppressComposite) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/image.py", line 138, in _draw_list_compositing_images a.draw(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2437, in draw mimage._draw_list_compositing_images(renderer, self, artists) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/image.py", line 138, in _draw_list_compositing_images a.draw(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/artist.py", line 55, in draw_wrapper return draw(artist, renderer, *args, **kwargs) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 1133, in draw ticks_to_draw = self._update_ticks(renderer) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 974, in _update_ticks tick_tups = list(self.iter_ticks()) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/axis.py", line 917, in iter_ticks majorLocs = self.major.locator() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 1061, in __call__ self.refresh() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 1081, in refresh dmin, dmax = self.viewlim_to_dt() File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 839, in viewlim_to_dt return num2date(vmin, self.tz), num2date(vmax, self.tz) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 445, in num2date return _from_ordinalf(x, tz) File "/home/beams/HAMMONDS/anaconda3/envs/bluesky/lib/python3.6/site-packages/matplotlib/dates.py", line 260, in _from_ordinalf dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC) ValueError: ordinal must be >= 1 John From: Dan Allan Reply-To: NSLS-II/databroker Date: Tuesday, April 16, 2019 at 10:15 PM To: NSLS-II/databroker Cc: "JPHammonds@anl.gov" , Mention Subject: Re: [NSLS-II/databroker] Any plans for a visualization tool for databroker? (#399) I suggest starting with the simple: db[...].table(stream_name='...').plot() Separately, we are working on a GUI that sits on top of databroker and lets you search, browse, and visualize Runs. I spent a couple hours on it today, in fact. We plan to build our tools on top of the databroker API and avoid interacting with Mongo directly so that the tools we write can work equally well on top of any storage backend. We have some prototypes that provide a db object on top of a directory of local files, an HTTP client, etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[JPHammonds]

I should mention that tools to look at the info in the database are really needed. I do not think that the users are going to really want to use the command line for this. The guy I am working with mentioned that he thought part of the reason for going with BlueSky was that since it would be used by a number of beamlines that there would be tools like this to go along with it. I am trying to move back to another project at the moment (totally unrelated) so I don’t have time right now to get to this.

John

From: Dan Allan notifications@github.com Reply-To: NSLS-II/databroker reply@reply.github.com Date: Tuesday, April 16, 2019 at 10:15 PM To: NSLS-II/databroker databroker@noreply.github.com Cc: "JPHammonds@anl.gov" jphammonds@anl.gov, Mention mention@noreply.github.com Subject: Re: [NSLS-II/databroker] Any plans for a visualization tool for databroker? (#399)

I suggest starting with the simple:

db[...].table(stream_name='...').plot()

Separately, we are working on a GUI that sits on top of databroker and lets you search, browse, and visualize Runs. I spent a couple hours on it today, in fact.

We plan to build our tools on top of the databroker API and avoid interacting with Mongo directly so that the tools we write can work equally well on top of any storage backend. We have some prototypes that provide a db object on top of a directory of local files, an HTTP client, etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/NSLS-II/databroker/issues/399#issuecomment-483920914, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIn0MIiyvJsdg5R15mOcBNffCD0P8ug-ks5vhpGfgaJpZM4czjHu.

[danielballan]

@JPHammonds We have finally made some good progress on this. See https://github.com/bluesky/bluesky-browser

It's not quite ready for users---still a bit too easy to crash---and it needs docs, but I expect it to be stable enough in a month or two.