Infrastructure Priorities for next cycle

[danielballan]

Some have asked (recently, @afluerasu) what will be the priorities for the next cycle. This is a list of the infrastructure-related aspects of the next cycle that @tacaswell, @klauer, and I are aware of. While the list is driven by beamline needs, it does not encompass or attempt to prioritize all their hardware, collection, analysis needs. That is a larger, separate issue.

The broad theme for bluesky/ophyd development is: "fix & fortify" (to steal @tacaswell's humorous reappropriation of the MTA ads). Two cycles ago, we deployed bluesky; in the present cycle, we revised ophyd; the coming cycle is for cleaning up and strengthening. Going forward, we will take all pains to avoid changes that would break existing beamline configuration (IPython profiles) or user code.

Here is a partial list:

• [x] Refactor "suspenders" (for several reasons -- one is NSLS-II/bluesky#389)
• [x] Refactor plans code (NSLS-II/bluesky#345)
• [x] Merge status object logical operators (https://github.com/NSLS-II/ophyd/pull/241)
• [ ] In beamline configs, refine read_attrs, configuration_attrs and names to match scientists' wishes
• [ ] Deploy metadataservice and metadataclient
• [x] Refactor databroker as an instance, to prepare for analysis store (https://github.com/NSLS-II/databroker/pull/41)
• [x] Do channel archiver integration via a plugin feature in databroker (https://github.com/NSLS-II/databroker/pull/51)
• [x] Take "baseline" readings at beginning and end of plans (NSLS-II/bluesky#317; https://github.com/NSLS-II/ophyd/issues/187)
• [x] Make get_events more useful for async scans, add a name arg for event stream name
• [x] Make get_table more useful for async scans: get_table(..., name='primary')
• [ ] Record msgs in event streams (https://github.com/NSLS-II/bluesky/pull/387)
• [x] Record RE state changes in event streams (https://github.com/NSLS-II/bluesky/pull/463)
• [ ] Discuss possiblity of adding hints about interpolation rules into data keys
• [x] Change Msg API -- this is probably our last chance before starting getting into that (https://github.com/NSLS-II/bluesky/issues/294)
• [x] Document 'monitor' Msg and maybe add a context manager or something.
• [x] Full text search in databroker (create index in MDS)

Nice-to-have:

• [x] Implement partial collection on flyers
• [ ] Refactor out bundling to support multiple bundles.
• [x] Experiment with HDF5 SWMR.

Many of these are nearly ready to merge as soon as the cycle ends. We aim to stop merging big changes and start running user-acceptance tests by Week 4 of the shutdown (the half-way point).

[cowanml]

On Mon, 21 Mar 2016, Dan Allan wrote: ...

• Refactor databroker as an instance, to prepare for analysis store ...

Each data layer (layer_0 = raw, layer_1 = initial processing, ...) is going to have it's own: db, broker, and file service instances?

[danielballan]

@cowanml Yes, each layer would have its own Broker instance -- say, db0, db1, etc. To be clear, the filestore client and metadata client instances will belong to the Broker instances that employ them. So, users and downstream code will interact through Broker instances -- rarely through clients directly.

[danielballan]

@cowanml P.S. Let's continue any discussion on this particular point on the relevant databroker PR I linked above.