Add a new document type for "bulk" external resources with no predetermined shape.

[maffettone]

Develop a stream_resource that manages an unknown number of contiguous stream_datum, with the potential for multiple streams. This is especially relevant when the data is expected to be ragged or has no pre-determined shape (number of rows).

The potential for multiple streams makes this construction slightly different from resource in that multiple counters, and stream names need to be tracked within a closure.

Description

• Addition of 2 new document types and json specifications
  • stream_datum
  • stream_resource
• stream_datum_page was not included, as this felt like a hat on a hat.
• Refactor ComposeRunBundle from a namedtuple into an iterable dataclass. This is for backwards compatibility and extensibility should new documents be needed in the future.
• Compose streaming run bundle as extension to previous return value. (Passes all previous tests)
• Extend DocumentRouter for new documents
• Extend RunRouter for new documents
• Extend Filler with methods for new documents
• Documentation Changes:
  • Add sections to data-model.rst, and update intro highlighting these components as (Experimental).

TODO:

• Documentation

Motivation and Context

Currently the implementation of external data is limited to filling in data on a per-event per-field basis, basically the smallest quanta of data that the system can express. While this works extremely well for something like image stacks of equal length, this works very poorly in cases where the data is natively ragged (single-photon detectors) or when there are many many relatively small data sets (e.g. scanned florescence data). The difficulties vary from fully breaking the ability to put the data in an xarray to pathological performance problems.

Closes #219.

How Has This Been Tested?

Testing environment included all event_model and bluesky requirements in Python 3.9.

All previous tests pass.

Tests added:

• test_em.test_compose_stream_resource
• test_em.test_document_router_streams_smoke_test
• test_run_router.test_run_router_streams

[maffettone]

Before going much deeper and embedding these new documents into the DocumentRouter, and Filler classes, I wanted to get a partial review (@tacaswell ) and clarify that my approach here was sensible.

A given stream_resource can point to multiple stream_datum with unique names. I.e. a single resource could point to the output stream of a single photon detector and the output stream of some thermocouple. Each datum can contain one or many rows (frames of the detector or voltage/temperature pairs of the thermocouple), but a single block_id, for the sake of constructing the full column in retrospect. The event_offset is there for the sake of book-keeping in that block 0 and block 1 may actually be separated by a gap?

[coretl]

We're happy with this so happy to see it merged whenever

[danielballan]

@tacaswell Is this ready to go? I will run the release process when this is merged.

[maffettone]

Unsure why I left Documentation under the TODO's list. Perhaps just because it was failing the documentation Action.

[tacaswell]

@coretl Lets see what breaks when you use this in anger 😈 .