Closed danielballan closed 10 months ago
I sketched out the old and new (with a single event, multiple events would make more use of the slice object)
Here's a suggestion as to how it could be implemented.
datum_kwargs
stream_name
, instead a pointer to the Descriptor it provides some or all of the data_keysdata_keys
list to show which data_keys of the Descriptor it providesseq_nums
is a slice
object showing the Event numbers it corresponds toindexes
is a slice
object passed to the StreamResource handler so it can hand back data and timestampsdata_keys
will be missing from data
and timestamps
The Event side of the diagram then becomes optional, if all detectors write external files then there will be no Event documents
@tcaswell @danielballan shall we begin writing this?
Yes, I think so. I'd like to do this in tandem with @DiamondJoseph's NeXus writers and a sketch of our proposed new storage model for Databroker to validate that everything works out the way we expect.
Can delete stream_names from StreamResource too
Notes from a chat with @coretl @tacaswell @callumforrester @DiamondJoseph @tizayi
The motivation here is to improve the situation for using the same detector in fly scans and step scans, with efficient access to data. Lots more to discuss before we implement these disruptive ideas, so nobody panic. :- )
describe()
method continues to indicate which keys are backed by external data. (Perhaps we use a new word for this new scheme.)read()
method will simply omit keys that are backed by external data. The RunBundler can verify that non-external keys are present and external keys (as declared by descriptor) are not present.collect_asset_docs()
. This should include aPartition
document with some parameters that will be used to associcate a slice of underlying data with a (length-1) slice of Events.Resource2
(gotta name this...) that should have a mimetype and a dict of arbitraryparameters
. The only restrtiction on paramters is that they are jSON-serializable.Partition
will also haveindex_start
,index_stop
corresponding to a slice in the underlying storage.