mhasself
commented
3 years ago (First, but in brackets: why are these spread out so much in time? We can get the acqs to all start/stop within a second or so of each other, right?)
Data should only be treated as a single "observation" if they are strictly co-sampled and if they have reasonable close (like <1 or 2 seconds) start and stop times. This is to resolve the trimming issue you describe, as well as make it possible to load all of them into a single AxisManager without worrying about slight time offsets. (Perfectly co-sampled data, spread across multiple files, was the origin of the "detset" -- a detset constitutes the detectors in one time-ordered set of files.)
So I would say that these acquisitions should be separate observations, called, perhaps "crate1slot2_1633987463", "crate1slot3_1633987475", "crate1slot4_1633987485", "crate1slot5_1633987498".
I would encourage you to use ObsDb to group those (separate) observations together, so you can easily gather and analyze all the files from that acquisition event ... e.g. make an ObsDb field, perhaps "acq_event=
At the observatory, we do plan to bundle multiple blades into a single Observation. This is the Bookbinder's job, using information from the main observatory database. It will copy the data into a new set of files, with all of them starting and stopping on exactly the same sample; it will also "reframe" the data to improve compression, load speed, and certain kinds of analysis. But the key difference for survey data at the observatory is:
a) we will know exactly what acqs were commanded to start and stop together b) the smurfs are all on the master timing system, which makes them perfectly co-sampled.
G3tSmurf is currently making observations per slot / stream_id. This isn't surprising given how the code is currently written but does not mesh with how context wants observations organized.
This adds new questions we need to deal with regarding synchronization because each slot will start streaming at slightly different times. Here's an example from the most recent streaming tests:
Note how each observation has a slightly different timestamp, that's because there are slight delays between calling for starts (and stops).
Do we define observations as the maximum overlap between all slots? If one slot starts streaming while another is already running do we always assume we only care about the overlapped regions? If the start of streaming is separated by X time do we make different observations?