search

bluesky / event-model

data model for event-based data collection and analysis
https://blueskyproject.io/event-model
BSD 3-Clause "New" or "Revised" License
16 stars 32 forks source link

Discussion: Handlers that take datum pages? #38

Open danielballan opened 5 years ago

danielballan commented 5 years ago

Currently, if you say, "I want all the data for this vector of N datum_ids," we have to do N function calls to handler.__call__, which only knows how to process one datum at a time. (We do at least avoid N database hits by pre-fetching all the Datum documents for a given Resource when you ask for one of them.)

Do you need to think about extending the "Handler API" to deal with a vector of datum kwargs?

tacaswell commented 5 years ago

I think yes (if only so the obvious loop is hidden), but in order to vectorize nicely, we need guarantees that the datum are contiguous / ordered so we can make use of vectoredized access of the underlying data storage. I have been thinking about this on and off for about a year now and have not had any good ideas...