search

sageserpent-open / plutonium

CQRS providing bitemporal object modelling for Java POJOs and Scala too.
MIT License
5 stars 0 forks source link

Scaling - new In-Memory Implementation that supports Fast Queries #37

Closed sageserpent-open closed 6 years ago

sageserpent-open commented 7 years ago

This is a real story that delivers fast queries (i.e., does not require full event playback for each query). It will finish off the emerging design from #36 , using the snapshot storage capability to yield items quickly in a scope query.

sageserpent-open commented 7 years ago

Plan:-

  1. Come up with initial cut of BlobStorageSpec. Just use simple histories, no annulments are other obsolete events. DONE

  2. Implement BlobStorage. DONE

  3. Add in obsolete events in BlobStorageSpec. DONE

  4. Check ItemStateStorageSpec and update. DONE

  5. Implement ItemStateStorage. DONE

  6. Just implement Timeline and use indirect testing via World and Scope. Will need to adapt PatchRecorder. DONE

  7. Address annulments. DONE

  8. Address annihilations. DONE

  9. Address experimental worlds. DONE

sageserpent-open commented 7 years ago

Having got BlobStorage into something plausible, albeit without annihilation support, let's focus on Timeline for a bit and use that to refine the API for ItemStateStorage from the top-down. A timeline gets its item cache from its item state storage, but it also needs to store events directly (at least for now) and to use the patch recorder to drive the creation of a new item state storage.

It looks like quite a bit of the old code for events will get rehashed in Timeline, at least to start with.

sageserpent-open commented 7 years ago

Problems - I've incorrectly modelled how snapshots are recorded - I have grouped them by event id, when I should have grouped them by time. One of the problems engendered by grouping via event id is that snapshots are rendered obsolete based on only a single event id - but with the benefit of hindsight, this is obviously wrong, as several events may occur at the same time with their own event ids, and these may patch the same items using patches under distinct event ids.

To make matters worse, not only can event ids accumulate together on a single snapshot via distinct events, but the process of selecting a best patch from a change and / or measurements can also mix up distinct events with their own event ids - in fact, all of the candidate patches are contributors to a snapshot, even if they weren't selected.

Need to stop and think about this...

sageserpent-open commented 6 years ago

It's finally done. Phew! Now for a retrospective... 😄