Example app and service templates `dotnet new -i Equinox.Templates; dotnet new eqx*/pro*` https://github.com/jet/equinox https://github.com/jet/FsCodec
~While this works, provides good throughput and other things, there's still stuff to do so it's really not ready to "review" in the sense of evaluating "is this a clean simple ingester?" atm (clue: atm the code is too messy, and there is an obvious need to add support for doing multiple concurrent readers)~
~TL;DR at the moment its a POC EventStore->Equinox.CosmosDb one-way ingester, but the general principle is that there are lots of uses for an efficient bulk sync tool working at the stream level in order to optimize ingesting performance (and batch writes to same stream) - this allows one to subset by Category and/or do transforms when propagating data onward~
~This is approaching the final shape of things in that - the Sync program is intended to be a continuous app you run in prod that can support Cosmos -> Cosmos and EventStore -> Cosmos ingestion - the Ingest Program only applies to EventStore -> Cosmos loading and is intended to be run once on a standalone basis~
The equinox-sync template's Sync app includes support for both the striped reading (which was originally pioneered in an Ingest application) (badged gorge mode ,-g) and working as a more traditional projector (which it reverts to after any gorge mode)
Work list:
[x] Make it ingest from EventStore into Equinox.Cosmos
[x] Efficiently (processing can keep up with a reader getting 4096 event batches from a local ES)
[x] Reporting status cleanly (status messages etc are clear and concise)
[x] provide striped mode when ingesting from EventStore (multiple readers, using offset computation based on @ scarvel8's event_global_position = 256 x 1024 x 1024 x chunk_number + chunk_header_size (128) + event_position_offset_in_chunk)
[x] dont report malformed exceptions
[x] report stream state
[x] implement parallel readers
[x] extend to also provide continuous ingestion
[x] ~support ingesting from CosmosDb into Cosmos~ now in dotnet new eqxsync
[x] Make 429 exceptions more succint
[x] extract EventStore walk and enumeration code to a separate file
[x] extend to record EventStore -> Cosmos progress in destination and/or aux collection
[x] unify -i and -g impls using max of both and then adjusting when switching to tailing
[x] implement cosmos connection pooling
[x] enhance slipstream submission protocol to let cosmosingester grab stuff without locking (or keeping message bus busy)
[x] support ingesting into EventStore by building an EventStoreSink
Marking as ready for review - I still have some tidying to do, but the basic split between Sync's EventStore mode and the Ingest Program (which started life as a Template) is now in place.
~While this works, provides good throughput and other things, there's still stuff to do so it's really not ready to "review" in the sense of evaluating "is this a clean simple ingester?" atm (clue: atm the code is too messy, and there is an obvious need to add support for doing multiple concurrent readers)~
~TL;DR at the moment its a POC EventStore->Equinox.CosmosDb one-way ingester, but the general principle is that there are lots of uses for an efficient bulk sync tool working at the stream level in order to optimize ingesting performance (and batch writes to same stream) - this allows one to subset by Category and/or do transforms when propagating data onward~
~This is approaching the final shape of things in that - the
Sync
program is intended to be a continuous app you run in prod that can support Cosmos -> Cosmos and EventStore -> Cosmos ingestion - theIngest
Program only applies to EventStore -> Cosmos loading and is intended to be run once on a standalone basis~The equinox-sync template's
Sync
app includes support for both the striped reading (which was originally pioneered in anIngest
application) (badged gorge mode ,-g
) and working as a more traditional projector (which it reverts to after any gorge mode)Work list:
Equinox.Cosmos
event_global_position = 256 x 1024 x 1024 x chunk_number + chunk_header_size (128) + event_position_offset_in_chunk
)dotnet new eqxsync
aux
collection-i
and-g
impls using max of both and then adjusting when switching to tailingEventStoreSink