possible usecase for opening multiple environment files, from the same app
eg in an event driven architecture, given, we want to persist the event history away, we might dedicate a separate eventlog mdb file, in which we persist the events sourced from the domainland, in some order. if persistence of events is not required, we might as well, just put these events in a zioqueue, process them, and discard them.
note: here event, is in the functional paradigm, and contains all the information that is needed to bring about a transactional update to the state.
now, we can round up these sourced-from-the-domainland events, in a zioland queue. but that's not all, because we got to persist these events in nvm⁴. we can enlist one zio fiber, with a handle to the eventlog mdb file, in one environment, to sequentially commit write for each event in the queue, to that mdb environment. fair enough.
[⁴] nonvolatile memory
another zio fiber, tasked with processing these events, might just open the same lmdb environment, in a read only mode, and read off the events in order, and process them.
this might involve changes to other mdb managed memory. maybe in same mdb data file, maybe spanning several mdb data files, this depends on the app's design. even though, we spread the risk of deletion, across several files, deletion of any of these, might leave us with incomplete data set.
note: it is not strictly necessary, that processing one event produces one transactional update in an lmdb environment. nothing's stopping us from bundling the changes eventually obtained from processing several of these events, into one lmdb txn.
doubt
could the effect of splitting an app's data, over several mdb environments, be very well achievable under a single lmdb environment, and a single lmdb data file?
is this a usecase fit for the dbi abstraction? we might simply create a dbi, call it events, maybe have an integer index, maybe it is the unix timestamp, and keep writing the dbi events.
or since writes, are fully serialised in one lmdb environment, could splitting the event log, on a separate mdb-environment of its own, give us more write throughput. because, could it be that, this way, we bypass the serialisation of writes, that happens on 1 lmdb env? maybe os level asynchrony is involved. some instances of different lmdb environments are open in my application, and my application is sending concurrent write requests to each of them, which then, in turn are processed concurrently, at os level. so in effect, we are released from the single writer model, given such orthogonality is permitted and feasible in the domainland.
so, an app might enlists several environments, with different characteristics, flags and configuration, each suitable to a particular region of the app needs. sounds legit?
[¹] true/false: the handling of an unhandled event, already persistent in the event log, should not be blocked by the act of appending an event to the event log. note that a complete event handling = 1) reading an event from the event log, + 2) persisting, in nvm, the changes brought forth by processing that event.
.
which one of these is correct
.
¹ wouldn't be achievable in a single lmdb-environment, if we define our domain logic, such that, it bundles
[²]the persistence of an event, in the event log, in environment323,
as a transaction separate, from
[³]persisting the changes, brought about by handling that event, in environment323.
.
so if, i gather an incoming event, and dispatch it as a write txn to lmdb, to persist it, any read transaction sent to lmdb in the meanwhile, for reading the prior event log, would go through, since readers don't block writers, but the subsequent write transaction, to achieve ³ won't go through, if ² hasn't yet gone through.
.
¹ could be achieved in a single lmdb-environment. if we define our domain logic, such that, we enqueue the event, in a zioqueue, in the zioland, and process the events, asynchronously in the zio land itself, and persist both, 1) the event data-object, and 2) the changes brought about by handling that event, and dispatch that as one transaction to lmdb. so eventually, persistence of incoming events doesn't block the handling of priorly persisted events, yet to be handled.
remember lmdb nesting transactions one behind the other, via the parent txn argument? #doubt what do we know about how does lmdb's execution of a parent transaction, follow this chain of nested transactions? the performance would still depend on whether lmdb executes these individual nested transactions, one after the other. or is it able to do concurrent updates on all the memory regions to be transactionally updated under the parent transaction? is it truly parallel, or effectively sequential? more importantly, does it even matter deep down at that level?
the smallest mutation is an mdb_put
since lmdb is a btree based thing, #doubt does it actually do the leaf modifications sequentially. from the bottom up?
i don't yet understand the degree of concurrency and parallelism operating on the cpu-memory bus.
because if it is effectively sequential, we might as well,
possible usecase for opening multiple environment files, from the same app eg in an event driven architecture, given, we want to persist the event history away, we might dedicate a separate
eventlogmdb file, in which we persist the events sourced from the domainland, in some order. if persistence of events is not required, we might as well, just put these events in a zioqueue, process them, and discard them.note: here event, is in the functional paradigm, and contains all the information that is needed to bring about a transactional update to the state.
now, we can round up these sourced-from-the-domainland events, in a zioland queue. but that's not all, because we got to persist these events in nvm⁴. we can enlist one zio fiber, with a handle to the
eventlogmdb file, in one environment, to sequentially commit write for each event in the queue, to that mdb environment. fair enough. [⁴] nonvolatile memoryanother zio fiber, tasked with processing these events, might just open the same lmdb environment, in a read only mode, and read off the events in order, and process them.
this might involve changes to other mdb managed memory. maybe in same mdb data file, maybe spanning several mdb data files, this depends on the app's design. even though, we spread the risk of deletion, across several files, deletion of any of these, might leave us with incomplete data set.
note: it is not strictly necessary, that processing one event produces one transactional update in an lmdb environment. nothing's stopping us from bundling the changes eventually obtained from processing several of these events, into one lmdb txn.
doubt
so, an app might enlists several environments, with different characteristics, flags and configuration, each suitable to a particular region of the app needs. sounds legit?
remember lmdb nesting transactions one behind the other, via the parent txn argument? #doubt what do we know about how does lmdb's execution of a parent transaction, follow this chain of nested transactions? the performance would still depend on whether lmdb executes these individual nested transactions, one after the other. or is it able to do concurrent updates on all the memory regions to be transactionally updated under the parent transaction? is it truly parallel, or effectively sequential? more importantly, does it even matter deep down at that level?
the smallest mutation is an
mdb_putsince lmdb is a btree based thing, #doubt does it actually do the leaf modifications sequentially. from the bottom up?
i don't yet understand the degree of concurrency and parallelism operating on the cpu-memory bus.
because if it is effectively sequential, we might as well,