Extracting/Separate ES Repo Lib?

I'm planning to use F# intrinsics to pretty much all testing/impl of Aggregates a la FsUno.

Within the sample are switchable GES/InMemory 'Store' impl consisting of two [F#] functions:-

readEvents
appendEvents

All that's missing from my perspective is a third implementation of those functions :- for NEventStore

I'll probably go to production against NES but obviously the ideal is to either have it switchable to GES when the time is right or at least not have cornered myself abusing NESisms that are going to be fundamentally incompatible.

While the merit of the LCD between them is obviously debatable, it nonetheless is something I am interesting in sweeping to the side for a time.

I'm wondering whether you have future plans/ideas/opinions re extracting an ES repo abstraction as a separate thing ?

I'm not talkiing about starting some superproject that will unify all Event Sourcing impls planet-wide; more like an OWIN-like 'spec' consisting of 2 function sigs with drop-in impls which would be adhere to that common 'interface'

While there might be merit in the idea, after having spent time working with both GES and NES I find little use for abstracting towards the lowest common denominator. Historically, I've always found that a fallacy when I spent time in ORM land and have no reason to believe it's going to be any better in ES land, even though the interface is narrower. I've never been in a situation where I needed to use one ES and then move to another ES either, so that might shed some light on why I don't see the point.

In its current incarnation - in AggregateSource - a repository is very much collection oriented. As such it makes no sense to have read events/append events repository abstraction (because that is very much persistent oriented). A repository - to me - is a collection oriented seam that lives in the model as an abstraction and in infrastructure as a concrete implementation. I'm not even happy with the IRepository interface in here, and hardly use it myself. My attempt can be classified as "a nice try" at abstracting reading from and writing to both NES and GES and is probably what triggered your question whether I would pursue that path ...

There are many flavors of doing eventsourcing, from the more functional, message driven, actor-esque approach to a more traditional "OO meets a bit of functional" approach (where AggregateSource's sweetspot should be). The kind of models I build (in C#) tend to have quite a few number of domain services in which multiple aggregates collaborate (but only one is affected) or a value object calculation is applied to a specific aggregate. I think I'd take a pretty different approach if I had to do the same in F# (never mind my noobness in that area). More akin to what @eulerfx and @thinkbeforecoding are doing.

So what are these differences I keep referring to that make NES & GES not good candidates for lowest common denominator? Fundamentally NES revolves around a commit (a changeset: a set of events that affects 1 aggregate/stream) while GES is geared towards events (still only affecting one aggregate/stream). Looking into areas such as reading, writing, meta data, idempotency you will notice subtle yet important differences between the two that will make you crave for something closer to the targeted api rather than the LCD abstraction. Depending on the flavor of eventsourcing in use, reading/writing happens in different places. In a traditional approach with multiple aggregates involved, you'll notice that reading is triggered inside the model, whereas in a "lets target this one aggregate and invoke behavior on it" the reading will be closer to the infrastructure and will need less abstraction as to how reading works/happens. It's not easy to explain the subtle nuances because there are so many variations one can come up with that will make it tilt more/less in one or the other direction.

My overall conclusion is that F# deserves a dedicated solution and style of doing ES that shouldn't be reminiscent of the C# version I've demonstrated with AggregateSource. The same goes for other areas such as testing. As for abstracting the data access I wouldn't go hunt an LCD pipe dream (not saying you are), and just implement a fully fledged implementation for GES and NES as two separate things, minimizing the points of change in your codebase if you do have to switch in the future. Building a production ready implementation that is able to deal with conflict resolution, aggregate loading, event up conversion, idempotency, error handling (connection retry, operation retry), etc ... you'll notice that abstraction is but a distant dream anyway and wonder why you've ever entertained the idea in the first place.

Now, this is to be taken with a large grain of salt and to be perceived as the ideas of one man. I'm not saying it's impossible, it's just not my mission ATM. That said, if there's any way I can help you achieve what you want, don't hesitate to tell me so.

:+1: - it seems a lowest common denominator would be a bad idea in so far as it would end up crippling some of the features of Event Store which are desirable (e.g. idempotency, clustering, subscriptions). It would likely be easier to write a persistence provider for NEventStore which targets Event Store in this particular case. On the other hand, I'm not sure I understand any compelling reasons to use NEventStore over building something simple over SQL Server at this point? (Obvious bias disclaimer).

@yreynhout Thanks for the thoughtful response.

First of all, I really shouldnt have used the 4 letters:- 'Repo'. As you've correctly inferred, I'm not alluding to collection semantics or any of the Repository notions as laid out in IDDD (much less your IRepository). I'm very much talking about a low level persistence mechanism a la https://github.com/NEventStore/NEventStore/blob/master/src/NEventStore/ICommitEvents.cs

I've implemented a system over NES without leveraging CommonDomainisms. I thus scanned AS from the perspective of "well he must be doing read/append to storage at some stage - oh look he does".

My general desire is for something functional - e.g. having handlers either throw or emit events without any OO 'model'. IOW having to register handlers and all that song and dance, having hoops to jump through in order to access the generated events in tests etc. is not a price I'm willing to pay unless I'm getting something really useful in return (i.e. I dont believe that there is an abstraction that can unify NES+GES and also fit my [as yet largely unrefined] app domain requirements)

The sort of commonalities I'm alluding to are more:

whether to make the interface Async (yes; even if NES doesn't do it now a) its on the roadmap b) GES is c) it just makes sense
whether reading should be in batches (@thinkbeforecoding reads batched in 500s)
how to manage all-or-nothing storage of events generated as a set in a manner that can work both with NES and GES (i.e. if splitting into batches is going bring additional complexities into play re e.g. conflict resolution) (I could easily be wrong but I believe subdividing the saving of a set of events across multiple Commits in NES would open up race conditions whereas GES allows submission of a related set of batches as one)
whether serialization should be managed a) by the app b) by the store c) by the lib it seems NES demands control of it but only uses it to collapse n events in a commit into a single blob which wouldnt be too much to push out. The serialization of the headers is the other use.

Unfortunately I am only aware of an interface gist from @eulerfx (+ the blog posts)

What I'm seeking to do is defintely the 'F# dedicated solution', which for me means: 1) I dont want or need a testing abstraction (due to built in structural equality, unquote and the fact that handlers just emit exceptions and a list of events) 2) I'm not at the stage where enough requirements are making running a GES for an Azure Cloud Service based app wins over using NES (or a simple lib) 3) I have no desire to reinvent the wheel given that NES has a good enough low level interface for me to read/append events (regardless of whether or not switching backing database is completely unlikely) 4) event upconversion, serialization and stuff is managed in the app

Thus I was looking for 2 impls which both provide a readEvents and an appendEvents which

are async
don't over-generalize (LSP/ISP)
let me get started storing stuff into Azure SQL [via NES] with a credible chance of being able at a responsible moment say "OK, time to move to GES" and at that point be able to leverage facilities unique to GES

If I've read/inferred correctly, your answer makes my restated question largely rethorical:

Despite all the above, do you think there might be a low level storage interface that can both be the "ES gateway" for 1) AS and 2) for an F# lib-that-isnt-a-lib-just-a-convention-really

@jen20 Firstly, unfortunately I have not done sufficient research on GES to be confident I am able to appreciate the full subtleties and ramifications implied in:

features of Event Store which are desirable (e.g. idempotency, clustering, subscriptions).

As a result, it's entirely possible I'm missing your point. Having said that, I believe I can say, as covered in depth in my response to Yves, my conclusions are largely at odds with yours.

I dont want to neuter GES by front-ending it with NES as ultimately the big option I want to leave open is to go to GES and start leveraging specifics when the time is right
I dont see the point in reinventing the wheel and writing/maintaining a buggy SQL store given that

a) NES has a low level interface that doesnt sell me a domain model (and that's my proposed level of usage) b) I think I can live with the constraints the NES Commits abstraction poses (and hence think it is valid to play the database neutrality/proven abstraction layer card while I'm using the NES version of my magic swappable layer)

However I'd be extremely interested to hear your rebuttal as (especially when measured relative to both yours and Yves' levels of experience! ) I have been roaming the ES landscape with massive blinkers given that I have neither:

a) implemented a store b) implemented a sufficiently broad variety of ES'd apps to be confident I understand what I really want from my ES

@bartelink I understand what you are trying to achieve. I think it's possible (with a lot of opinions), just not in a general sense.

I find the kind of code that goes into an "ES gateway" to show little commonality when coding it for NES or GES. Just looking at the repository implementations in AS should give you a pretty good idea. But even beyond that, when I look at closed source implementations I've done myself, there's a reason I like the ability to diverge. As a simple example, let's take multi-tenancy. There's various ways of going about it. I could model tenant identification as an explicit parameter in all calls to an ES or I could tuck it away in infrastructure, having derived it from the current request context, using a strategy to select the proper ES connection or I could have shifted it towards deployment where each tenant gets its own compute instance and some configuration setting denotes what tenant the instance is for. All choices driven by non-functional requirements. Catering for all these, next to the non-multi-tenancy scenarios, in a generic way is possible, but at what cost? I can hear the bells of overabstraction ringing. There will always be a "next scenario" that challenges that abstraction or awful workarounds that will ensue to bend things into what would have been easier if written by hand.

An "ES gateway" interface itself might be easier to define, which - as I've understood from your OWIN allusion - is what you're after. Yet, it will still suffer from the overabstraction mentioned above. At least such is my opinion. To reiterate: Yes, it's possible to define an ES gateway interface that will work over both NES and GES in your situation, it's just not going to be "THE" ES gateway interface. No, I have no ambition of putting such an abstraction into AS. Yes, you can use AS and build some code to talk to your ES gateway. But honestly, why would that last part even be a goal? At the end of the day you're writing your handler code in F#, and have some NES or GES specific decorator/wrapper that reads events and writes events. For reading you'll be concerned with deserialization, upconversion, getting back to a previous state. For writing you'll be more concerned with idempotency, meta data, serialization, etc ... In both cases, NES can do a lot of handholding that GES doesn't do, but isn't that hard to write either (and from what I read isn't even desired in your case). I think DIY is appropriate here. There'll be some refactoring in an isolated area of your code base if and when you switch, but that's about the extent of it.

It seems I also failed to underline that AS is not concerned with "writing to an ES", a by product of dealing with collection oriented repositories. It's been my experience "writing" lives closer to the edges (e.g. the transport that brought the command). The kind of meta data you want to capture is easier to capture at that edge (who, what, where, when, why, any form of context). It's just easier if everybody does that for themselves.

I find it hard to give a good, constructive answer, because I do not really understand the underlying motivation for your question. On one side you seem to be seeking an abstraction across two ES products (lib/server), on the other side you're interested in said abstraction from an "ease of transition" perspective in your particular situation, looking for some particular traits along the way. In any case, the answer will not be found in AS.

@yreynhout Thanks Yves - you haven't missed much in there; I'll ponder and reply when I've spent some time in my hammock :)

To add something to the discussion, I've found no real need for common code to implement ES since I read Rinat's code a few years ago. This is every time just a few lines of code, that feel very natural, and that can have small variations depending on your requirements : Keeping aggregates in memory or not, Snapshots or not etc.. This kind of code is surely easier to replace with new one when you want to change your infrastructure that making it more complicated to be switchable...

yreynhout / AggregateSource

Extracting/Separate ES Repo Lib? #16