search

DevrexLabs / memstate

In-memory event-sourced ACID-transactional distributed object graph engine for .NET Standard
https://memstate.io
Other
365 stars 46 forks source link

use EventStore embedded as alternative to FileStorage #38

Open rofr opened 6 years ago

rofr commented 6 years ago

In the EventStore project, implement a storage provider that uses an embedded event store node, see https://eventstore.org/docs/dotnet-api/4.0.2/embedded-client/

goblinfactory commented 6 years ago

@rofr Why do we want to do this? I'm not saying we shouldn't, just that this Jira lacks any context for me to contribute, or consider alternative solutions to the desired outcome or problem to be solved. I'm assuming eventstore embedded is a client library? If so, wouldn't that need to be a pluggable option considering your LGPL licence? You'd ideally want memstate to be reliant on as few external libs as possible. If you some information around the "why" then the discussion for that feature can live here and not clutter up the gitter chat.

(some explanation of some of the shortfalls behind the default fileStorage and what the eventStore embedded (client?) offers would be very helpful. This will also help me with documentation as I'd like to consider somewhere to put notes for "best practices" and "recommendations", limitations of default FileStorage etc. Very important information.

Also, is this the lib you're talking about ? : https://eventstore.org/docs/dotnet-api/4.0.2/embedded-client/

A

rofr commented 6 years ago

Eventstore is battle tested, robust, reliable, durable, fault tolerant etc. Our filestorage is currently far from this level of quality, so why reinvent the wheel? Also we already have support for remote eventstore, so adding an embedded option seems like low hanging fruit.

There are some other nice features as well, for example the web based admin UI where we can view the journal entries.

And of course this provider would live in the event store module, so would add a hard dependency to the core memstate package.

goblinfactory commented 6 years ago

Assuming a new dev comes along and just wants to build something with a persistent model, what would his setup look like, and what would his getting started code look like hypothetically if we use eventstore? Would this change his installation and getting started assuming eventstore becomes the new default StorageProvider? If the default storage provider is not reliable, or durable or robust, then perhaps we really shouldnt ship it at all? (Getting started code should be battle tested and usable in live production systems, imho.)

goblinfactory commented 6 years ago

Would it make sense to create separate issues for the current HostFileSystem (IVirtualFileSystem)? I'd be happy to write some tests proving or testing any concerns or assumptions we have about the provider? To be totally honest, it's what I need myself for a bunch of simple prototypes that I don't want a complicated stack for, and having something that scales out of the box, and is stupidly fast is very attractive. I can imagine a lot of projects starting and sticking with the basic fileProvider because "it works", I'd want to be able to show people what it can, and cannot do for you with simple tests.

rofr commented 6 years ago

Using the embedded eventstore would be a simple config settings and a nuget download of the event store module which in turn has a dependency on the eventstore client nuget. Besides the single config setting, it would be entirely transparent to the rest of the users code.

goblinfactory commented 6 years ago

Would be good to have a human friendly error message that's a bit aware of the typical providers, so that if a provider is specified that we know about, we can give an error message with instructions on how to fix, e.g. "You've specified XX provider, and it looks like you've not included the package in your project, please run the following nuget command , " In such a way that a beginner can install just 1 nuget package, write a few lines of code, run his app, see the message, install 1 more nuget package, press F5 and now it works. (a sort of pit of success)?

rofr commented 6 years ago

Testing edge cases and failure conditions for file operations on multiple platforms and with multiple different drivers and hardware is not a trivial task. We've not had a single incidence of file corruption with origo since we added the Packet abstraction which is yet to be implemented in memstate. File storage will still be the default storage and it needs to be sufficiently reliable.

rofr commented 6 years ago

I'm thinking a Memstate.Bundle nuget package with all the important packages bundled would solve this beginner problem. newtonsoft, eventstore, postgres and memstate core. But sure, useful error messages.

goblinfactory commented 6 years ago

We don't need to test everything, just the most common expected use case, and any of our own personal use cases, then the documentation can be explicit about what has been tested, and what has not. We can expand testing if anyone reports a bug, and hopefully our tests can be well enough written that they can easily be adapted if someone wants to test a specific use case we've not covered and of course they're welcome to submit a pull request.

goblinfactory commented 6 years ago

re: memstate.bundle : I did not know that was possible. Are you allowed to do that with an LGPL licence? Just because something is in a nuget package, doesn't mean the combinations of the licences are not complicated as you'd effectively be distributing newtonsoft etc. I thought the purpose of package configurations was to force the user to download the packages, so that you never distribute the actual packages, only the recipe of what packages are needed and the user chooses to accept licence terms when each package arrives.

goblinfactory commented 6 years ago

I might be wrong but I thought that bundling is for internal corporate use, not for publishing to public nuget servers? Something along these lines ? http://www.marcusoft.net/2011/09/creating-local-nuget-repository-with.html

rofr commented 6 years ago

Only our own libraries would be in the package bundle. External dependencies would be downloaded transitively by the user.

goblinfactory commented 6 years ago

Ah, ok ... that makes sense. So if you need eventstore you install the "memstate.core" plus the "memstate.eventstore.xx" (roughly) and you get the eventstore.xx dependant packages.

rofr commented 6 years ago

Maybe we shouldn't use the term "bundle", you are not the first person misled by that wording. The point is to make onboarding easy. The first-timer shouldn't have to figure out which of 12 possible nuget packages they need to get a simple example going. I'll create a separate issue for nuget packaging strategy and implementation.