Entity Framework Core providers

[galvesribeiro]

Although this is not a high-priority item I would like to gather input from people in it.

I've watching people asking about a storage provider that can be mapped to an arbitrary relational table or document structure, change table names, etc. In other words, multiple easy ways to have a single flexible provider.

I understand that the state of a grain is internal to itself and it should never change from outside. However, there are (a few) legitimate cases where read from outside Orleans those structures or have external side-effects (like a trigger) actioned by the state change without need to rely on Orleans mechanisms (i.e. streams) are a nice-to-have feature.

Entity Framework Core is very well designed if compared with previous version, and has a good level of performance/flexibility so a provider using EF Core would be a good idea.

So, I would love to contribute a storage/membership/statistics provider based on EF Core and achieve those targets:

1. Easy-to-use version tolerant storage provider -> EF Core has a nice and easy to use migrations API and tooling and can keep the grain state classes POCO or make use of surrogates (e.g. to support 1 POCO class having multiple normalized tables under the storage).
2. Leverage a large variety of backend stores in a single provider -> EF Core now not just support relational db backends (like SQL Server, Oracle, pgsql, MySQL, etc...) but non-sql/document-based stores (like Azure Table Storage, DocumentDB, MongoDB, Redis, etc).
3. Allow storage-level triggers to be fired when they change.
4. Allow people to make storage level relational references to grain state for read-only purposes.
5. Allow custom functions to run at storage level to perform the storage provider operations (i.e. stored procedures/functions)

And other benefits...

Downside -> History say that ORMs are (usually) not as performant as directly SQl. I'm aware that use directly SQL and store procedures would be extremely fast as it is on OrleansSQLUtils today. It is a tradeoff on flexibility over performance just like comparing BinaryFormatter serializer with JSON.Net one or JSON.Net with Bond/Protobuf. One that want to use this provider is aware of the price they pay for that flexibility.

In other words, this provider aims on flexibility if schema, multiple backends while using the same provider code/package and version tolerance. Again, it is not a replacement for the current relational providers. It is an option in a diff project with a new package.

The good news is that EF Core is extremely fast if compared with other ORMs and even EF 6.1.3.

I would like to hear from people about this proposal and if it would be something that Orleans team would accept to be on core repo or if I should go alone on OrleansContrib.

Thanks

[Maarten88]

If someone needs this why not use a non-stateful grain and write your own WriteStateAsync, calling EF or (better) any other ORM/Database? That's what I do: I run migrations on Startup, and inject the Database Connection / Context using DI. It's easy to make a base class grain and reuse it in an application. Also: the current Orleans Storage abstractions make this proposal pretty hard to implement nicely as a provider i.m.o. I think providing this as a grain base class instead of a storage provider might work better.

[galvesribeiro]

@Maarten88 you can always do that in a base class and ignore completely the storage APIs.

I've been experimenting last night with an EF Core provider and it indeed is useful.

The point on that is being flexible. Even with the storage abstractions we have today. It is not a problem. Its a lot easier to diagnostic problems once we are using same API as everyone else.

[AwsomeCode]

I think EF Core does not support Mongodb yet. And I will prefer Mongodb driver over EF Core.

[galvesribeiro]

@AwsomeCode there is a MongoDB driver in beta for EF Core.https://github.com/crhairr/EntityFrameworkCore.MongoDb

[AwsomeCode]

@galvesribeiro I Like EF Core also. but how crhairr alone going to solve the complexity solved by Mongodb Csharp Driver Team. I came from EF Core and its solve some different problems for SQL but NoSQL is completely different from SQL database and that's why Mongodb Csharp Driver is completely different from EF Core or Other ORM's.

[veikkoeeva]

Linking to https://gitter.im/dotnet/orleans?at=58dd6e7ab52518ed4dc3b7e9 so I can find myself over the weekend. :)

@galvesribeiro In short, the issues originally the provider isn't written EF isn't performance what comes to ADO.NET, but more like running reliably (resiliently) taking into account the physical aspects of the deployment, ones disaster recovery plans, flexibility to change things on the fly, tapping into special features (e.g. if this stems from the PostgreSQL of late: TimescaleDB, PipelineDb, foreign data wrappers (PolyBase in SQL Server parlor)) and tooling many enterprises tell one needs to use.

I get though the idea of using EF Core as a connector/driver to access any storage with the "same code". It's less code for the maintainer at least. Likely one would save the data as blobs then too and would loose or need to arrange "aggregate queries" or queries very specific to some persistet classes in some way or modify the storage API to accommondate that. It feels in that case application specific, normal, non-storage provider code would be an easier route.

[jason-bragg]

Something for consideration:

We've been investigating a more composable approach to adding features to Orleans than relying on base classes. The result of this investigation is what we've been loosely referring to as a 'facet' system. Basically a 'facet' of the Orleans feature set can be exposed in a grain via constructor injection.

For instance, for storage, rather than using a base class, one might be able to access Orleans grain persistence (StorageProviders) via an object (facet) injected at grain construction time. Something like:

public class MyGrain : Grain, IMyGrain
{
    private readonly IPersistentState<MyState> myState;

    public MyGrain(
        [PersistentState("Azure", "MyState")]
        IPersistentState<MyState> persistentState)
    {
        this.myState = persistentState;
    }

    public async Task SetTime(DateTime time)
    {
        myState.State.Time = time;
        await myState.WriteStateAsync():
    }
}

The above code is just an example, but the idea is that the IPersistentState facet would have the behaviors of a stateful grain inheriting from Grain without the inheritance. It would be wired up to the storage provider "Azure" much like a grain would have been, and have comparable ReadStateAsync. WriteStateAsync, ClearStateAsync capabilities.

This opens up the grains to have state in multiple backends, as well as using different persistence patterns, like event sourcing, journaling, .., within a single grain. For instance:

public class MyGrain : Grain, IMyGrain
{
    private readonly IPersistentState<MyState> myState;
    private readonly IEventState<MyEvent> myEvents;

    public MyGrain(
        [PersistentState("Blob", "MyState")]
        IPersistentState<MyState>> persistentState,
        [EventStore("Table", "MyEvents")]
        IEventState<MyEvent> eventState)
    {
        this.myState = persistentState;
        this.myEvents = eventState;
    }

    public async Task SetTime(DateTime time)
    {
        myState.State.Time = time;
        await myState.WriteStateAsync():
    }

    public async Task OnNewUser(NewUserEvent newUserEvent)
    {
        await myEvents.AddEvent(newUserEvent);
        await onNewUser(newUserEvent);
    }
}

In the above sample code, MyGrain accesses part of it's persisted state in 'blob' storage using storage providers, while also using event sourcing backed up by table storage. Such a combination would not be possible using grain base classes.

Since this system is being considered as a general approach for exposing future Orleans features it will likely afford the community a general extensibility point for introducing new features, like new storage patterns, which seems relevant to this thread.

We're still (probably) a couple weeks from introducing this capability, so it's a bit premature to plan around it being there, but the main takeaway is that you shouldn't necessarily expect to limit storage solutions to those that play well with the existing storage patterns because we hope to afford a bit more flexibility soon.

[galvesribeiro]

Fantastic @json-bragg.

Although regardless of where it will be implemented (in the current or new extension API), the EF provider I'm talking here, is agnostic.

But yeah, looking forward to have it! :)

[vyshkant]

@jason-bragg have your great ideas been implemented?

[jason-bragg]

@vyshkant Facet system and persistent state facets are in. Docs still in the work.

Orleans Facet System initial infrastructure #3279 Persistent state facet #5373

[ghost]

We are marking this issue as stale due to the lack of activity in the past six months. If there is no further activity within two weeks, this issue will be closed. You can always create a new issue based on the guidelines provided in our pinned announcement.

[ghost]

This issue has been marked stale for the past 30 and is being closed due to lack of activity.