OData NxT - The Future of OData (Project Proposal)

[hassanhabib]

Here's some thoughts for OData NxT

• Transcendence- We need to be able to provide transient queries. meaning that if I want to relay my OData queries from one API to the next it should be easy to communicate that through some sort of protocol. I have an proposal for IQueryableAsync .NET implementation while still conforming to OData protocols.

• Modularization - we need to allow people to build extensions and modules to add new extensions - If I want to create an extension to provide some new functionality then they should be able to create the extension and offer the new functionality.

• Technology-Agnostic from ASP.NET routing. We need to reconceptualize our OData implementation so it's not tightly coupled to a certain implementation for ASP.NET - we want that concept to allow other protocols like gPRC for instance to allow.

The most important part about all of this is to allow our framework to be extensible, agnostic and modular. I will follow up with more details about this initiative soon.

[hassanhabib]

I will be setting up public design discussions and we will share the progress on this new initiative with everyone soon.

[chrisspre]

Thanks Hassan for initiating this.

• Transcendence- We need to be able to provide transient queries. meaning that if I want to relay my OData queries from one API to the next it should be easy to communicate that through some sort of protocol. I have an proposal for IQueryableAsync .NET implementation while still conforming to OData protocols.

Need to watch the whole video but the pieces I saw are already impressive. Can you please elaborate in which way Where(city => city.Name == "Redmond") is different from $filter=name eq 'Redmond' ? I clearly see the syntactical differences but I was wondering what else is different.

• Modularization - we need to allow people to build extensions and modules to add new extensions - If I want to create an extension to provide some new functionality then they should be able to create the extension and offer the new functionality.

Can you give some examples for extensions you have in mind.

The next version already makes some progress how it plugs in into the ASP.NET pipeline (middleware/routing) and also how EDM and model builders are factored. Is this sufficient or are you thinking about even more granular modules ?

• Technology-Agnostic from ASP.NET routing. We need to reconceptualize our OData implementation so it's not tightly coupled to a certain implementation for ASP.NET - we want that concept to allow other protocols like gPRC for instance to allow.

OData is couple to HTTP in many ways. ODL is a .NET implementation and the quasi web service implementation standard for this is ASP.NET, I am not sure what technology you would want to swap in this stack. How would a hypothetical ODATA over gRPC look like? Not saying it is not imaginable but I don't see this at the moment.

Maybe this is something that is in line with your thinking here and I am wondering about sometimes. ODL very much expects that types implementing IQueryable are returned. This means the flow is: request -> custom code (in controller action ) -> queryOptions get applied -> response.
Is it not sometimes desirable to request -> custom code -> apply queryOptions -> custom code -> response, where my code that I write can inspect and optionally change the query options, loads data (using the query options, i.e. applying the query option) and then explicitly do something with the data before it gets returned to the framework (formatted etc) This would allow to use the power of IQueryable QueryProviders but not require it.

[hassanhabib]

Hi @chrisspre

For transcendence there should be no real changes to the syntax. In my video it was just a POC to show an example of what would happen when a protocol of some sort can be converted and translated directly into LINQ. but we definitely want to stay in conformance with OData Standard. The idea of transcendence here is to allow these queries to transform easily into LINQ with the least amount of footprint possible. But it doesn't stop there. We want to have API-1 to receive an OData query and be able to transcend that query downstream to API-N without any issues. This is the most important part about the RESTFulLinq video - to convert the query to something already generic and familiar in C# and then let the consumers of that query pass it around or ship it to an entirely different API without any problems.

About the modularity here's some thoughts. Imagine that I wanted to develop a new module to allow OData consumers to do fuzzy search. so something like odata/students?$filter=soundsLike(FirstName, 'Sarah') I should be able to design, develop and publish an extension that would allow other people to simply download my nuget package with the new extension and now they have the capability enabled. This needs to be dead simple and easy. even if I wanted to create an entirely new $ functionality like $iterate for instance It should be something as simple as implementing or conforming to a contract/interface then publishing so everyone could take advantage of the new capability.

As for the decoupling from ASP.NET or any other protocol. My thought process here is to allow OData to be configurable to support multiple communication protocols. Assume that I have an input parameter in a simple console application that reads files from a system. Now, imagine that the console is offering an IQueryableAsync (not a thing yet) as a return type - I should be able to pass OData query to my console app (which is not ASP.NET Core or anything like that) and I should be able to get back my query results. Take that example and apply it on any new communication technology, gPRC, REST or even WCF. the idea would be to pass a query parameter somehow, and whatever is underneath that implements OData as an agnostic technology should do the trick without any hard dependency on any of them.

What do you think?

[hassanhabib]

Here's our very first session: https://www.youtube.com/watch?v=l024p4_3q2c

Some actionable items to update this thread with:

• Experimenting with the current state of transcendence - it seems that we have the right component in place, the next step is to see how easy we can make this be.

[hassanhabib]

@xuzhg and I had a great session diving into the details of AST and Expression Trees in the current implementation of OData - watch it here:

https://www.youtube.com/embed/6AvFqhkALmg

[willsmithgithub]

The "Transcendence"/ Transference? I watched your video architecture discussion. It seems to me with the diagram you were showing an API Gateway (?) fronting the Odata microservice. Seems to me need aggregation/slicing to 1..N Odata services, abstracting those away with your LINQ ideas. This would lead to something like GraphQL concept of joining data sources applied to Odata. That seems like would a strong position to be in. bad example just grabbed after search: https://hasura.io/blog/remote-joins-a-graphql-api-to-join-database-and-other-data-sources/

[OnurGumus]

As an F# developer using giraffe instead of classic ASP.NET web api, routing dependence is one of the setbacks for me. I'd highly appreciate a more agnostic approach.

[TehWardy]

My initial thoughts on this proposal ...

Surely the trick here is to implement a LINQToOData provider so that a local "generated proxy". Concept or idea ... i'm thinking something like how WCF services used to do this but using the OData $metadata / swagger def on the client side.

That would give you a strongly typed client side OData context that you execute a LINQ query on essentially gets translated to an OData question to give you back an IQueryable

Pseudocode idea ...

// how WCF would do it
var proxy = new MyODataClient("https://.../swagger/v1.swagger.json");
var result = proxy.GetAll<City>().Where(c => c.Name == "Redmond").ToArray();

// smarter ligthweight method ...
var proxy = new ODataQueryable<City>("https://localhost/");
var result = proxy.GetAll<City>().Where(c => c.Name == "Redmond").ToArray();

The above issues a HTTP GET to ~/City?$filter=Name eq 'redmond' on the API. The second example would only need the definition of the objects, no meta from the API is needed, that we could build the LINQ then translate that from the object defs entirely, the client would only need the definition of City and all its referenced types in the graph.

For server side implementation surely that RESTFulLinqService class used in the video could be implemented as an optional middleware so that consumers can just call ...

app.useLinqQuerying();

Then in our controllers we can do something like ...

public IActionResult Get(ILinqOptions opts) => Ok(opts.ApplyTo(Db.Set<T>());

... this then brings it inline with how Odata options work too. The hard part will be when an API has both OData and Linq options supported ... how would routing work to determine what's odata and what's linq, the simplest way I can think to answer that is to support a $linq param or something in calls.

I'm assuming most people implement their API's like I do and separate their entity types from their business logic, so having a simple lightweight "DTO dll" would serve this approach well.

I'm also assuming that really all that's going on here is what takes place in DynamicLinq mapped form the URL. Does this not present a code injection problem though if not parsed properly?

[HuntJason]

PLEASE coordinate with the API Versioning and Swagger/Swashbuckle/OpenAPI teams in any upcoming releases.

[HuntJason]

My initial thoughts on this proposal ...

Surely the trick here is to implement a LINQToOData provider so that a local "generated proxy". Concept or idea ... i'm thinking something like how WCF services used to do this but using the OData $metadata / swagger def on the client side.

That would give you a strongly typed client side OData context that you execute a LINQ query on essentially gets translated to an OData question to give you back an IQueryable

Pseudocode idea ...

// how WCF would do it
var proxy = new MyODataClient("https://.../swagger/v1.swagger.json");
var result = proxy.GetAll<City>().Where(c => c.Name == "Redmond").ToArray();

// smarter ligthweight method ...
var proxy = new ODataQueryable<City>("https://localhost/");
var result = proxy.GetAll<City>().Where(c => c.Name == "Redmond").ToArray();

The above issues a HTTP GET to ~/City?$filter=Name eq 'redmond' on the API. The second example would only need the definition of the objects, no meta from the API is needed, that we could build the LINQ then translate that from the object defs entirely, the client would only need the definition of City and all its referenced types in the graph.

For server side implementation surely that RESTFulLinqService class used in the video could be implemented as an optional middleware so that consumers can just call ...

app.useLinqQuerying();

Then in our controllers we can do something like ...

public IActionResult Get(ILinqOptions opts) => Ok(opts.ApplyTo(Db.Set<T>());

... this then brings it inline with how Odata options work too. The hard part will be when an API has both OData and Linq options supported ... how would routing work to determine what's odata and what's linq, the simplest way I can think to answer that is to support a $linq param or something in calls.

I'm assuming most people implement their API's like I do and separate their entity types from their business logic, so having a simple lightweight "DTO dll" would serve this approach well.

I'm also assuming that really all that's going on here is what takes place in DynamicLinq mapped form the URL. Does this not present a code injection problem though if not parsed properly?

I'm not certain how this is different than ODataQueryOptions. I feel like Linq just opens up all options rather than allowing the developer to restrict to what they want permitted.

[hassanhabib]

Quick update from our last session with @xuzhg

In this video, Sam and I will show you how to take in a raw #OData query in a simple console application and convert it into Abstract Search Tree.

This video is a part of a series to revolutionize OData to make it a technology agnostic technology with capabilities such as transcendence, modularization and pluggability into any communication protocol.

https://www.youtube.com/watch?v=jqCRv5sGrqk

[doertli]

Would be great to see OData playing nicely with GraphQL. Not just for queries, but also subscriptions.

[TehWardy]

Would be great to see OData playing nicely with GraphQL. Not just for queries, but also subscriptions.

GraphQL baffles me ... technically it's ok to have a body in a HTTP GET according to the standard but i've not seen it until GraphQL.

@hassanhabib Would be cool to see how OData works with say ... multiple sets of IEnumerable and then using the model figures out how to stitch things together such that we don't require IQueryables (causing a leaky abstraction of functionality in things like EF to be required to be pulled all the way through to our API layers.

Another neat feature would be to be able to take an OData querystring and apply it to an IEnumerable to build an expression tree as it's own smaller lib without a model requirement for those using things like Swagger that don't want to build OData models and just want to use MVC and allow for OData querying in their own controller code.

This may also be a smarter way to rebuild the code in the EnableQuery attribute which is insanely deep in call tree depth.

[hassanhabib]

In this session, Sam and I will show you how to convert a raw OData query into an IQueryable then to a List. This effort completes 50% of the way to implement full transcensance from one component to another though OData protocols.

https://www.youtube.com/watch?v=ccRPRlROx0s

[TehWardy]

@hassanhabib The act of dealing with an OData string should be straight forward through just raw dynamic LINQ "within the OData expression scope". This in principle means that if we have an IEnumerable and an OData string we should be able to apply the string to the enumerable, and by extenstion from n expression tree composed on top of an IEnumerable we should be able to retrieve the OData Query portion of the URI.

My thinking is as follows.

What if we could do something like this ...

var i = new List<T>();
var results = i.ODataApply("?$filter=Value gt 123&$select=x,y,z");  <-- composition of OData on top of a list

My issue with your approach is this need for all this declaration of metadata and the OData model before simply building what amounts to an expression tree composed over an IEnumerable

Thinking it more "generally speaking" OData queries are basically a small subset of what is already doable using Dynamic LINQ. So knowing what T is (because we have a List) and parsing the OData query we can then tokenize that and translate it to an expression tree composed over the initial list for an IQueryable.

Why do we need the OData model at all other than to define T which in this context we already have?

....

The other "useful scenario" here is where I do the above then do something like ...

var results = await new HttpClient(new Uri("base url")).GetAsync(results.ToOdataString());  <-- the reverse from the server

... to retrieve the items from the server that relate to the constructed client side expression tree.

It's this simplicity that matters to consumers of the OData api's, the OData model is really more important for determining what you're interacting with (like getting the definition of T as XML / json).

In my opinion as a consumer anything more than this is the result of over complexity in the OData API. I'm seeing a repeated pattern of the implementation path tht you and @xuzhg are going down that is quite declaration heavy.

One thing that might be helpful though if an Odata model is needed would be a "model from" type thing like this ...

var modelTypeSet = new Type[] { typeof(someType), typeof(SomeType2), typeof(someType3) ... };
ODataModel model = modelTypeSet.BuildODataModel();

This reduces a ton of overhead on the consumer and builds out the model entirely in framework code "correctly" so if these conventions / the implementation needs to change for some reason this can be altered completely within the framework.

By Extension we could then do something like this in our Api applications on the Server Side ...

var controllerArray = AppDomain.Current.GetAssmeblies()
   .SelectMany(a => a.GetExportedTypes())
   .where(t => t.HasAttribute(ODataController);

 MethodInfo[] missingImplementations;
if(!model.IsFullyImplementedBy(controllerArray, out missingImplementations))
{
       throw ODataNotImplementedException(missingImplementations);
}

This focus on having us as consumers build the model is generally not the thing we care about, as much as it would help to be able to manipulate the model, actually building one can be done automatically through reflection IMO and so needs to be moved internally in to the framework.

[hassanhabib]

@TehWardy I agree - I want to simplify this so you don't have to deal with the models. The same way a non-EDM approach works. let me try some of what you proposed here and get back to you on that thread.

[TehWardy]

@hassanhabib nice! Hopefully i'm providing positive feedback here and not just crapping on your awesome efforts. Shout if you want some extra collab on this though, this is a key tech for me and I'd defintely get involved to see it succeed, especially given that you have the of @xuzhg too on the Microsoft side ...

Would love to see more of this type of collab work with Microsoft teams. I'm hooked on the youtube series for this so far :)

[hassanhabib]

Hey guys, here's our latest session with @xuzhg - this time we will try to turn around and convert an Expression into an OData query! check it out here!

https://www.youtube.com/watch?v=8OMv_GwqWnw

[TehWardy]

Good stuff @hassanhabib ... One thought I had ... what if the concept of client and server was ripped out altogether from the library. My thinking here is something like: On the Server We let AspNet do all the server side comms and serve up controllers ect and pass the relevant args in to the controller actions, and use attributes ect to help with binding. On the client We Let HttpClient do all the work and build extensions / subclass HttpClient class that can take an expression tree and translate that in to an OData query with some configuration.

This leaves the real work of translation between OData strings nad expression trees to the core logic code as a separate library and removes any need for web framework at all to use OData. This got me thinking ... Neat, with that I could use OData queries over something like SignalR to get real time event driven APIs.

I did have one concern though ... there's a lot of reflection work going on here that could be reduced by just assuming the query is correct and attempted to build the expression tree parts on top of an IQueryable, if there's a failure, you can drop in to reflection to find out why.

This should give us a net perf increase. Expression Trees are a brutal black art though wish I knew more about them to provide more advice, but Sam is on it! Keep up the awesome work though it's definitely going down the right path and giving @xuzhg tons to think about ;)

[hassanhabib]

@TehWardy the reflection part is just how OData is implemented today. OData NxT is out opportunity to re-think and rewrite all of this. stay tuned.

[TehWardy]

Yeh I figured as much ... I also figure that in part that's where EDM types come in handy, as we can get the type info from our EDM model without the overhead of a reflection call. I wondered if the trick here is to have a "enable checks" option that allows being turned off for a straight translation from the odata in to the expression tree by literally just reading it and applying it. Of course to know how to apply some knowledge of typing is needed so that didn't feel like a complete / well thought out solution in my head.

Not actually doing this chunk of work myself though i'm not entirely clear how much of that typing info you really need and how much assumption you can make. My logic was along the lines of "hey if it aint right you'll get a failure trying to either build or execute the expression tree that you can catch".

No next part of the video series yet @hassanhabib i'm eagerly awaiting it?

[hassanhabib]

@TehWardy my apologies for the delay. Just working through some scheduling and planning issues for the project. We are still here and active - a new video in the series coming out next week Friday. I may however challenge the idea of EDM. But my thoughts aren't fully formed yet around a simpler easier to understand alternative yet.

[tomasfabian]

Hi everyone, @hassanhabib a few years ago I started to think about how it would be possible to filter, project, etc. push notifications server side. I experimented with the idea a little bit and I used OData to transfer the query to the server in order to influence the published messages via SignalR. This intro is related to your Technology-agnostic requirement.

I’ve also recently implemented IQbservable<T> (Q is not a typo) for ksqldb. ksqlDB.RestApi.Client is a .NET LINQ provider which allows you to subscribe to a stream of events server side and observe these records (materialize them) client side.

In case that someone is not familiar with the purpose if this interface, then it can be compared to IObservable<T> (RX.NET), but executed server side similarly as IQueryable<T> is an AST counterpart of IEnumerable<T>. Basically these four interfaces are symmetrical and provide means for Pull vs Push (iterator vs observer pattern) in combination with client vs server side execution.

Edit: I added the following table to visually depict the above mentioned abstractions:		CLR	AST
pull	IEnumerable<T>	IQueryable<T>
push	IObservable<T>	IQbservable<T>

It is alright in my opinion to use my library let`s say in a server side Blazor application or in a microservice. But on the other hand I'm convinced that the message broker shouldn't be exposed in the same way as databases aren’t. They are usually hidden behind a (micro) web service.

So I’ve recently again started to think about using OData queries as an intermediary between servers and clients (UIs), and I found this great proposal of yours. I think that it will be possible to create a C# LINQ query client side, translate it into OData query URL with OData.Nxt.Client, then server side an expression tree could be created with OData.Nxt.Server (as you suggested too), we can traverse the AST with an expression visitor, and translate it into KSQL with ksqlDB.RestApi.Client. IKSqlQueryProvider is then able to execute the subscription and the materialized records would be resent/republished with WebSockets to the client. Currently OData / EF Core does the same by fetching/pulling data through HTTP / SQL etc.

What do you think about my thoughts?

Let me know in case that I can help you somehow with your tremendous effort.

[hassanhabib]

@tomasfabian I love this idea. This is essentially the transcendence aspect of this effort. I'm going to publish a roadmap for OData next week and maybe we can incorporate your ideas and make something bigger than the both of us :)

[TehWardy]

Loving this!

I figured the general idea here is to do something Kendo UI does with it's OData data source but with observables over SignalR ... https://demos.telerik.com/kendo-ui/grid/odatav4 https://demos.telerik.com/kendo-ui/grid/signalr

... essentially combining the functions of both.

I think the biggest hurdle here is making all that integration work as intended and remain "dev friendly".

Weirdly i'm on the same page here and looking to solve similar problems of data being updated perhaps by automation or a user action elsewhere and keeping the data your currently looking at up to date in real time. The idea I had was to have the act of EF seeing a CRUD operation take place (through it's change tracker) on a given entity that the user has access to "trigger" an event that pushed a notif like "entity of type with primary key has been updated" that would then allow the observable to request that back from the API.

I had the same idea behind this where I coined IODataCollectionObservable<T> and it's related IODataObservable<T> for a single entity but hit an interesting complexity curve when I started piling in related entities that the base observable was meant to track.

Taking the approach described by @tomasfabian would integrate all that, my only reservation is the complexity involved in all the moving parts ... perhaps a vision that has the parts exposed such that we as consumers can plug in and remove as needed rather than having a library that abstracts all the complexity behind this from us.

My thinking here being ... when it doesn't behave, getting tickets resolved will be tough! At least if it's more granualar in feature choice tickets might be "when I use x and y it's fine, but if I add z it breaks" rather than "my mvvm binding is broken help plz" and then you'd need to ask "uh ok, but where is this happening?".

Based on past experiences ... Look at how much we already have to build out to prove an issue in EF (they often ask for an entire repo with a repro project in it to explain the problem) ... this is going to be orders of magnitude more complex to debug.

with something like Kendo (as linked above though) you can often get away with just posting your grid init code block.

[tomasfabian]

@TehWardy I totally agree with you. We are trying to solve similar/same problems individually, such as refreshing UIs based on CRUD operations. For sure it will increase the complexity of our systems, that we will have to deal with. It will introduce new challenges to us as you mentioned, such as good traceability, observability etc. We will have to also be able to guarantee one of the following delivery options:

• at least once processing
• at most once processing
• (exactly once processing)

I also wrote a package for SqlServer's Change data capture, so you can actually even limit, filter out row-level table notifications server-side with ksqldb (I didn't see this option in the kendo example you provided us), replicate the redo logs, and so on. My whole idea was to take advantage of the stream-table duality for CRUD operations. The same could be also solved with EF's change tracker as you've mentioned, but without the ability to easily horizontally scale the solution, guarantee at least once processing, .... Of course in some cases this is an overkill.

The downside is that you have to manage Kafka brokers, ksqldb, Kafka Connect, possibly even Kubernetes since this is a highly scalable, fault tolerant distributed approach.

So what I would love to achieve, is to bring these possibilities also to the UI clients with the help of OData.NxT, because at the moment it is only safe to use it with internal server-side consumers. The ideal would be to initially load some (paged) data and subscribe to only the relevant change notifications.

@hassanhabib I'm looking forward for a potential cooperation. Let's go further with our common (community) dream:

LINQ TO EVERYTHING!

[TehWardy]

@tomasfabian I'm curious ... What does IKSqlQueryProvider actually do? My understanding is that EF (or whatever the ORM is that we use) already handles our SQL interactions and that our problem with OData is merely a client to server and back again issue rather than being a full stack one?

In short: Adding a standardised subscription mechanism in to the API layer should be enough to trigger clients in to pulling the changes after an operation is actioned.

... so my thinking is ... huh, why get the db involved here? Or did I misunderstand your position?

I only say this as I think the real magic in all this work @hassanhabib is doing is that we would end up with the pieces and thus more flexibility if done right rather than a bigger more monolithic separation of concerns issue. I am of course assuming that I have massively mis-understood your position hence my curiosity.

[tomasfabian]

@TehWardy @hassanhabib's proposal is 100% correct and GREAT and I don't have any objections against it. I was just presenting its usage with IQbservable<T> for server side observables (push queries) as a transcendence aspect of his effort. I wanted to describe him and to the OData community how I’m going to use OData.NxT.Server and LINQ expressions constructed by it.

Simply put IKSqlQueryProvider does the same as usual ORM's do, but in this case the data is pushed to the clients, not pulled by clients. It handles your KSQL interactions.

The stream processing database is necessary to do some kind of stream processing server side, it’s not mandatory. As an example you can create a push query that joins 2 or more tables/streams, projects the data, filters it and emit only the appropriate not filtered out, enriched messages to the subscriber(s). These queries are potentially endless and they continuously notify your consumers about updates or new data based on your LINQ query.

[TehWardy]

@tomasfabian Ah ok ... You're stream processing all transactions with the DB? Neat architecture, does it handle concurrency well, I guess it does if everything is a single stream.

I've been bolting on my own eventing model on top of EF such that entities in the change tracker when a save changes call takes place are fed in to a custom event raiser, my thinking was that I could do things like execute a workflow or drop a notification (or the entity) on to a signalr hub for subscribers to pick up.

I hadn't considered writing my own ORM lol!

[hassanhabib]

Updating this thread with our game plan (roadmap) for OData NxT. Check it out here: https://www.youtube.com/watch?v=o1OC8FIXj5o

We are moving the party over to this repo: https://github.com/OData/OData.Neo

[TehWardy]

Nice ... thank you for keeping us in the loop :) and keep up the great work!

[maulik-modi]

@xuzhg , @hassanhabib and @tomasfabian, Can we borrow the concept of Modular Fluent API Configuration from https://github.com/Biarity/Sieve?

[julealgon]

@xuzhg , @hassanhabib and @tomasfabian, Can we borrow the concept of Modular Fluent API Configuration from https://github.com/Biarity/Sieve?

You mean borrow it from Entity Framework, AutoMapper, or most IoC containers? It is not a new concept and definitely not introduced by that fairly new and unknown library.

[hassanhabib]

Hey guys, I created a Discord community for OData Neo here: https://discord.gg/9A8gRGT6