Map JSON values stored in database to EF properties

[JocaPC]

SQL Server, PostgreSQL, MySQL, and Oracle databases enable developers to store JSON in columns and use values from JSON documents in queries. As an example, we can create product table with few fixed columns and variable information stored as JSON:

Product[id,title,decription,datecreated,info]

info column can contain JSON text. Use cases are:

1. I can put different non-standard information about products in JSON column as a set of key:value pairs in JSON column, e.g. {"color":"red","price":35.99,"status":1}.
2. I can have new kind of single table inheritance where I can put all properties specific to some leaf classes as a bag of key:values instead of additional columns.

In SQL Server/Oracle we can use JSON_VALUE function to read JSON values from JSN column by keys, e.g.:

SELECT id, title,
       json_value(info,'$.price'),json_value(info,'$.color'),json_value(info,'$.status')
FROM product

PostgreSQL has ->> operator and MySQL has json_extract function that are similar.

Problem

Currently, JSON fields can be mapped to entities in entity framework as string properties and we can parse them using Json.Net library. It would be good if we could map properties in EF model to (JSON text+path).

Proposal

1. Basic - Could we map property from EF model to string column that contains text and specify path of value in JSON (e.g. price, status) ? EF can extract json value on JSON path and populate property in EF model with proper type casting.
2. Medium - Enable updates of properties mapped to json value. If some EF property that is mapped to JSON values is updated, EF could format all of them as JSON and save them back in the in JSON column.
3. Advanced - Enable LINQ support over JSON properties. If we use Select(obj=>obj.price), or Where(obj => obj.price < 100) in LINQ queries, these predicates could be transformed to JSON_VALUE for SQL Server/Oracle, -> for PostgreSQL. This way we will be able to query JSON values directly from EF.

edited by @smitpatel on 16th March 2018

You can use JSON_VALUE function using user defined function feature in EF Core 2.0 https://github.com/aspnet/EntityFrameworkCore/issues/11295#issuecomment-373852015 explains how to do it for SqlServer.

Some open questions/possible subtasks:

• Partial updates to JSON documents (would likely require modeling as owned entities)
• Allow configuring JsonSerializationOptions to customize JSON property name etc., on a property-by-property basis (but also allow some bulk configuration default?)
• Use or allow using the new System.Text.Json source generator introduced in .NET 6.0?
• Indexes should probably out of scope of general relational support, as support differs considerably across databases.

Issues

• [ ] #27779

[divega]

@JocaPC We are of course busy completing more basic functionality than this at the moment but eventually I would love us to enable all of this. BTW, I think providing a way to declare indexes for values in the JSON payload (so that those queries can be optmized) would also be nice.

[gdoron]

@rowanmiller It would be helpful and user friendly if we could add to our POCO a dynamic property and through the modelbuilder mark it as JSON (modelbuilder.Entity<Product>().Property(x=> x.CustomFields).AsJson() where CustomFields is dynamic/ ExpandoObject).

And then we will be able to nicely query the dynamic object like:

context.Products.Where(x=> x.InsertDate > DateTime.Now && x.CustomFields.Retailer.Name == "Doron")

// Or 
context.Products.Where(x=> x.InsertDate > DateTime.Now && 
                           x.CustomFields.Retailer.Items.Contains("Foo"))

Though I'm just not sure how indexing will work in SQL-Server 2016 for JSON columns @JocaPC

OData is using something similar with Open Types and it's intuitive and very easy to use.

[omric-axonize]

OData Open Type story sounds like a perfect fit to SQL-Server 2016 + EF Core!

[JocaPC]

You can add index on computed column that exposes JSON value:

1. Create non-persisted computed column with expression JSON_VALUE(jsonCol, '$.Retailer.Name')
2. Create index on that computed column

Original queries don't need to be rewritten. When SQL Server finds a query that uses JSON_VALUE and if path in JSON_VALUE matches computed column that has index, it will use indexing. See https://msdn.microsoft.com/en-us/library/mt612798.aspx

[erichexter]

I would like to see this support.

[DaveSlinn]

My team is currently investigating EF for a new application we are undertaking and I was hoping to use this very feature. If we were so inclined to invest in working on this feature, for our own selfish needs, would that be worthwhile contributing?

[ikourfaln]

We realy need this functionality

[ikourfaln]

At the moment, I think that we can just use FromSql function to execute raw SQL and include JSON_VALUE. example:

var contacts = _context.Contacts.FromSql("SELECT Id, Name, Address, City, State, Zip " +
                                        "FROM Contacts " +
                                        "WHERE JSON_VALUE(Info, '$.Moniker') = @p1", moniker1);

[soycabanillas]

My two cents (what I'm doing for now) for proposal 1:

I configure my entities this way:

public class ConfigCTransfer : EntityMappingConfiguration<CTransfer>
{
    public override void Map(EntityTypeBuilder<CTransfer> entity)
    {
        entity.HasJsonValue(x => x.Serialized, y => y.Data.Status, z => z.FStatus);
    }
}

x.Serialized is the JSON/text field. y.Data is the object that is serialized into Serialized. Status is just a property of Data. z.FStatus is the column where I'm going to store the value of y.Data.Status.

the heavyweight is done by the HasJson extension method. I include the code (not a clean one) for copy paste, but the important thing here is that I'm setting the FStatus property as a computed column, as @JocaPC commented.

    public static void HasJsonValue<T, U>(this EntityTypeBuilder<T> value, Expression<Func<T, string>> jsonFieldExpr, Expression<Func<T, U>> jsonPathExpr, Expression<Func<T, U>> dataFieldExpr) where T : class
    {
        var jsonPathSegments = PathFromExpression(jsonPathExpr);
        jsonPathSegments.RemoveAt(0);
        var jsonPath = string.Join(".", jsonPathSegments);

        var jsonFieldSegments = PathFromExpression(jsonFieldExpr);
        var jsonField = string.Join(".", jsonFieldSegments);

        var sqlDataType = new SqlServerTypeMapper().FindMapping(typeof(U));
        //var typeName =  sqlDataType.DefaultTypeName;
        var typeName = sqlDataType.StoreType;
        var property = value.Property(dataFieldExpr);
        var isNullable = property.Metadata.IsNullable;
        var isForeignKey = property.Metadata.IsForeignKey();
        var maxLength = property.Metadata.GetMaxLength();
        if (maxLength != null) typeName = typeName.Replace("(max)", $"({maxLength.Value})");
        var columnDefinition = $"CAST((json_value([{jsonField}],'$.{jsonPath}')) AS {typeName})";
        if (isNullable == false || isForeignKey) columnDefinition = columnDefinition + " PERSISTED";
        if (isNullable == false) columnDefinition = columnDefinition + " NOT NULL";
        value.Property(dataFieldExpr).HasComputedColumnSql(columnDefinition);
    }

    //For other solutions, see:
    //http://stackoverflow.com/questions/1667408/c-getting-names-of-properties-in-a-chain-from-lambda-expression
    //and
    //http://stackoverflow.com/questions/2789504/get-the-property-as-a-string-from-an-expressionfunctmodel-tproperty
    public static List<string> PathFromExpression<T, P>(Expression<Func<T, P>> expr)
    {
        var result = new List<string>();
        MemberExpression me;
        switch (expr.Body.NodeType)
        {
            case ExpressionType.Convert:
            case ExpressionType.ConvertChecked:
                var ue = expr.Body as UnaryExpression;
                me = ((ue != null) ? ue.Operand : null) as MemberExpression;
                break;
            default:
                me = expr.Body as MemberExpression;
                break;
        }

        while (me != null)
        {
            string propertyName = me.Member.Name;
            //Type propertyType = me.Type;
            result.Add(propertyName);
            //Console.WriteLine(propertyName + ": " + propertyType);

            me = me.Expression as MemberExpression;
        }
        result.Reverse();
        return result;
    }

The EntityMappingConfiguration thing is just a friendly way of having the configuration of an entity in its own class. You can see the implementation and the discussion here: https://github.com/aspnet/EntityFramework/issues/2805#issuecomment-218548872

If anyone needs helps with this, just ask me.

This has several limitations, though, at least: 1 - The properties are computed columns. The value will not be set until the entity is saved. But it will be of use for indexing. 2 - It doesn't support polymorphism, at least not in an easy/clear way. You can't use the same data for different entities... or I haven't found the way:

a - You can't use different classes to point to the very same table #6001 b - You can't use different classes to point to the very same field #240

Sorry if something about my conclusions is not completely accurate. There are others more capable of validating them. I just tried to expose my experience so far.

[soycabanillas]

And...

3 - This solution only supports SQL Server. No support for in memory database.

[bjorn-ali-goransson]

I think the comment of @ikourfaln is superiour (and simplest to implement, ie already done) in combination with the flexible mapping idea.

[ma3yta]

When it will be done?

[ikourfaln]

@Ma3yTa :disappointed: not yet decided. must be moved from Backlog to a Milestone

[bjorn-ali-goransson]

I find the implementation complexity for this feature, compared with its usefulness, to be amazingly disproportionate.

2016-09-29 16:48 GMT+02:00 IKOURFALN Slimane notifications@github.com:

@Ma3yTa https://github.com/Ma3yTa 😞 not yet decided. must be moved from Backlog to a Milestone

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFramework/issues/4021#issuecomment-250488400, or mute the thread https://github.com/notifications/unsubscribe-auth/AAoyABSu8XPPrfjW7sIuoEmL_Vfjs_zsks5qu8_GgaJpZM4Gxzpa .

[adrien-constant]

I also think this would be very interesting to have JSON query support in EF Core.

[bjorn-ali-goransson]

Actually, for me, LINQ support would be secondary (or ternary) to the actual mapping functionality. Nice, of course, but if it takes much more time the "milestones" should be split up.

IMO ... LINQ (as in: the query expression parsing bit) is a leaky abstraction that's not all that useful. It's often quite confusing. The IEnumerable extensions are stellar, though, the all-time best idea that the .NET team has conceived of. But I digress.

[bjorn-ali-goransson]

Is it possible to work around this limitation in todays codebase?

For example, I'd like a simple integer to be mapped to a Link object that contains an ID ...

[rowanmiller]

@bjorn-ali-goransson it's not natively supported but with Field Mapping support (added in 1.1 - currently in preview) you could map the integer to a private field and then have a property that converts to/from the Link object. See this post for details on 1.1 Preview1 and how to use field mapping https://blogs.msdn.microsoft.com/dotnet/2016/10/25/announcing-entity-framework-core-1-1-preview-1/.

[rpokrovskij]

Why just not to start from simpler things: smartly serialize entity (with all its "navigation") to JSON using model information to avoid circular references? Together you will be able just to insert JSON DB fields "as is". Then user will be able easy to send the serialized result to further layers (usually SPA). It should cover 95% of JSON DB field usages.

[DenisAgarev]

Yes, agree that we want mapping for just selection. It's not critical to serialize/deserialize on update but would be very helpful with odata sorting and filtering.

[ma3yta]

Good to have something like UserType in NHibernate http://blog.denouter.net/2015/03/json-serialized-object-in-nhibernate.html?m=1

[ma3yta]

@rowanmiller maybe this helps to implement json support https://www.linkedin.com/pulse/quering-json-using-hibernate-marvin-froeder

http://jasperfx.github.io/marten/

https://github.com/jasperfx/marten

[jovanpop-msft]

@rowanmiller @divega Maybe JSON mapping could be implemented using backing fields. Here is a proposal: https://github.com/aspnet/EntityFramework/issues/7394

[ma3yta]

Guys from NHibernate started to implement this feature https://nhibernate.jira.com/plugins/servlet/mobile#issue/NH-3930

[marchy]

This would be fantastic to see. More and more data in databases is unstructured or has shallow hierarchies that need not span multiple tables thus don't map to simple columns (such as the Complex Types feature would bring). JSON is a perfect structure for these scenarios but right now it takes not only the programming complexity hit, but more importantly a performance hit to serialize/deserialize as string blobs.

When we are competing against NoSQL DBs that naturally crunch JSON with high-performance, the whole relational world could really use a one-up to compete better against those solutions – which often come with their own slew of problems (lack of integrity, migration / data corruption over time).

[jovanpop-msft]

Here is the article that explains how to map objects and arrays to JSON fields. Using not-mapped/backing fields we can map non-value properties to JSON columns in database.

The bigges thing that is missing is some ability to translate LINQ queries like this:

_context.Blogs
         .Where(b => b.Owner.Status == 1)
         .OrderBy(b => b.Owner.Email)
         .Take(10)
         .Select(b => b.Owner.Name);

to something like this:

SELECT TOP 10 Name
FROM Blogs b
      CROSS APPLY OPENJSON( b.Owner ) WITH (Status int, Name nvarchar(400))
WHERE Status = 1
ORDER BY Email

[marchy]

@jovanpop-msft Fantastic article link Jovan. Thanks for that share. The in-database translation is exactly the golden feature we were looking for.

Fingers crossed for future support on this in the not too distant future =)

[ma3yta]

@divega @rowanmiller Guys, When expect this feature will be done?

[CoskunSunali]

I would love to see this implemented. I personally think that this is more of a "must have" rather than a "nice to have" feature.

I understand that there are other basic requirements that have to be implemented before resources are available to implement this one but still...

[gdoron]

Must have, on a feature that was just introduced in SQL server 2016 that almost no application is taking advantage of. It's not useless by any mean, but the definition of must have is not "this feature will help me very much"

P.s. This feature WILL help me very much...

[bjorn-ali-goransson]

Yes it's important.

But on the other hand, if you need a JSON storage solution, you're not really the primary target audience for EF Core - it's relational DB consumers. This can be found out both by looking at the EF Core goals and history, and by looking at the fact that neither this nor the flexible mapping is getting any attention. The team is busy developing relational database support (and other features).

Maybe you should start to look at other solutions. There are RavenDB (costs though), MongoDB, CouchBase, and some other serious alternatives.

[CoskunSunali]

@gdoron

the definition of must have is not "this feature will help me very much"

Exactly what you said. The two definitions are different and I am free to define it as a "must have" feature where as you define it as a "will help me" feature. The EF Core team also is free to say "not to be implemented" and close this issue. Right? I prefer the team to speak for themselves.

And oh yeah, we are very much taking advantage of the native JSON data support in SQL server 2016. You might not, that is your concern.

@bjorn-ali-goransson Forget about EF Core, SQL server itself is a relational database, however it has native JSON data support introduced in SQL server 2016. PostgreSQL and MySQL are also relational databases but they support native JSON data. Thus your assumption that I am not really the primary target audience for EF Core is just wrong. And if it was right, SQL server 2016 - being a relational database - should not support JSON data at all. Right?

The point you are missing is that an application could be implemented on top of a relational database and at the same time support dynamic schema only for some of its fields on some specific tables.

So the suggestion you make to look at no-SQL databases is not really of any help.

I suggest you read https://blogs.technet.microsoft.com/dataplatforminsider/2016/01/05/json-in-sql-server-2016-part-1-of-4/ where it states:

As the most-requested feature on the Microsoft SQL Server connect site, with more than 1,000 votes, support for JSON text processing has been added to SQL Server 2016.

SQL server developers could have very well said "this is a relational database, you are not our primary audience, go look at no-SQL databases" but they have not.

Everyone's requirements does not have to fit in yours. Just like the JSON text processing being the "most requested feature" on SQL server 2016 and may be you never needing it at all.

---

I don't know why you guys paying attention to unnecessary things like how I defined the feature (must have vs will help me) or if the EF Core team has other things to do first (I already said I understand that!) but seriously, I wrote my comment so the team of EF Core can speak for themselves and also know that there is one more developer here who is very much looking forward for having this feature implemented at some point in the future.

[CADbloke]

SQLite also supports JSON in System.Data.Sqlite: https://stackoverflow.com/questions/36671285/system-data-sqlite-cant-load-extension via the JSON1 extension: https://www.sqlite.org/json1.html

[CADbloke]

some things I found on the topic for those looking to self-solve this... http://www.reddnet.net/entity-framework-json-column/ which references https://stackoverflow.com/a/14785553/492 and links to source code now at https://github.com/NullDesk/TicketDesk/blob/ea1b7cb711989a2a2a9fc39d3753f695e7397bdc/TicketDesk/TicketDesk.Domain/TdDomainContext.cs#L104

This is how I do it now .. https://stackoverflow.com/a/37207034/492

[adamvanvliet]

Definitely a must have. Stuck with SQL Server for some of our enterprise customers, but wanting to take advantage of JSON column. Would love to see this receive some attention.

[asadmalik3]

@divega What is the status of this request? Are you guys going to work on it?

[ajcvickers]

@asadmalik3 This issue is in the Backlog milestone. This means that it is not going to happen for the 2.1 release. We will re-assess the backlog following the 2.1 release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[smitpatel]

Update: You can use JSON_VALUE function using user defined function feature in EF Core 2.0 https://github.com/aspnet/EntityFrameworkCore/issues/11295#issuecomment-373852015 explains how to do it for SqlServer.

(also added to first post)

[weitzhandler]

See #2282. Also please check out Impatient, and read the issue I posted there.

[thomasouvre]

I would also love to have these features. In the meantime, I'm working on an EFCore extension project that compiles the call to OPENJSON through a linq expression and an extension method. You can see the results here.

[neridonk]

This would be cool, i hope you have enough time due the danger of a currency burn

[weitzhandler]

Any development on this? This feature can potentially unlock a whole new world of dynamic types with EF! And in turn support NoSQL with EF.

[CoskunSunali]

I bet this is one of the most requested features of EF core with 130+ 👍 and 30+ ❤️ yet no attention resource wise.

[mutanttech]

I am reading this today and still no solution to query the JSON Data stored in SQL Server 2016 in a table column using EF CORE. Should we resort to using ADO.Net for now and calling procedures. Please suggest.

[smitpatel]

@mutanttech - You can use https://github.com/aspnet/EntityFrameworkCore/issues/11295#issuecomment-373852015 for querying part, if that is the only blocker.

[CoskunSunali]

@smitpatel Be careful using it. Check details at https://github.com/aspnet/EntityFrameworkCore/issues/11295#issuecomment-449521486

[smitpatel]

@CoskunSunali - Thanks for bringing it to attention. Though I believe, it would not be difficult to define function based on OPENJSON similarly.

[mutanttech]

Thanks @CoskunSunali , @smitpatel for the response. The solution can work but not really sure if reliable. I really wanted to use EFCore, but I have started to think use Stored Procs & Dapper to get the stuff done for now. What do you think of this approach?

[aaronhudon]

@mutanttech the solution provided in #11295 is legitimate and reliable. The query translates to JSON_VALUE correctly. It's just unfortunately not bundled with EFCore.

[vovikdrg]

Would be nice to be able to query like this

drop table dbo.#TestTable 
CREATE TABLE dbo.#TestTable 
   ( 
   Id int NOT NULL, 
   Settings varchar(512)
   ) 

INSERT INTO dbo.#TestTable (Id, settings)  VALUES (1,'{"Languages": ["uk"], "Content": [ "modern", "ballroom"]}') 

select *  from dbo.#TestTable  
    CROSS APPLY OPENJSON(settings,'$.Languages')  WITH (lang   varchar(10)   '$')
    CROSS APPLY OPENJSON(settings,'$.Content')  WITH (content   varchar(10)   '$')
where ISJSON(settings) > 0 and lang in ('uk') and content in ('ballroom')