2nd level caching (database layer). Move away from Entity Framework

[mariannk]

EF affects performance because slow it still doesn't support 2nd level caching (cache entities between HTTP requests). We have to find some workarounds such as custom entities for caching, model caching, etc. It could make nopCommerce really very-very fast

And looks like EF team doesn't plan to support it. Everything will be fine when EF start supporting 2-level caching. But there's no any estimate.

So we can consider other ways to go. For example, use Dapper instead of EF. Or we can fully migrate to it along with linq2db (https://github.com/linq2db/linq2db)

[m1dst]

+1 for Dapper Even the devs of Orchard have started a rewrite using Dapper.

[VahidNaderi]

I suggest using dapper like what Stackoverflow does, go on with EF and use dapper for wherever needs performance tuning.

[m1dst]

@VahidNaderi which is exactly what I have done in my plugins

[AndreiMaz]

@m1dst are they (Orchard) going to migrate to Dapper in the next versions?

[m1dst]

According to Sébastien Ros from the MVC team who is writing Orchard2 in ASP.NET CORE using Dapper. As for older versions, I have no idea.

See ASP.NET Community Standup (3rd May) https://youtu.be/ELsO-qdaMQQ?list=PL0M0zPgJ3HSftTAAHttA3JQU4vOjXFquF&t=994

[dmatrak]

What about petapoco micro ORM?

[rc201612]

+1 for Dapper.

[oumiller21]

+1 for Dapper

[rchamorro]

I think migrating to linq2db (https://github.com/linq2db/linq2db) would be easier because it allows doing linq queries and it is really fast (https://github.com/FransBouma/RawDataAccessBencher)

[samdubey]

+1 for dapper

[iyilm4z]

+1 for dapper of course

[haitran2011]

+1 for dapper

[nhunghuynhthihong]

+1 for dapper

[B-Esmaili]

+1 for dapper

[MaxMommersteeg]

+1 for dapper

[DevPartner]

+1 for Dapper

[FredBell]

Funny we just downloaded 3.8 and were planning on getting rid off all the EF code and replacing it with dapper. Our plan was to use stored proc as much as possible becuase from experience we found them easier to maintain large project than an ORM. This is because the DB and the code are 2 separate layers - and the sprocs are the interface - so it allows to optimize the DB for performance without changing the code.

I dont know if the nopCommerce team will ever get it done... but just in case +100 for Dapper :)

PS: We love nopCommerce and we appreciate all the guys working on it.

Fred

[B-Esmaili]

I am thinking of two major improvements on Nopcommerce. First one is migrating to Asp.net core , which as the MS team claims it processes requests 2300% faster than latest version of ASP.NET MVC (4.6). and the other one is migrating to dapper which demands giant change on code base.If these two happens to Nopcommerce i would know no rival (in terms of performance) for Nop.

[rfiori]

+1 Dapper Principally in read cases its is necessary

https://github.com/StackExchange/dapper-dot-net#performance-of-select-mapping-over-500-iterations---poco-serialization

[samdubey]

Has Anyone implemented forked repository with Dapper?

[matiasmolleja]

@AndreiMaz @m1dst Yes, the new version of Orchard is using Dapper already. They use Yessql, wich "wraps" Dapper itself. The Net.Core + Dapper couple seems to provide amazing performance improvements, according to their weekly meetings. And I should mention that they are comparing to nhibernate, not EF.

[MaxMommersteeg]

I don't think we have to move off from EF entirely. For most developers it is still the easiest ORM out there. However, nopCommerce's performance could be improved at some points. Why not have a hybrid solution, using Dapper for executing critical queries and keep using EF for the others? I prefer a solution like Dapper over adding more Stored Procedures, that imho decrease understandability of nopCommerce. https://msdn.microsoft.com/en-us/magazine/mt703432.aspx

[tomakali]

As of now EF is lagging behind in terms of performance https://www.exceptionnotfound.net/dapper-vs-entity-framework-vs-ado-net-performance-benchmarking/

https://dontpaniclabs.com/blog/post/2014/05/01/speed-comparison-dapper-vs-entity-framework/

will there be focus on speed on auto scale configurations?

[cyborgdoom]

There has been a ton of performance enhancements in EF Core...Both those articles are a mute point now.

[tomakali]

still +1000 for dapper

[tolpeit]

+1 Dapper

[cyborgdoom]

Well it's just like anything out there... go with whatever makes you happy.... nothing is perfect.

[ivanslater]

+1 Dapper!!! Lets make Nop faster as it should be!!!

[cyborgdoom]

Funny how people keep linking to old articles... EF Core has had major performance enhancements .

[FredBell]

Not really funny. There are still major performance issues. I just did a deployment and had to rewrite over 50% of the code base. It took me over 6 months of work to do it. Originally I had page loading between 1 to 2.6 seconds. Which of course is nuts. I rewrote the domain layer and data access layer (among many other things) and now I got it to load in 100 to 200 ms

So believe what you want. But numbers are a pretty good indicators.

Anyhow. Don't get me wrong. I don't want to start another discussion on this topic. I really don't care anymore. I have done what needed to be done and now all is good. As a matter of fact it is to my advantage to have my competition use a slow platform. So this is likely the last time you hear from me on this topic. Unless one day I wake up and feel particularly generous, then maybe I will help out... maybe...

Regards,

Frederic

On Aug 7, 2017, at 18:01, Cyborgdoom notifications@github.com<mailto:notifications@github.com> wrote:

Funny how people keep linking to old articles... EF Core has had major performance enhancements .

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/nopSolutions/nopCommerce/issues/239#issuecomment-320812556, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMKb-3d8hyl4WRoFyA0HQkeM1yBuSAgAks5sV6VQgaJpZM4HFu9F.

[cyborgdoom]

Out of curiosity how many records were you dealing with? I just really haven't seen any kind of issue like you describe and we have an app that deals with lots of records using .NET core web api and EF Core... and angular4 on the front end...

On Mon, Aug 7, 2017 at 6:03 PM, FredBell notifications@github.com wrote:

Not really funny. There are still major performance issues. I just did a deployment and had to rewrite over 50% of the code base. It took me over 6 months of work to do it. Originally I had page loading between 1 to 2.6 seconds. Which of course is nuts. I rewrote the domain layer and data access layer (among many other things) and now I got it to load in 100 to 200 ms

So believe what you want. But numbers are a pretty good indicators.

Anyhow. Don't get me wrong. I don't want to start another discussion on this topic. I really don't care anymore. I have done what needed to be done and now all is good. As a matter of fact it is to my advantage to have my competition use a slow platform. So this is likely the last time you hear from me on this topic. Unless one day I wake up and feel particularly generous, then maybe I will help out... maybe...

Regards,

Frederic

On Aug 7, 2017, at 18:01, Cyborgdoom <notifications@github.com<mailto: notifications@github.com>> wrote:

Funny how people keep linking to old articles... EF Core has had major performance enhancements .

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ nopSolutions/nopCommerce/issues/239#issuecomment-320812556, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMKb- 3d8hyl4WRoFyA0HQkeM1yBuSAgAks5sV6VQgaJpZM4HFu9F.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nopSolutions/nopCommerce/issues/239#issuecomment-320820950, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGpSEjGnsMTJVP5eDAPtkQN6kdvCTQzks5sV7PagaJpZM4HFu9F .

-- Dan Swatik (860) 368-0545 http://www.dswatik.com

[sdanyliv]

I’m crying when people decide to move to dapper and return to raw SQL. Linq is the best invention of this century. You can write typesafe queries and make query decomposition which is not availaible in SQL world. There is linq2db, that is faster than dapper and very flexible in creating own extensions. It supports more than 10 DB Engines with their SQL dialects! You can write complicated linq queries that will be optimized as much as possible. It is more closer to SQL than EF and contains bulk DML operations. You can create custom aggregation functions, use Window Functions, MERGE statements, bulk insert records and many other SQL related things that you can imagine. Linq is not slow, it is how EF trying to work with database - hide from you database specific things which is bad when you need to tune performance. After fighting with EF developers usually give up and write pure SQL, which is return back to bad mainteinability and support.

[a-patel]

'maybewont' is suitable tag for this issue.

[frankiDotNet]

+1 for linq2db! Fast, easy configuration and easy using!

[a-patel]

EntityFrameworkCore 2.x is better now

[ivanslater]

I have heard about it too. But I have never seen any tests about EF Core X Dapper or other DB framework yet.

[sdanyliv]

Yes, EF.Core is faster for now. But it is still buggy and with limited SQL support.

There are some performance testing links: https://liiw.blogspot.com/2017/02/ef-core-vs-linq2db.html (Linq to DB) https://weblogs.asp.net/fbouma/net-micro-orm-fetch-benchmark-results-and-the-fine-details

[samdubey]

https://wildermuth.com/2018/07/28/Avoid-Lazy-Loading-in-ASP-NET

[Leftyx]

My 2 cents. I don't think the problem in NopCommerce is EF Core vs Dapper. Going through the code base and doing some tests and profiling I can see a huge number of queries running to load a page, especially the first time. I don't quite understand why on earth we are loading localized properties one by one for each field of a product. SELECT + 1 must be avoided at all costs especially in high traffic applications. Do we really need to cache localized properties for products ? Wouldn't faster and better in terms of performances and resources to join them and load them all the time ?

This is what I am talking about:

4 queries to load 4 fields on the same entity ? Seriously ? I have 334 categories/subcategories in my ecommerce.

[a-patel]

Agreed to @Leftyx comment.

[clearbucketLabs]

My 2 cents. I don't think the problem in NopCommerce is EF Core vs Dapper. Going through the code base and doing some tests and profiling I can see a huge number of queries running to load a page, especially the first time. I don't quite understand why on earth we are loading localized properties one by one for each field of a product. SELECT + 1 must be avoided at all costs especially in high traffic applications. Do we really need to cache localized properties for products ? Wouldn't faster and better in terms of performances and resources to join them and load them all the time ?

This is what I am talking about:

4 queries to load 4 fields on the same entity ? Seriously ? I have 334 categories/subcategories in my ecommerce.

Agree, when looking at application insights, the amount of queries/hits to the DB for this application gets up to 1 million on low traffic. you don't want to see high traffic.... plus the cache default is 60 minutes, non-sliding for everything. they all expire at the same time.

if these aren't fixed, someone's just going to fork a new version of nopcommerce that saves you 10x on your hosting bill. even then, more memory and CPU only gets you so far... it's just inefficient, worse than wordpress and woocommerce.

[toddca]

Check out GrandNode

[tomakali]

If nopcommerce needs a good upgrade It should be Onecore + dapper/linq2db + headless architecture

Or just fork umbraco and build eCommerce framework on top of it

Like WordPress + woocommerce

[clearbucketLabs]

Check out GrandNode

That's intersting but MongoDB? no thanks. it still has non-async functions, boatload of DI into controllers.... that alone is a performance killer. but i guess it's better than stock.....

[Leftyx]

@clearbucketLabs,

I've tried to fix some of those issues but I smashed my face against a wall. Too many layers to fix. Too many crazy things are going on there. My biggest problems is with localized properties. We have 300K products with descriptions in 4 different languages. Each localized attribute for an entity is fetched an put in a cache. A product has at least 5 attributes. Loading a list of products means a few hundreds of queries, just for that. Same thing for categories and we have more than 300. There's no chance for us to move to Azure as we would have to pay a few thousands of $$$ to run this monster. At some point I was trying to remove the idiotic Repository layer and generate some real view models but it's a massive job. Some of those layers are not decoupled. The dependencies are crazy. We are thinking of building a new e-commerce from scratch using a modern approach with a modern front-end framework like vuejs. Let's see.

[arvand]

What was the conclusion? Is there any future plan to support dapper or be able to switch to another ORM?

[clearbucketLabs]

What was the conclusion? Is there any future plan to support dapper or be able to switch to another ORM?

There's no need, EF core has been shown to be as quick as dapper with no tracking used so no point using TWO frameworks. the problem is lazy loading and not including navigation properties in a single query which ends up doing N+1 selects and various other anti-patterns.

[arvand]

Well, Dapper still seems to be faster. I remember in our project EF was lagging behind especially when queries got complex, but choosing the suitable ORM is choice that the developer needs to make based on the project's needs. Performance can be an issue for many large scale e-commerce sites. Another problem aside from navigation properties is that EF context is passed to other layers which makes it hard to switch other ORMS.

So to rephrase the question: Is there a plan to support other ORMs and how can we contribute to it?

[clearbucketLabs]

Well, Dapper still seems to be faster. I remember in our project EF was lagging behind especially when queries got complex, but choosing the suitable ORM is choice that the developer needs to make based on the project's needs. Performance can be an issue for many large scale e-commerce sites. Another problem aside from navigation properties is that EF context is passed to other layers which makes it hard to switch other ORMS.

So to rephrase the question: Is there a plan to support other ORMs and how can we contribute to it?

I doubt it, but it's open source so give it a kick start. you need to add a layer of DTO's and remove lazy loading at the presentation layer to make it easier to swap the ORM.

[Leftyx]

: Is there a plan to support other ORMs and how can we contribute to it?

@arvand few hundreds of queries which run to load most of the pages wont' be faster with dapper. EF is slow but the problem sits somewhere else.