Discussion on many-to-Many relationships (without CLR class for join table)

0xdeafcafe commented 9 years ago

Note:

Issues #10508 and #19003 have been created to track actual work on many-to-many relationships. The new issue is locked so that it can be subscribed to for updates on the implementation without noise.
This older issue has been re-purposed to be discussion about many-to-many relationships in EF Core. It will not (for the time being) be locked because it is important to us that we don’t shut down channels of communication with the community.

With regard to feedback, I think it is worth reiterating some comments made a few months ago. We read and consider all feedback (and try to make the right choices based on it) no matter how it is delivered; that is, whether it is provided politely and with respect or not. Yelling at us or others with different opinions at best makes you feel good and us/them feel bad. It doesn’t achieve anything concrete. We are doing our best to prioritize features and implement them in the best way we can. We really appreciate your feedback and it makes a big difference on shaping the product. We personally appreciate it more when feedback is delivered in a respectful way, but please don’t stop providing constructive feedback.

Original issue:

I'm assuming there isn't a way to do Many-to-Many relationships in EF Core yet, as I've found nothing mentioning how to do it. Are there any examples on getting round this? Or am I just completely missing it, and it's actually there after all.

gavrilyuc commented 7 years ago

What is the approximate date? I want to know, whether to wait or write your crutch.

ntrpnr commented 7 years ago

We need this.

ajcvickers commented 7 years ago

EF Team Triage: We really appreciate the recent feedback on this issue. Feedback from developers is extremely important in determining how resources across the team are assigned. However, please keep in mind that that feedback is only one of several inputs into the planning process and that--like in all software projects--resources are limited and every decision to implement a specific feature usually involves postponing others.

To shed a little more light onto our planning process we have created some documentation on the common considerations we take when deciding what to work on next.

We read and consider all feedback (and try to make the right choices based on it) no matter how it is delivered; that is, whether it is provided politely and with respect or not. Yelling at us at best makes you feel good and us feel bad. It doesn’t achieve anything concrete. We are doing our best to prioritize features and implement them in the best way we can. We really appreciate your feedback and it makes a big difference on shaping the product. We personally appreciate it more when feedback is delivered in a respectful way, but please don’t stop providing the feedback.

pwen090 commented 7 years ago

@ajcvickers Thank you so much for the work you and others on the team do. I think simply some of us are just wondering, without anger or judgement, if there is any new guidance on this post 2.0 release? I think the last guidance from you was "It's currently on our backlog and not scheduled for any of the immediate upcoming releases."? Totally cool if that guidance still stands, just wondering! Either way thank you for everything.

manigandham commented 7 years ago

+1 for this.

We have a complex domain model which we are waiting to use EF Core with. Unfortunately the amount of extra work and complexity of join classes, along with the lack of lazy loading, has kept this from being viable for us.

ajcvickers commented 7 years ago

@pwen090 It's still on the backlog which means that it unfortunately doesn't fit for 2.1. We will consider again when planning for the release after that.

BFPLima commented 7 years ago

I recognize the MS effort and team work, but without this feature and lazy loading is very frustrating to me.

cakirpro commented 7 years ago

+1 for many-to-many relationship support without including an entity class for the join table. +1 for lazy loading. we are eagerly waiting for these features. Thank you in advance for your effort.

fangspeen commented 6 years ago

I agree whole heartedly:

+1 for many-to-many relationship support without including an entity class for the join table. +1 for lazy loading. we are eagerly waiting for these features. Thank you in advance for your effort.

mxmissile commented 6 years ago

https://lostechies.com/jimmybogard/2014/03/12/avoid-many-to-many-mappings-in-orms/

slamotte commented 6 years ago

That's an excellent read, @mxmissile, thanks for sharing.

The comment "A junction entity is a normal entity waiting to happen" really resonates with me as something I find important more often than not is recording when the association was created, or who created it, etc. Without a join table there's no way to store that. And in reality there are often other attributes that need to be associated with the join, not just time/user-stamp metadata.

Not that this precludes the desire to have join-less many to many, but it makes you wonder if maybe explicit join tables should be considered.

ajcvickers commented 6 years ago

@mxmissile @slamotte The way we plan to implement many-to-many in EF Core should mitigate some of these issues. Essentially, the entity type for the join table will still exist in the model, it will just be created by convention and in "shadow state" so that there does not need to be a CLR type for it. There would then be skip-level navigation properties that jump over the join table entity from one many side to the other.

But because it is still just an entity type it means that:

Other properties can be added to it, even while keeping it in shadow state
It can easily be promoted to an entity type with a CLR type backing it without fundamentally changing the model
Even after promotion to a "real" entity type, the skip-level navigation properties will still work, so parts of the code that don't need to know about the join table still don't have to know.

So the idea is that people can start with simple many-to-many relationships handled by EF, but then can grow-up to explicit mapping of the join table with additional payload, etc. when needed.

cakirpro commented 6 years ago

@mxmissile Thank you for the link. It is interesting and some arguments are fair enough. However when you consider a model where you have a relatively many number of entities and complex relationships (like an entity having m-m relationship with multiple entities), it becomes very complex to model "avoiding many to many relationships" and having to create extra entities.

Therefore I like the approach @ajcvickers explained because you need both approaches. While sometimes it is enough to let EF handle the mapping, on the other hand you sometimes need extra properties (like sorting) and would be nice to have the option to assign to entities.

bassebaba commented 6 years ago

@mxmissile you are very welcome to come and:

Change all my legacy databases
Fix all other software that already uses those legacy databases

Do you guys know how many times/projects during my years with EF have used code first? 0. Database first? 100%. Code first is a luxury for "privileged" developers. All coding I have to do is against already existing databases.

Many-to-many is out there. Everywhere. In legacy systems. No number of blog posts will change that. Heck, if I got to choose, I'd probably kick out mssql altogether and go for NoSQL or something else cool that the kids use nowdays :P

This is why I'm so grumpy. I'm stuck and I really need:

A good m-2-m implementation
Lazy loading
A flawless reverse-engineer (database first) tool.

@ajcvickers okay, but how will this work in my scenario, database first, legacy systems?

mxmissile commented 6 years ago

@bassebaba honestly I feel you. I am working with a DB built in 1997, would I use EF with it? Never! NHibernate does everything (and more) I need when working with a legacy DB. Greenfield though? EF Core all the way!

I would not think @jbogard's article I posted to be "bible". It only offers some suggested points to think about when dealing with many-to-many scenarios.

ajcvickers commented 6 years ago

FYI, I just published some blog posts that show how to use and hide the join table when using many-to-many relationships in EF Core 2.0. The approach as a couple of significant limitations, but it may be useful to people until this feature is implemented. See at https://blog.oneunicorn.com/

HappyNomad commented 6 years ago

Once this feature and #8881 are both implemented then, given a many-to-many relationship between Product and Org, will the following work?

modelBuilder.Entity<Product>().HasQueryFilter( p =>
    EF.Property<IEnumerable<Org>>( p, "PurchasedBy" ).Any( o => o.ID == TenantID )
);

I'll need to access a many-to-many shadow property like this in my scenario. I'm guessing that the "skip-level navigation properties" that @bricelam mentioned will allow me to navigate straight to Product, skipping over the shadow join entity.

ajcvickers commented 6 years ago

@HappyNomad That seems correct with regard to skip-level navigation properties. @anpete would be better able to answer the filter question.

anpete commented 6 years ago

Seems reasonable 😁

mhosman commented 6 years ago

There is some time estimation to have this working?

khionu commented 6 years ago

Read up, there's none, sadly. This thread is filled with discussion about why this is a backlog feature, despite it seeming to be a needed key feature.

jbogard commented 6 years ago

@mxmissile nothing I write should be a bible for anything.

I do know I've regretted every time I've used a "many-to-many" hiding feature in ORMs, from NHibernate to EF6. But I wouldn't say every time this feature has been used by an EF6 user has been a regret. Just the times I've used it on the systems I've built.

mxmissile commented 6 years ago

I know for sure. But the pros do outweigh the cons in this case. IMO of course.

On Fri, Oct 20, 2017 at 12:54 PM Jimmy Bogard notifications@github.com wrote:

@mxmissile https://github.com/mxmissile nothing I write should be a bible for anything.

I do know I've regretted every time I've used a "many-to-many" hiding feature in ORMs, from NHibernate to EF6. But I wouldn't say every time this feature has been used by an EF6 user has been a regret. Just the times I've used it on the systems I've built.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-338293380, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOWiMJzUFQJcVVeifX9xqMlQqXIWjkRks5suOxrgaJpZM4DPrA7 .

daddydrac commented 6 years ago

This guy has a post on how to maintain relationships in many to manys with rest routes in EF core 2.0, but I havent been able to make this work in a real app with async controllers handling rest routes. Any guidance is appreciated: https://blog.oneunicorn.com/2017/09/25/many-to-many-relationships-in-ef-core-2-0-part-1-the-basics/

HappyNomad commented 6 years ago

Thanks @ajcvickers and @anpete for your replies to my question above at https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-338265306. As a follow-up, will the following also work? It involves an additional many-to-many that's between Product and AuxiliaryProduct.

modelBuilder.Entity<AuxiliaryProduct>().HasQueryFilter( a =>
    EF.Property<IEnumerable<Product>>( a, "Products" )
        .Any( p => EF.Property<IEnumerable<Org>>( p, "PurchasedBy" ).Any( o => o.ID == TenantID ) )
);

And how about https://github.com/aspnet/EntityFrameworkCore/issues/3864#issuecomment-342752182?

ajcvickers commented 6 years ago

@HappyNomad Looks reasonable to me, but I may be missing something.

dougajmcdonald commented 6 years ago

Ouch, just came across this issue and now have to re-write much of our data access code to handle the new type of joining entity on either side of the relationship. Significant issue :(

Asking for an explicit joining entity is fine, but having to name it add an ICollection<EntityAEntityB> on both sides is a killer

gregbowyer84 commented 6 years ago

oooh! deal breaker... any timeline for the implementation?

khionu commented 6 years ago

@gregbowyer84 read the whole thread, please.

HrDahl commented 6 years ago

Until this enhancement is implemented we are all forced to use a join entity class which is frustrating and a tedious task. I will defently look forward to the day they prioritize this feature. But based on the creation date of this thread, I am not sure we'll ever get it.

wandelson commented 6 years ago

Pleaseeeeeeeeeeeeee ...

jaredcnance commented 6 years ago

Can we please lock this issue. I think it is well established ☝️ that the community wants this feature. Further comments to that effect only serve to create noise for users who are subscribed to this issue and are waiting for a resolution.

jfevia commented 6 years ago

This link does not provide an acceptable solution. It's just a workaround for a key feature in ORM's. EF6 supports this out of the box so I don't understand why EF Core doesn't.

When one provides a better solution, all previous features are included and you build on top of those.

These features are high priority but we think EF Core would be a compelling release for the vast majority of applications without them Source: Roadmap

Your "thinking" is wrong. This is a setback.

sky-code commented 6 years ago

this thread started in 2015, funny thing that ms needs 3 years for implementing this feature which exists in previous version of EF (of course if ms implement this feature in 2018)

mxmissile commented 6 years ago

Instead of all the gnashing of teeth, maybe submit a PR?

ptMuta commented 6 years ago

@Travis Shush. That would require work and understanding of the underlying mechanics from them. They're already using most of their capacity crying like little babies.

On Thu, Dec 7, 2017, 16:21 Travis notifications@github.com wrote:

Instead of all the gnashing of teeth, maybe submit a PR?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-349980526, or mute the thread https://github.com/notifications/unsubscribe-auth/AFLY27ZSnqzvjqMQ5AEerivaCHW0HQXEks5s9_RzgaJpZM4DPrA7 .

jfevia commented 6 years ago

Yeah, I'll submit a PR, so the project is then neglected and replaced by whatever fancy new name MS come up with. Now, where have I seen this before...

No worries though, I'll be back to EF6 and come back to EF Core when this project is taken seriously for productivity. Maybe in another 3 years.

Good luck :)

khionu commented 6 years ago

I'm bumping the call for this thread to be locked until the EF team actually gets around to addressing this issue.

To summarize this thread for everyone who is not reading the entirety (omitting the salt):

Adding Many2Many would require a massive PR changing fundamentals of the library
- Said changes need to be done with extensive knowledge of the inner workings. Ergo, this is almost guaranteed to be an in-team task
The EF team definitely plans on doing this, eventually. No ETA
The current workaround (which is agreed to have it's benefits in a schema, but messiness in the ORM usage) is to have a Join Table Entity.

It would have been nice for this to go unsaid, but this is a public thread of professional context. Making a comment just to display the fact that you're unhappy with the status of this issue is redundant and a waste of your productivity. The EF team is well aware of this issue, and the comments over the last three years make it clear that they need to do it. The ball is in their court.

ajcvickers commented 6 years ago

Today we are taking two actions based on the extent and nature of the discussion here:

Issue #10508 has been created to track actual work on many-to-many relationships. The new issue is locked so that it can be subscribed to for updates on the implementation without noise.
This older issue has been re-purposed to be discussion about many-to-many relationships in EF Core. It will not (for the time being) be locked because it is important to us that we don’t shut down channels of communication with the community.

With regard to feedback, I think it is worth reiterating some comments made a few months ago. We read and consider all feedback (and try to make the right choices based on it) no matter how it is delivered; that is, whether it is provided politely and with respect or not. Yelling at us or others with different opinions at best makes you feel good and us/them feel bad. It doesn’t achieve anything concrete. We are doing our best to prioritize features and implement them in the best way we can. We really appreciate your feedback and it makes a big difference on shaping the product. We personally appreciate it more when feedback is delivered in a respectful way, but please don’t stop providing constructive feedback.

serafimprozorov commented 6 years ago

Hi guys,

I don't know what do you think about many-to-many support, but I think that it is epic fail to have no many-to-many support in 2018. Also it is strange that you can't implement it during THREE years. I have read a lot of internet resources and I still can't understand two things: will this feature be implemented someday and if the answer is positive when will it be implemented. Should I use EF Core and wait implementation of this feature or I have to use another ORM 'cause you have no plans or interest in implementing it?

jbogard commented 6 years ago

How is the lack of this feature affecting your ability to deliver software to your end users or customer? Genuinely curious since I intentionally avoid this feature in other ORMs for explicit join classes.

On Fri, Jan 26, 2018 at 6:40 AM Serafim Prozorov notifications@github.com wrote:

Hi guys,

I don't know what do you think about many-to-many support, but I think that it is epic fail to have no many-to-many support in 2018. Also it is strange that you can't implement it during THREE years. I have read a lot of internet resources and I still can't understand two things: will this feature be implemented someday and if the answer is positive when will it be implemented. Should I use EF Core and wait implementation of this feature or I have to use another ORM 'cause you have no plans or interest in implementing it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-360774563, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGYMtfdIBpGTb3t-rrquFMuWzRkaf54ks5tOcfJgaJpZM4DPrA7 .

serafimprozorov commented 6 years ago

@jbogard It is not affecting me or my team now, we are just selecting the ORM framework for new version of our software. As for supporting this feature in ORMs, NHibernate has normal many-to-many support, as far as I know Linq2DB also has it. And suddenly... the old Entity Framework (6-) also has this feature. I don't understand why m-to-m without additional entity suddenly became a bad idea. So if the vendor's new products have less functionality and stability than it's older libraries and frameworks, if it can't develop new features and fix lacks, if it can't to keep it's own promises to community if it breaks up the library public API, if it can't implement basic features in the normal way (for example -- group by) I prefer to use other library from other vendor. For example if I have the project depends on the EF I have to rewrite my data model if I want to use EF Core, I can't just change the library and reimplement factories. All these things make me sad.

bassebaba commented 6 years ago

@jbogard As I have pointed out before, lots of us are stuck with old legacy databases with existing data where M2M is already present, therefore we have no other choice but to keep using M2M.

jbogard commented 6 years ago

Right, so you can model that today with a join class, which is my default in any ORM.

On Fri, Jan 26, 2018 at 7:36 AM bassebaba notifications@github.com wrote:

@jbogard https://github.com/jbogard As I have pointed out, lots of us are stuck with old legacy databases with existing data where M2M is already present, therefore we have no other choice but to keep using M2M.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-360786201, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGYMj6Z6CgnvbDd2L1VcgGw21lEttVkks5tOdT4gaJpZM4DPrA7 .

danroot commented 6 years ago

@jbogard Lack of this feature has affected three projects of mine in the past few months. In each case, we wanted tagging. Instead of having just MainModel and Tag classes, we have to have MainModel, MainModelTag, and Tag classes. Anywhere that uses Tags then also needs to know about MainModelTag. In some cases, the Tag class needs to be shared with other models, so that's +1 class for each scenario, and +N mapping logic anywhere Tags are added/removed. It also affects JSON serialization, so we've had to do ViewModel mapping in places it might not otherwise be needed. I am working on some extension methods that hide some of this, but it is extra code and logic I would not need otherwise. As far as real business impact, just this week, a lack of automatic M2M in EF Core added about ~8 hours and ~150LOC in one project. So I wouldn't say it's stopping me from adopting EF Core completely, but it is a hindrance and does have an impact for us.

bfsmithATL commented 6 years ago

I would love to have this feature, it has affected a few design aspects on a project I'm working on. I've had to include five different models (& respective tables) to represent the join entities. Not the end of the world though. I'd like to see more simplicity here, but I've been able to easily work around the many:many shortfall in EF Core. There are plenty of blog articles and illustrations on how to achieve this. Using Automapper and DTOs I'm easily able to hide the "artifacts" of these join entities from models that are in transit to & from front-end clients. I also learned (later on) there is now a feature in EF Core you can use to hide properties of EF-tracked models so you can suppress these artifacts in this way without using Automapper and DTOs.

Now, does this really stink from an architectural and new-app-greenfield-dev perspective? Yes, certainly. But the gains are elsewhere. The benefits gained by sticking with EF Core (and .NET Core) is ensuring future ease of portability of the application to any hosting platform. This is huge. Usage of EF Core also ensures that the community stays strong and enough development resources stay allocated to keep it alive. You must always remember the nature of open source development. It's not "free." Everything costs money somewhere along in the line in the dev lifecycle, in the form of cash or time. The EF Core team has direction and priorities and those all must be balanced and evaluated. There is a reason this feature has not been implemented yet. I'd love to know the details as to why, just out of curiosity. We can easily petition the team in more effective ways. Nothing is stopping us community members from pulling together a gofundme campaign in the spirit of this feature and gifting a donation to the team to fund additional dev resources to work on this feature, if that is in fact the inhibitor to getting this feature. :-) Any takers? :-D

popcatalin81 commented 6 years ago

I've worked on an EF6 project where the amount of many-to-many relations compared to all others was insanely high.

Examples of M:N Relations, NOT from the real Project, just the idea: Product<->Categories, Product<->Tags, Product<->Features, Tag<->Clasiffications, Category<->Clasiffications, Product<->Classifications, Classification<->Pricings, Category<->Pricings, Product<->Pricings ... (and it keept going for 100+ of these).

The queries were highly complex involving lot's of intersections and unions, with dozens to hundreds of lines. I don't even want to imagine how the queries would look like with 100+more join tables in the model. I haven't worked on that project for years now, but lack of many-to-many would have basically meant a product rewrite, no other way to migrate from EF6 to EF Core.

For most projects lack of many-to-many is nothing more than an annoyance, for others, it's a must-have feature. I wouldn't tackle a similar project today without using an ORM with many-to-many support.

neridonk commented 6 years ago

to sum everything up it costs more time due the increase of complexitity and code coverage etc. around 1/8 +. I appericiate all your work so far and I know in america is a inflation incoming , but can you save our time pls a little bit.

mxmissile commented 6 years ago

america is a inflation incoming

Now we are using politics to get this through? lol

EmreAbi commented 6 years ago

Allah belanizi versin MS.

dotnet / efcore

Discussion on many-to-Many relationships (without CLR class for join table) #1368