search

backdrop / backdrop-issues

Issue tracker for Backdrop core.
144 stars 40 forks source link

Added: entity reference module in core. #1301

Closed jenlampton closed 2 years ago

jenlampton commented 9 years ago

Description of the need

I use node and user references all the time. It would be great if we could add these to core.

Proposed solution (Updated 2022)

Move the Entity Reference contrib module into core. It has been refactored to remove any external dependencies. Entity Reference is the most obvious candidate based on the following advantages:

1) It has the highest install count. 2) It has the most flexible API and number of extending modules. 3) It is a proven solution and includes an upgrade path for D7 Entity Reference users.

Previous solution (Prior to 2022)

This was the original solution proposed in 2015. After the stalling of the new "Reference" project and the subsequent porting and updating of Entity Reference, this approach has failed to gain traction.

1) We should create a new module reference that is a rewrite of the Drupal 7 entityreference module, in language that is simpler and easier to understand. It will more closely resemble node_reference and user_reference from the Drupal 7 references module. There is now a new Reference module in contrib 2) Once the module is ready, we should move this module into core. When is the module "ready"? When it has support for node references, and all blockers in the contrib queue have been resolved.

Core Blockers for adding the reference module into core

We also have a single issue in the module queue documenting all the issues there that need to be resolved for core inclusion: META: Blockers for getting into core

Alternatives that have been considered

Additional information

Follow-up for merging in some bug fixes after this is merged:

We have a follow-up issue for removing the term-reference field, and replacing it with a reference field. We'll get support for terms in first, long before we remove the previous term reference field:

It's been recommended that we mark that field type as deprecated for 1 major cycle of backdrop core, and remove it in the next. Use this issue for moving term reference field out of taxonomy module and marking it as deprecated:

The core issue below is needed for file entity support:

Draft of feature description for Press Release (1 paragraph at most)

Backdrop now includes a new field type that can be used to reference other entities such as content, categories, or user accounts.

Advocate: @herbdool

jromine commented 8 years ago

@jenlampton yes, this is why I initially ported references instead.

herbdool commented 8 years ago

Seems that @docwilmot has a working module, though?

On Aug 26, 2016 3:02 PM, "John Romine" notifications@github.com wrote:

@jenlampton https://github.com/jenlampton yes, this is why I initially ported references instead.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/backdrop/backdrop-issues/issues/1301#issuecomment-242823294, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBZex1TLrCRbZ6VYvClgiXo8efDb3x3ks5qjzhXgaJpZM4GLpck .

jenlampton commented 8 years ago

@herbdool there's a big difference between "working" and "core quality". We can't just add a working module to core because it's closer than all the alternatives. Most modules get a rewrite when they go into core, and upgrade paths are tested & included.

I'm hearing concerns about how to get off of node_reference, user_reference, and term_reference if we add references now. Please keep in mind that we don't get away from that problem if we go with entity_reference or reference instead. We still need the upgrade path, it's just a matter of whether we need it at the time we put something in core, or if we can do it later on when we improve the something that's in core.

It sounds like there's some concern about my crazy idea (and understandably!). If we are going to push a reference module to 1.6 instead of 1.5 I'd rather stick to @mikemccaffrey's approach and move forward with reference. (I also like it's namespace better :)

quicksketch commented 8 years ago

Hi all, no updates here in the issue, but we discussed this in person during the last weekly meeting: https://www.youtube.com/watch?v=MAMIaVeyr1g

A little banter in there were we weighed both sides. But ultimately we punted this to the PMC for decision making. By unanimous decision we're moving this to 1.6.0 rather than trying to rush something through.

I think Mike's new Reference module is the best bet we have, but we have a lot more work to do here (upgrade paths especially). Let's continue working on this for the next release. The Roadmap has been updated to reflect this: https://backdropcms.org/roadmap#1.6.0

jenlampton commented 7 years ago

I'd like to commit to working on an upgrade path from term_reference starting around Dec 15th. I've been dealing with issues like this one in Drupal, and the more I think about it, I don't feel good about leaving two ways to do the same thing in core. It's only going to create confusion for users, and extra work for developers to have to support two different APIs to provide one feature in contrib. We need to kill term_reference.

jenlampton commented 7 years ago

I know @mikemccaffrey has been working hard on getting tests ready for Reference, but I haven't gotten around to upgrade paths yet. Tonight is feature freeze, so I don't think references is going to make it for 1.6.

We do have a solid contrib module that can be used today. We can work there on upgrade paths, and continue making progress in contrib until we can move references into core.

I'm going to move the milestone from 1.6 to 1.7. If the contrib project is ready by the next release I'd like to consider it again for inclusion.

klonos commented 7 years ago

Tonight is feature freeze, so I don't think references is going to make it for 1.6.

As I also suggested during our past dev meeting, lets please favor a stress-free 2017 for contributors over getting things done. I am totally fine with taking our time and getting things done the right way the same way we pushed WYSIWYG to next releases until we felt confident about it.

@mikemccaffrey 👍 and please keep up the good work and do not let deadlines bring you down. I am sure you'll eventually get there. I just wish I could help.

jenlampton commented 7 years ago

Since reference now has a stable contrib release, we can work out issues there and re-schedule this for core inclusion when we feel it's ready. Removing the core release milestone.

ghost commented 5 years ago

Just closed https://github.com/backdrop/backdrop-issues/issues/51 as a duplicate of this one.

stpaultim commented 2 years ago

We've had several dev meetings over the last several weeks on the topic of development priorities for the 1.21.0 release. This issue has gotten a lot of attention. @hosef has indicated that quite a bit of progress has been made along with a scaling back of expectations regarding what must be accomplished before this module gets into core.

There is some mixed feelings about whether or not it's realistic to get this into the 1.21 release of core.

It is my understanding, that adding this to core simple means:

1) Testing and fixing the current contrib module 2) Dropping this into the core modules directory

Folks can help us get this into core by doing the following:

1) Installing and using the stable release of this module and reporting bugs. https://backdropcms.org/project/reference 2) Looking through the issue queue for this module and helping with any issues found there. https://github.com/backdrop-contrib/reference/issues

@hosef - can you say anything more about where help is most needed?

I'm inclined to schedule a sprint day focused on this module - if there is interest and anyone thinks it might be helpful.

Getting this into the 1.21.0 release would be a pretty big victory, in my opinion.

herbdool commented 2 years ago

I'm not convinced it's ready.

In this issue summary it lists core blockers that prevent this from going into core. Plus there are blockers in the contrib issue queue.

I've brought up issues that need to be resolved before I'm convinced it's ready too.

philsward commented 2 years ago

As much as I would love to see one module replace all others and included in core, I feel like it needs a lot more scrutiny before including it. It would be a great addition for new sites, but I think it needs to be ready for the migration of existing BD and potentially D7 sites to be considered for core.

When I played around with reference a month ago or so to replace term_reference, it was obvious it isn't ready to replace existing methods (I don't remember the specifics off hand). I ended up using term_reference instead. Surprisingly, references is a lot farther along than I figured, but it still isn't quite ready.

Point is, this should, as contrib, do everything the other modules do, then be tested for migration from the other modules to this one before being included in core.

This isn't a core module to take lightly.

I would be OK with it not being prime-time for a seamless D7 to BD migration (that could come later) but at least a solid migration from the other iterations on BD to the new BD iteration should be solid before including it in core.

Long story short, if reference isn't ready and put into core, people will still need the other modules to accomplish what isn't ready in reference, making it that much more difficult for mass acceptance.

That's my take on it. I'm good either way and will do what I can to help test to move things along.

As a side thought, once it is in core, we may also need a way to block the install of references and entity references from the project installer with a note that those modules are only available as a manual download for existing sites that haven't yet migrated those modules to reference. I'm guessing this will mostly encompass D7 sites.

klonos commented 2 years ago

Back in July this year, in https://github.com/backdrop/backdrop-issues/issues/1739#issuecomment-877487307 @jenlampton mentioned that that was the only blocker.

klonos commented 2 years ago

... we may also need a way to block the install of references and entity references from the project installer ...

When we merge contrib modules into core, we take care of this as part of the task. We have the backdrop_merged_modules() function for that + we usually update the respective contrib module with something like dependencies[] = system (<1.21) which makes it "incompatible" with newer versions of core.

herbdool commented 2 years ago

... we may also need a way to block the install of references and entity references from the project installer ...

No way should we block those other modules from being installed! Reference is incomplete and doesn't have an ecosystem that works with it.

Personally I still don't understand why we don't just put entity reference into core instead of trying to recreate it all. There's been some vague references to the code base, but I don't understand the argument. Entity Reference works in Backdrop, has other contrib modules that work with it or can be easily ported. And any issues people have with the codebase can be fixed. If someone will just point them out specifically.

Reference is not ready. And there are multiple blockers as I pointed out in the module's issue queue.

olafgrabienski commented 2 years ago

No way should we block those other modules from being installed! Reference is incomplete and doesn't have an ecosystem that works with it.

I agree, and if I recall correct, the question has been discussed somewhere with the result to not block other modules without having reached feature parity and a working upgrade path. Does someone remember the discussion and can provide a link to it?

philsward commented 2 years ago

@herbdool @olafgrabienski I should have clarified... I suggest to block those only after reference is in core and the appropriate action for site builders is to migrate the contrib to the new core module because the contribs are considered obsolete. Not before it's in core.

@herbdool, it was mentioned a while back that entity_reference is a spaghetti of code that is difficult to incorporate, especially with its heavy reliance on ctools. I don't personally know or understand the details of this though. I think @jenlampton can offer some insight on this?

herbdool commented 2 years ago

@philsward entity reference is already ported and doesn't rely on ctools any longer so that part isn't an issue. (It'ss not the only part of core that used to rely on ctools). I help maintain it and don't think the code is any worse than some parts of core. It's definitely not spaghetti code.

jenlampton commented 2 years ago

Personally I still don't understand why we don't just put entity reference into core instead of trying to recreate it all.

Here's the reasoning from the Original Post:

Put entity_reference into core: we decided that the code for entity_reference module in Drupal 7 was too obfuscated for Backdrop. It depends on the Entity API most of which is not strictly necessary, as long as you use the core APIs that are provided. A rewrite would allow us to create a module with identical features that is simpler and easier for everyone to contribute to.

The current problem is that entity_referece depends on entity_plus, but everything is possible with core alone. The unnecessary layers of abstraction in entity_plus make the code harder for people to work with, reducing the number of people who will be able to contribute to it.

I haven't looked at the code for a while, so I don't have any specific examples, but I do know that entity_reference is still missing features that were present in both references and term_reference, and if the code had been approachable and easy to work with, those features probably would have been included by now.

herbdool commented 2 years ago

The only reason entityreference depends on entity_plus is because core does not yet support entity_view(), which is something that reference is also blocked on (https://github.com/backdrop/backdrop-issues/issues/1739). There is one other, optional call to entity_plus but that wouldn't have to go into core. So removing dependency on entity_plus is no more troublesome than it is for reference.

If the focus had been on improving entityreference UI instead of creating something from scratch, we'd probably be much further along by now.

docwilmot commented 2 years ago

I also feel we should move Entity Reference to core. No point struggling (and apparently failing, after all these years) to build a new module from scratch, when this one has been proving itself on D7 and Backdrop for years. We dont entity_view to be in core first either.

jenlampton commented 2 years ago

I thought it was the metadata wrapper madness that was why it needed entity_plus, and that definitely doesn't belong in core. The entity_view() issue seems like it could be an easy fix.

If entity reference doesn't use metadata wrappers (I might be thinking of something else) and everyone else is in favor of putting entity reference in core instead of reference, we should probably start a new issue to manage the todos for what that process would look like. I don't think anyone has seriously considered it since we've always been working toward reference.

Just noting here that we would be setting aside our second principle in favor of the third. (which might be okay?)

hosef commented 2 years ago

Ok, I spent some time reading through this issue and several other old issues that were related.

Some things I would like to point out before moving on:

As I understand the history of the issue, there were 3 main problems with using the entityreference module:

If people now think that the entityreference module is stable enough to be the default reference solution, we could just rename the functions, update the UI, and call it a day.

hosef commented 2 years ago

As far as entity metadata wrapper goes, I think that is way more simple than what we currently do to look up field values from entities.

philsward commented 2 years ago

One implementation to rule them all is what I'm in favor of.

How we get there or why is less important to me than making sure whatever does get included, is rock solid.

indigoxela commented 2 years ago

One thing to consider: what do people actually use.

It seems like the "Reference" module has the fewest installs.

Some personal feedback: one key feature for me is, that I can use a view as base for referenced nodes/users/terms.

philsward commented 2 years ago

Does it make more sense to go off BD installs or Drupal installs?

olafgrabienski commented 2 years ago

Does it make more sense to go off BD installs or Drupal installs?

In my opinion, both are interesting. Here is a current snapshot from the Drupal statistics, both for D7:

Both modules are quite popular. Interesting that Entity Reference is more popular in Drupal but less popular in Backdrop.

laryn commented 2 years ago

Note: Judging popularity is a bit of guesswork depending on when a port was done or an official release made (so that it showed up in the installer)

indigoxela commented 2 years ago

Interesting that Entity Reference is more popular in Drupal but less popular in Backdrop

Keep in mind that both modules have not been released at the same time. References has been ported 2016 and Entity Reference in 2019. So the latter is available for a way shorter time.

Ah, @laryn just beat me to it :stuck_out_tongue_winking_eye:

ghost commented 2 years ago

Here're my thoughts as a core committer:

reference has/is being written for Backdrop from the ground-up. It doesn't have any old, legacy code, no baggage, etc. It's clean, simple, and easy to maintain going forward. entity_reference on the other hand is a Drupal port and, as such, is complex, has extra baggage, dependencies, etc. and raises the barrier-to-entry for new/existing maintainers.

For people coming to Backdrop from Drupal, they have options in contrib that match what they're already using in Drupal. So they're taken care of. Therefore, in my mind, an option in core is specifically for people building a new Backdrop site from scratch. For them, I'd much prefer to see the clean, simple solution that is reference, even if that means it takes longer to finish and include (again, we have working contrib modules for the time-being, and down the track I'm sure there'll be an 'upgrade' process from contrib's entity_reference to core's reference).

herbdool commented 2 years ago

It doesn't have any old, legacy code...

Isn't almost all of Backdrop "legacy code"?

There's a big benefit to legacy code where a lot of exceptions and bugs have already been encountered and fixed.

MrHaroldA commented 2 years ago

There's a big benefit to legacy code where a lot of exceptions and bugs have already been encountered and fixed.

Only if that code is completely unchanged, and used on the same framework it was written on. Since it's a port, I'd say that it is unstable until proven otherwise.

klonos commented 2 years ago

For the record, I agree with what @BWPanda said. We should aim for simplicity and allowing more than a handful of people to be able to contribute to whatever ends up being in core. People with existing D7 sites can use the 2 existing equivalent contrib modules when they move to Backdrop, and when we eventually have an upgrade/migration path in place, they can move to the core-provided solution if they choose so.

herbdool commented 2 years ago

@MrHaroldA not exactly starting from zero here. There are over 200 installs. But we're not just talking about entityreference here, but also comparing it to reference.

I find it curious that people are making a big deal about entityreference when core has a lot of examples of equally "complicated" modules. I'm not exactly sure what complexity people are pointing to in entityreference (would be good to get some actual examples from objectors). Layout has a complicated system of classes. Views uses a system of plugins for "access", "argument", "display" that extend the views_plugin class that is quite similar to how entityreference has plugins for "behavior" and "selection". Surely no one is arguing we "simplify" Views as well.

I'm not even sure what simplicity would look like in these examples. Does it mean that only procedural programming should be used? (I'm skipping over Entity Meta Wrapper since I've already pointed out entityreference doesn't actually rely on it.)

This issue seems to be the big exception to the codebase in Backdrop. We've ported and simplified where possible, and also maintaining most of the backwards compatibility.

philsward commented 2 years ago

As of Nov 7, 2021

Reported Drupal 7 Installs: 532,890 Reported Entity Reference Installs: 191,555 Reported References Installs: 68,129

Combined installations of a module that deals with referencing: 259,684

Nearly half of all D7 sites use some sort of module for referencing entities. This doesn't include those who are only using Term Reference. We should use this to show urgency that a decision needs to be made so efforts can begin making it happen if the goal is to help capture D7 folks looking to migrate.

This discussion feels like it needs some urgency, but as folks have pointed out, BD has working contribs for both of the D7 variants which means there is really no urgency after all...

I bring this up because I'm stuck trying to figure out: "Which one do I use that will cause me the least headache later?" At the very least, I think a formal decision needs to be made and clarified with urgency so those of us who want to build out a site for the future, don't have to worry about choosing a path that will have bad potential side-effects later.

Based on all of the various discussions, it appears that the actual debate is between the BD "Reference" module and the D7 "Entity Reference" module. I think we can mostly agree that the D7 "References" module is off the table as a contender. (Yes, I realize the D7 modules are ported to BD; I mention D7 to point out where they originated)

Between BD Reference and D7 Entity Reference, it appears that Entity Reference is stepping up to be the winning choice. Is this the general consensus we can all agree on or are there further discussions that need to take place in favor of BD Reference?

ghost commented 2 years ago

@philsward D7 ER looks to be the best choice for new sites, unless you're willing to help test & improve B Reference. I agree there's no urgency for a core reference-type module, so I suggest waiting for Reference to be finished, including upgrade paths from the D7 modules, and then have that as core's reference-type functionality.

One more argument for Reference in core is that work has already been started on it. Choosing ER instead will void all of the work done on Reference so far. Whereas the work done on ER will never be voided as it'll live on in contrib regardless (I don't see a Reference living on in contrib should ER be chosen instead).

philsward commented 2 years ago

@BWPanda the main work on Reference was done by someone who doesn't appear to be active in the community any longer. I have a feeling they won't have an opinion either way if we abandon the work. @hosef who has recently worked on Reference, is in favor of ER.

Rereading through the most recent comments, there is an overwhelming agreement to abandon the Reference module in favor of Entity Reference.

The decision by the community is very clear and doesn't appear to need any more debate.

Like @jenlampton mentioned, the next step is to create an issue that outlines the work necessary to make it happen.

Who is the most familiar with ER and could come up with a task list to follow in the new issue?

stpaultim commented 2 years ago

A little over 2 weeks ago, I posted an update suggesting that we try to push the Reference Module forward with the hope that it MIGHT get into 1.21.0 (or at least that we MIGHT make some substantial progress in that direction). I was pushing this option, because I THOUGHT we had consensus that this was the best solution AND because the consensus at the development meetings appeared to suggest that the Reference module was nearly ready to include in core (with some scaled down expectations).

Since then a number of other people have vocally expressed concerns that the Reference module (in their opinion) is NOT close to being ready and that the Entity Reference module is a better candidate anyway. The discussion in this issue (among those participating) moved dramatically in that direction, while all discussion in our weekly dev meetings suggests that the general policy has not changed and that Reference is still the preferred solution. See recording of meeting on Nov 11th (https://www.youtube.com/watch?v=tb767sTTgjA) - the first 20 minutes was spent on this issue.

As far as I can tell, only one core maintainer is advocating for Entity Reference, while at least two core maintainers still (in my reading of the situation) seem to prefer the Reference module.

I would love to see the core maintainers discuss this issue and come to some consensus. Until that happens, I would say that this issue (at least the Reference module option) has stalled and is unlikely to get much attention as long as there is no consensus on which direction we are moving.

jenlampton commented 2 years ago

We could escalate this to the PMC, and ask them to get a decision from core maintainers.

ghost commented 2 years ago

I have my opinion (expressed above), but whether or not the PMC agrees, it'd be good to get them to make a decision (based on core committer's suggestions and their own deliberations) so we can move forward either way.

herbdool commented 2 years ago

Here's a branch of Entity Reference that no longer requires entity_plus: https://github.com/backdrop-contrib/entityreference/tree/issue-21c. All the tests are passing. (Both ER and R would still benefit from a view and buildContent method in core to make the code better.)

By the way, the high level discussion is not always helpful. The assumptions have been that ER is full of "legacy code" and R is fresh and new. But take a look at these two files, for example: https://github.com/backdrop-contrib/entityreference/blob/issue-21c/views/entityreference_plugin_display.inc and https://github.com/backdrop-contrib/reference/blob/1.x-1.x/views/reference_plugin_display.inc. The file in Reference is almost an exact copy from Entity Reference. Good. Copy and clean-up.

As @hosef said above: "If people now think that the entityreference module is stable enough to be the default reference solution, we could just rename the functions, update the UI, and call it a day."

I'm all for it.

stpaultim commented 2 years ago

By the way, the high level discussion is not always helpful. The assumptions have been that ER is full of "legacy code" and R is fresh and new.

I believe that this is a vast over simplification of the arguments I've heard against Entity Reference. However, I am not really in a position to contribute to this discussion very much. I have a strong suspicion that a real time discussion of core committers could resolve this issue. It seems as if some folks who were initially resistant to Entity Reference as a solution have softened their positions. Although, I'm still not sure we have consensus.

I believe pretty strongly that a real time discussion sometime in the next month, might position us to move this issue forward substantially during this release cycle. It is my sense that this issue is too complicated to resolve in the issue queue alone.

I think it would be much better to get a consensus from core committers rather than take this to the PMC.

stpaultim commented 2 years ago

Interesting (and I think relevent) comment in the forum: https://forum.backdropcms.org/comment/3743#comment-3743

keiserjb commented 2 years ago

I decided to test the Reference module since Reference/References/Entity Reference is on the list to get added to core. I must say the names and the different functionalities are confusing. I like free tagging and think it is expected behavior. Reference doesn't do it yet. https://github.com/backdrop-contrib/reference/issues/30 I became interested because I recently got https://github.com/backdrop-contrib/noderefcreate working and wondered how it fit in with this discussion. It relies on References and Node Reference but if Reference offered similar functionality then it wouldn't be necessary.

stpaultim commented 2 years ago

Recently the discussion seems to have swung in the direction of Entity Reference. But, I'm still unclear where this is going.

quicksketch commented 2 years ago

I did a code review of Entity Reference today, and my opinion is that the module is well situated to be pulled into core with minimal changes. It no longer has any dependencies (like CTools, Entity API, or Entity Plus). It leverages a plugin system similar to Views and Layout modules, with classes that provide grouped collections of functionality.

At this point I think it would be worthwhile to start working with a PR that puts entityreference module in core. Creating the PR we should try to maintain the project git history, as we did with Views when it was merged into core. And then we'll see about the other changes needed to incorporate it, such as changing the default install profile, adding to backdrop_merged_modules(), and adding any update hooks needed.

We still have not "officially" decided on using entityreference, but we have agreed during weekly meeting that entityreference is the most used, most flexible, and most depended-upon solution we have. At this point, I think it would also be the easiest solution to integrate, simply because it already has all the features we need.

herbdool commented 2 years ago

Thanks @quicksketch for reviewing the module. I'll attempt a pr and see what people think.

docwilmot commented 2 years ago

What should be the approach for handling taxonomy_term_reference though? We cant remove it; contrib may be implementing Field API alter hooks etc. Options?

Or maybe this isn't a problem? I dont know Field API that well.

quicksketch commented 2 years ago

What should be the approach for handling taxonomy_term_reference though?

I was thinking something along these lines:

  1. Add Entity Reference to core.
  2. Separate the Term Reference field from Taxonomy module.
  3. Update install profiles to use Entity Reference and disable Term Reference module by default.
  4. Provide either a core conversion tool or contrib module to convert from term reference to entity reference.
  5. In Backdrop 2.x: Remove Term Reference module. Move to contrib.