Added: entity reference module in core.

[jenlampton]

Description of the need

I use node and user references all the time. It would be great if we could add these to core.

Proposed solution (Updated 2022)

Move the Entity Reference contrib module into core. It has been refactored to remove any external dependencies. Entity Reference is the most obvious candidate based on the following advantages:

1) It has the highest install count. 2) It has the most flexible API and number of extending modules. 3) It is a proven solution and includes an upgrade path for D7 Entity Reference users.

Previous solution (Prior to 2022)

This was the original solution proposed in 2015. After the stalling of the new "Reference" project and the subsequent porting and updating of Entity Reference, this approach has failed to gain traction.

1) We should create a new module reference that is a rewrite of the Drupal 7 entityreference module, in language that is simpler and easier to understand. It will more closely resemble node_reference and user_reference from the Drupal 7 references module. There is now a new Reference module in contrib 2) Once the module is ready, we should move this module into core. When is the module "ready"? When it has support for node references, and all blockers in the contrib queue have been resolved.

Core Blockers for adding the reference module into core

• [x] Add view function: https://github.com/backdrop/backdrop-issues/issues/1739
• [x] Define label and status entity keys: https://github.com/backdrop/backdrop-issues/issues/3103
• [x] Add options array for files: https://github.com/backdrop/backdrop-issues/issues/2299

We also have a single issue in the module queue documenting all the issues there that need to be resolved for core inclusion: META: Blockers for getting into core

Alternatives that have been considered

• Put references into core: we decided not to put individual reference modules into core because in Drupal people seem to prefer having a single field type for everything. We would also prefer the consistency since we believe it will make a better user experience.

Additional information

Follow-up for merging in some bug fixes after this is merged:

• https://github.com/backdrop/backdrop-issues/issues/5665.

We have a follow-up issue for removing the term-reference field, and replacing it with a reference field. We'll get support for terms in first, long before we remove the previous term reference field:

• https://github.com/backdrop/backdrop-issues/issues/51.

It's been recommended that we mark that field type as deprecated for 1 major cycle of backdrop core, and remove it in the next. Use this issue for moving term reference field out of taxonomy module and marking it as deprecated:

• https://github.com/backdrop/backdrop-issues/issues/5633

The core issue below is needed for file entity support:

• Handle file stream wrappers : https://github.com/backdrop/backdrop-issues/issues/2300

---

Draft of feature description for Press Release (1 paragraph at most)

Backdrop now includes a new field type that can be used to reference other entities such as content, categories, or user accounts.

---

Advocate: @herbdool

[klonos]

:+1:

Besides field references, we should have a CKEditor addon (and/or a text format) so that people can add user/node references to a text field. Similar to how we add @user and #issue here on GitHub.

[herbdool]

I have a pull request in backdrop-contrib/references that fixes a few minor things for references, including the test. Other than that it seems to work fairly well.

[quicksketch]

I think it'd make most sense to make a single "References" module than it would to move User and Node Reference into core. Then we'd also need to convert Term Reference to References.

[mikemccaffrey]

Silly question, but why would we build off of the reference module instead of the entityreference module?

Rather than having individual modules to reference each entity type, why not support a single entity reference field which allows optional targeting by type and bundle?

[mikemccaffrey]

Wow, nevermind about entityreference. We stand little chance of untangling that huge mess of ctools abstraction hell.

However, is it unreasonable for us to build a single reference field that works across all entity types? What are the challenges in the way of doing that?

[herbdool]

Not really a silly suggestion that entity reference would be better. Nate said it might be a good thing to put into core. At least in some form. But that will be a lot harder than using this module.

If and when references is in core I think it would be a good idea to add an upgrade path from this one. On Oct 24, 2015 6:19 PM, "Mike McCaffrey" notifications@github.com wrote:

Wow, nevermind about entityreference. We stand little chance of untangling that huge mess of ctools abstraction hell.

However, is it unreasonable for us to build a single reference field that works across all entity types? What are the challenges in the way of doing that?

— Reply to this email directly or view it on GitHub https://github.com/backdrop/backdrop-issues/issues/1301#issuecomment-150856550 .

[jenlampton]

@mikemccaffrey the plan is to put references module into core because we like it's code (and dislike the entity reference code - sounds like you agree!) However, we do want to include the feature entity reference provides. That will mean writing some new code for references module for a generic "reference a thing" field, then having a multi-step form for "what kind of thing" and "what kind of kind of thing" etc etc.

[klonos]

...the plan is to put references module into core because we like it's code (and dislike the entity reference code...

I don't think I disagree here, but from an end-user point of view - not codebase-wise. I'll have to test them both again to be sure I have a better idea of their features and UX. Just wanted to point the fact that Entity Reference has been merged into D8 core:

I realize though that we are trying to be in parity with D8 feature-wise and not necessarily architecture-wise, plus this is a goal and not a promise, so best effort. So, as I said, just pointing the fact out.

[mikemccaffrey]

Hey folks, I'd just like to reiterate that I'm very interested in working on this issue and to create a reference module that combines the best of all the different existing implementations, and perhaps leverages the relative-simplicity of backdrop to streamline the code and interface.

Who else is interested in helping out with this effort?

[ghost]

Two thoughts on references :

1. Reference field is not very usefull without a useable "items" browser, where you can drag "items" from - so I'd design the browser first, the field second.
2. EntityReference benefit is mitigated if a given field instance can reference only a single entity type - else it is forced to a high level of abstraction, plugins, etc... My point of vue is to node everything, so the browser can be a simple view, and we already have a node reference...

[serundeputy]

@mikemccaffrey one idea is if you just make a fork/port of node reference or whatever starting point you want and start to architect it.

once you see the challenges and identify tasks; you can make issues in your repo and assign/appeal for help to willing developers.

I for one am looking for something to sink my teethe into for the 1.4 cycle, but I'm not sure if that will be redirect, focus on drush or both, but if there are sort of bite size issues in your repo we can all chip in. I say you be the architect and see what you can dole out.

[mikemccaffrey]

@gifad: 1. Reference field is not very usefull without a useable "items" browser, where you can drag "items" from - so I'd design the browser first, the field second.

Well, the data architecture should be pretty simple, so there is no reason not to tackle that first. Basically, we just need to store the entity type and entity id being referenced. The UX needed to help the users find the content they are looking for are the tricky part. For the time being I'd like to start with the autocomplete on the entity title field, but we can certainly also try to develop a more visual browser.

@gifad: 2. EntityReference benefit is mitigated if a given field instance can reference only a single entity type - else it is forced to a high level of abstraction, plugins, etc...

Actually, we already have a good deal of abstraction in the form of the Entity API. The information coming from hook_entity_info gives us all the entity types, their bundles, their primary key, and base table. And we can perhaps add some additional keys to those return arrays so entities in backdrop can also indicate which fields and values control things like their published status.

@serundeputy: one idea is if you just make a fork/port of node reference or whatever starting point you want and start to architect it.

Well none of the modules seem like an especially good starting point, so I think I am going to start with a new project and copy the best parts out of each of the implementations. I'll probably start it in my own github account, move it to contrib when it is ready for testing, and then we can move it into core for 1.4.0.

[mikemccaffrey]

Okay, here is the very very beginning of the new module, if anyone is interested: https://github.com/mikemccaffrey/backdrop/tree/reference/core/modules/field/modules/reference

I'm going to try to place it as a sub-module of the core field module, like List, Number, Options, and Text. (should we move other core field module into there as well, such as Date and Link?)

[klonos]

Exciting! Let me know when it is ready for a general spin.

[MrHaroldA]

Who else is interested in helping out with this effort?

Where do I sign up? ;)

[jenlampton]

👍

[jenlampton]

Pushing to 1.5. But let's get it done in contrib if ready early.

[quicksketch]

So in our weekly meeting today (https://www.youtube.com/watch?v=GsORuW80I3U) we officially pushed this to 1.5 after the status update. In the mean time, @mikemccaffrey is going to move the References project into the backdrop-contrib group on GitHub, where we can collaborate together on it as a contrib project, then move it into 1.5.0 (the release after this one), which will be September 15, 2016.

[MrHaroldA]

I've created a couple (temporary) repositories for what I've cooked for my Domains-like module; including a simple reference implementation. Is this of any use?

https://github.com/MrHaroldA/reference https://github.com/MrHaroldA/domain

[herbdool]

@MrHaroldA might be some use to collaborate with @mikemccaffrey to get a contrib version working. Looks like it's not getting into 1.4 but will live as a contrib module until then. Before it gets into core it'll need some tests also.

[mikemccaffrey]

I've moved the module that I have been working on into backdrop-contrib: https://github.com/backdrop-contrib/reference

Once we get it working there, we can move it into core.

[biolithic]

Thanks Mike!

On Thu, Jun 9, 2016 at 3:05 PM, Mike McCaffrey notifications@github.com wrote:

I've moved the module that I have been working on into backdrop-contrib: https://github.com/backdrop-contrib/reference

Once we get it working there, we can move it into core.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/backdrop/backdrop-issues/issues/1301#issuecomment-225010295, or mute the thread https://github.com/notifications/unsubscribe/ABJ8DrTebBaJVCUdwy2Uv-NlGCg2sQBKks5qKHIBgaJpZM4GLpck .

[mikemccaffrey]

Hey folks,

I've been trying to figure out a better way to allow the reference field to reference all different sorts of entity types and bundles as needed. I think I've hit upon a workable way, where on the field instance settings form, you check off all the entity type / bundle combinations that should be referenced. Then in the selection widget on the edit forms, the user would select the type of content to be referenced, and then use the autocomplete to find the right item.

Here is a screnshot of a field instance settings form, showing the widget in the default value section:

Let me know what you think of this approach!

[quicksketch]

Thanks for sharing the screenshot! My first thought on this is that the list of Entity types + Bundles could get very long very quickly. With each content type adding 2 items (comment and node), plus other types of entities that may be added in the future (files, etc).

You mentioned during weekly meetings that we also have a problem with the auto-complete only working for a single type of entity. Is that the case when creating content using this field as well as when setting the default value?

Overall, I'm not sure we need the ability to reference multiple types of content within the same field. I've never run up against such a situation. Even if I had the choice, I feel I might be inclined to make two separate fields anyway just to help the situation of auto-completing in a sane way. I might end up with user accounts and nodes that have the same title for example, I'd want to be able to select each separately rather than them all being in the same list. Displaying threaded together different types of entities also seems like it would present problems, how would you select the display modes for different types of entities (maybe you've already thought of that though). What do you think? Are we overshooting here?

[herbdool]

This looks like a 2.0 feature. For 1.0 can we aim for something that is robust but with fewer features? Being able to have autocomplete references to any kind of content or files sounds awesome but I do find it hard to know of a use case. Doesn't mean it doesn't exist, but likely will be in the 20% or less of people using reference.

(I'm using 2.0 and 1.0 in the generic sense, not specific to Backdrop)

[klonos]

Well, there is a very simple case where being able to lookup in multiple tables would be useful...

You see, with "simple" references we can for example reference a user from a node, but if we need to do the opposite, like reference nodes of various content types from a user, then we'd need to be able to lookup in multiple tables. Right?

Consider the backdropcms.org case: creating a user or Organization profile and wanting to link those to any contribution they've made. We would need a reference field that would allow multiple values and that would be able to do a lookup in module nodes, theme nodes, layout nodes and perhaps also events, blog posts, comments, documentation nodes, ...perhaps more.

That would be a valid use case and IMO not a 20% one. Of course, we can get a "simple" version of reference in for 1.5.0 and then iterate on improvements, it's just that it'd be best if we build it in such a way that adding those improvements does not require API breaking (and thus improvements can be added in the 1.x cycle).

My 2c

[klonos]

My first thought on this is that the list of Entity types + Bundles could get very long very quickly.

...let me ponder on that. Perhaps I can either find a common pattern of something that we (Drupal/Backdrop) do in such cases or come up with a new idea.

[klonos]

I googled around for a solution to this, but was unable to find something ready-made that we could use 😞

My idea was to use a "mutated" version of either a fieldset or an accordion that shows/hides parts of its content (based on attributes). Mockups upcoming...

[herbdool]

In your example @klonos wouldn't the linkage be between the user account that's recorded with the node or comment? Even with a profile a Views relationship can be made to get the user related to the profile and get all the content. Maybe I don't understand the example completely.

At any rate, from what I understand, being able to use a View as a reference was already possible in the older Drupal modules so I figure that would be a good to have here as well. In that case, any/all content types could be referenced. The key barrier, though, is crossing between entities. E.g between files/users/content type/ etc.

I'd prefer that we replicate the functionality that's already present in Entity reference and/or References before trying to add on to it.

[klonos]

...so the user loads the form and they see no bundles configured to be referenced by default:

...they click the "add bundles" fieldset end they are presented with a (long) list of available bundles to select from:

...lets say they randomly select a few of them:

...after saving the form, the selected entity bundle checkboxes bubble up and are rendered outside the collapsed fieldset that holds the rest of the available (but not selected) entity bundle checkboxes:

This way, when the user scrolls through that form, they are still able to see an overview of the selected entity bundles (no extra clicks = better UX), but at the same time, the long list of the rest of the available bundles is not in their way (less scrolling = better UX).

...lets say they need to remove a bundle and add another:

...after saving the form, the new bundle(s) bubble up outside the collapsed fieldset and the disabled ones are hidden in it:

I believe that such an implementation would address your concern about a long list of bundles @quicksketch. @wesruv thoughts?? Others?

[jenlampton]

Before we get too excited mocking up UI's, I'd like to remind everyone that we should avoid - at all costs - using either the word "entity" or the word "bundle" in any user interface. Below is an example most human beings would be able to understand:

"What would you like to reference?"

• Content
• User account
• Category

"Limit X by type"

• Post
• Page

Overall, I'm not sure we need the ability to reference multiple types of content within the same field. I've never run up against such a situation.

I also agree that we should not seek to over-engineer the initial module for core. We have 2 solutions for this in Drupal, neither of which include this feature. Though it might be nice to have it is most certainly not a requirement and if it presents any sort of challenge at all we should drop the feature. It's not very important, if it were, it would already exist :)

[klonos]

@herbdool I was thinking this use case:

• Project nodes in backdropcms.org are actually 3 separate node types (modules, themes, layouts)
• Project nodes are automatically created by a non-actual user (the "drop").
• A registered user would want to reference from their profile which projects they maintain or have contributed code or documentation to.

...in this scenario, which is an actual use case, the user is not creating the nodes, so that info cannot be pulled that way. We'd want to provide a "projects contributed to" multi-value reference field that would allow the user to autocomplete search across module, theme and layout nodes.

Although you might argue that this is likely a design flaw (project nodes could have been set to be a single content type for example, and the distinction between project type could have been a taxonomy tag), still this is a real-life use case where you can't ask the site owner to redesign their content types and migrate their content in order to be able to use references. We'd need to accommodate that.

[klonos]

...we should avoid - at all costs - using either the word "entity" or the word "bundle" in any user interface.

@jenlampton 👍 of course. The mockups were drafted in a few minutes as prof of concept for dealing with long lists of "things to be referenced" 😄

[klonos]

...the goals with the mockups was to keep the list out of the users' way (less scrolling), but at the same time provide an overview of already selected things-to-be-referenced without requiring additional clicks.

[jenlampton]

Project nodes in backdropcms.org are actually 3 separate node types (modules, themes, layouts). this is a real-life use case where you can't ask the site owner to redesign their content types and migrate their content in order to be able to use references. We'd need to accommodate that.

This is a non-issue. All existing reference fields already allow for this relationship and cutting the new feature mike has described does not affect our ability to do it.

[klonos]

@jenlampton perhaps I did not understand correctly what is the feature that @mikemccaffrey was talking about that you and @quicksketch believe should be left out. I thought that that feature was the ability to search on entity autocomplete by targeting multiple entities...

Overall, I'm not sure we need the ability to reference multiple types of content within the same field.

If that was not implemented, then how would one (in the example of backdropcms.org project nodes) be able to search through a single, multi-value autocomplete field against module nodes, theme nodes and layout nodes?

Perhaps haven't been getting much sleep lately has started to effect my brain 😜

[klonos]

...perhaps that is what @herbdool was referring to when he said

...being able to use a View as a reference was already possible in the older Drupal modules so I figure that would be a good to have here as well. In that case, any/all content types could be referenced.

That way (again in the specific use case of backdropcms.org that I have in mind), one could create a view of content of all type, then filter by content type = module AND theme AND layout, and then use that view as the reference auto-complete. Maybe? Right?

[klonos]

Are we talking about the same thing? 😄 💤

[herbdool]

I think that's what I'm saying. Existing modules allow you to use multiple node bundles, but not across entity types (can't combine users plus files plus users plus category).

[jenlampton]

Overall, I'm not sure we need the ability to reference multiple types of content within the same field.

I think @quicksketch stated this poorly. The feature that can be left out is referencing different entities (for example nodes + users) in the same field.

how would one (in the example of backdropcms.org project nodes) be able to search through a single, multi-value autocomplete field against module nodes, theme nodes and layout nodes?

Multiple bundles of the same entity need to stay, and are not on the chopping block :)

[klonos]

That makes better sense then 😄

To be fair, although at the moment I cannot think of any case where targeting multiple entities would be of use, it still does not seem far-fetched.

[jenlampton]

To be fair, although at the moment I cannot think of any case where targeting multiple entities would be of use, it still does not seem far-fetched.

I agree! However, if it presents a significant challenge it should be dropped. it's not as important as single-entity references.

[jenlampton]

CRAZY IDEA: Since we are running out of time for writing a new module for Backdrop 1.5, I want to propose instead that we add the (already ported) references module to core. This is something we can actually do in the 3 weeks we have remaining before release of 1.5, and meets all our requirements for the feature we need to add.

After References is included, we can work on adding more entity-reference like features, and an upgrade path for node/user/term reference in the future, if needed.

A universal reference module can be added in 2.0.

[herbdool]

I'm all for "minimal viable product" and incremental improvements and seems to fit well with Backdrop philosophy. References works, it can reference two types of entities which covers a lot of use cases and it is possible to provide upgrade paths as it gains new functionality to reference more types of entities and perhaps different kinds of widgets.

On the con side, it might make it harder to port modules from Drupal that rely on entity reference (though this is true of the Reference module in progress as well).

A question @jenlampton: do you have a list of "requirements" that it meets in order for to be added to core?

[docwilmot]

If it helps, I stripped the CTools from Entity Reference and got it to work normally on minimal testing. I used direct function calls to replace the plugins (should be hooks?), and cant figure out a replacement for entity_access (so just returned TRUE), but those were the only two major issues. Also the tests and examples werent checked. Have a look https://github.com/docwilmot/entityreference. Minimal testing mind you.

[jenlampton]

@docwilmot the problem is that we don't like the code style in entity reference, so we wouldn't want to put in a direct port without a major rewrite. The rewrite is what we don't have time for, so we're looking for an alternative that would be ready to go in with minimal work.

[docwilmot]

@jenlampton my thinking is that if we're going to put in a minimally viable references module this one is the better option. If we put in References,

• it won't do what we want, i.e. give the ability to reference any entity with a single field type
• we'll eventually need to remove it again, including changing the API and UI, and
• we'll need upgrade paths for user's existing node and user references

If we now minimally modify Entity Reference to tast and put it in, we can incrementally/drastically improve/change the codebase later without anyone in the front end being the wiser.

[herbdool]

@docwilmot has a point. Neither the end user nor the site builder probably cares about the code style.

Another advantage of Entity Reference is that modules like: https://www.drupal.org/project/entityreference_prepopulate, https://www.drupal.org/project/entityreference_view_widget, http://drupal.org/project/entityreference_dynamicselect_widget are easier to port. (Though it looks like https://www.drupal.org/project/multiple_selects will work with references too).

[serundeputy]

@docwilmot @herbdool those are fair and valid points, but it does not change the fact that using the Entity Reference path will not be able to be completed in the 1.5.0 cycle.

The choice here as I see it is:

1. Choose Reference brand new module and push to 1.6.0
2. Choose Entity Reference and clean up code style and push to 1.6.0
3. Choose References which has upgrade path and working code; Leave in 1.5.0 cycle and iterate to bring in the other aspects of Entity Reference and Reference

I think option 3 adds immediate value to our users, but will require more effort by devs to enhance the functionality without breaking UIs and APIs.

[jenlampton]

I tried to look at entity reference again yesterday to see how bad the code was, see if maybe we couldn't make a push on that for 1.5, but it immediately gave me a headache. Someone else want to see if there are any easy abstraction reductions we can do without breaking it?