Implement automatic schema generators for all Drupal entities in GraphQL 4.x

[MurzNN]

GraphQL 3.x branch have very useful feature, that automatically expose all entities, that exists on Drupal, to GraphQL schemas. So users can simply install module, and get all entities available via GraphQL, that's cool!

But, as I understand, this feature is not exists in 4.x branch, because all manuals describe that users must manually describe schemas only for needed entities and other stuff.

This is big upgrade stopper for 3.x version users, and for most of new users too!

Is this feature planned to implement in 4.x core module? If not in core, maybe implement it into separate submodule?

Without this feature, for users will be better to use REST or JSON API, that expose all entities automatically.

[MurzNN]

If we afraid, that this auto-generator will catch many not-needed entities, this can be solved via whitelist or blacklist (or modern allowlist / blocklist :smile:) on module settings page. Also we can limit of recursion to 2 levels, but and with those limits this will be better, than manually creating schemas.

[Kingdutch]

I think not automatically exposing anything in version 4 of this module is actually a feature rather than a detriment.

Any automated mapper (be it for REST, JSON API or GraphQL) will use the data structure that Drupal uses. However, whatever was best for Drupal may not be most useful to (or even desired for) your front-end consumers. Examples are that all nodes would appear to be related even if they have very different functions. What does it mean to expose a Paragraph Entity? The group module also provides a very flexible building block that may be used in different manners. In Open Social the group module is used in 3 different contexts (groups, discussions and courses). Exposing all those as the same "Group" data structure, isn't great practice from an API perspective.

One thing that all these things have in common though is that Drupal has a great internal system of describing how these things differ (bundles) and are connected (entity relations). Lets look beyond simply dumping Drupal's internal data representation out into the rest of the world and instead see if we can build a schema builder that allows a user through a UI to describe the structure of their graph based on what data is available. This could be a separate module in contrib or a submodule of the GraphQL module.

[philbert440]

I think this would make the more sense to be added to the examples module

[prrenish]

GraphQL 3.x branch have very useful feature, that automatically expose all entities, that exists on Drupal, to GraphQL schemas. So users can simply install module, and get all entities available via GraphQL, that's cool!

But, as I yunderstand, this feature is not exists in 4.x branch, because all manuals describe that users must manually describe schemas only for needed entities and other stuff.

This is big upgrade stopper for 3.x version users, and for most of new users too!

Is this feature planned to implement in 4.x core module? If not in core, maybe implement it into separate submodule?

Without this feature, for users will be better to use REST or JSON API, that expose all entities automatically.

Really needing this cause there are not a lot of examples and tutorials out there which gives the confidence to use this library.

[Kingdutch]

I think if the automated exposure is what you're looking for then it makes more sense to just use the 3.x version of the module. In this case I don't think a higher version number necessarily means it's better, just that it's different.

@klausi or @fubhy perhaps an explanation of the differences (and future support plan) of the two versions can be added to the Drupal.org project page to help people understand the differences?

As for examples for the 4.x version, we're actively developing with the 4.x version in Open Social, which is open source: https://github.com/goalgorilla/open_social/pull/1994 I'd encourage you to dive into the code to see what we're doing. I think it's rather well documented (although I'm biased as I've written the code). Be sure to look at the comments throughout the code as well as at the commit messages using GitHub's blame feature. A lot of the commits contain paragraphs on what's happening. You may even be able to read the commit history as a story of getting started with the GraphQL module.

[MurzNN]

So automatic exposure of all Drupal entities into GraphQL interface for 4.x will not be possible because of refactoring the module? Or it can be still implemented in 4.x, like in 3.x, but not yet done?

[klausi]

Updated the project page a bit, basically the 3.x version is now in maintenance mode and will not get bigger updates anymore.

I would be open to add an automatic schema-generating submodule in 4.x if somebody wants to contribute that! There is no technical limitation for such a submodule and should work with the API details we use in 4.x. We don't plan to work on it otherwise, since we also think that a designed GraphQL schema makes more sense for us.

[iBobik]

@klausi How about to add link to this discussion to that paragraph on the project page? It took long time for me to find this statements. :-)

Btw, I think it yould be also good to write your statement about schema-generating submodule in 4.x to that paragraph, because it is better to hire potential community work for this than for maintainance of 3.x.

[klausi]

Hm, I think I would not add that to the project page, since we don't want to encourage people to use a generated schema.

So I think I would leave the discussion just in this issue for now.

[MurzNN]

since we don't want to encourage people to use a generated schema.

Why using a generated schema is bad for people? Declaring all entities and its fields by hands is too expensive, easier to reuse JSON API that automatically expose all entities, than describe manually schema for each entity and field.

So, on most sites we still use 3.x version of Drupal GraphQL module, because of having GraphQL Core submodule, that generates automatically schemas for all available entities, and can't upgrade to 4.x, because of miss this feature.

[simesy]

it makes more sense to just use the 3.x version of the module. In this case I don't think a higher version number necessarily means it's better, just that it's different.

I understand this thinking, but my concern is that choosing v3 will leave me without an upgrade path to improvements in v4. The project page lists v3 as the "old version" and in "maintenance mode", which suggests future improvements may only exist in v4.

Edit: maintainers should support whatever you want to support, I'm just noting that choosing v3 has signs of being a regrettable decision. :D

[pjohans]

Just tried to upgrade to 4.x. That was a bad idea with our setup - nothing exposed out of the box.

I totally agree with MurzNN - automatically exposing all enitities is a great feature in GraphQL .. and the reason we are using it

[erikseifert]

You can provide schema generator's via drush for existings bundle/entites. This would greatly speed's up development.

[wendelltron]

You can provide schema generator's via drush for existings bundle/entites. This would greatly speed's up development.

This is the best idea. Then the generated schema can be pared down as needed. I am surprised this issue is as old as it is and that there has been no forward progress on v4 for users like us.

Clearly the maintainers do not understand how many people out there are using 3.x and have no easy path forward.

Even something as simple as generating an allNodeArticle schema similar to what v3 gives you out of the box is clear as mud from the existing v4 documentation.

And even if you get entity_load_multiple working (which isn't documented well) the performance of v4 when you have about 4-5k nodes on a headless site has been a problem (vs v3) for the same scenario).

With v4 it is better not to expose 'everything' I get that.

And while v3 is not bug free v4 feels far from complete and not for anything but trivial setups and trivial node counts. (when we're talking about static site generation).

Would love to see a drush command that provides schemas for bundles/entities and equivalent allNodeWhatever entries in explorer as v3 did and then ca tune from there. Is there a bug bounty for this? I'd contribute $500 to it.

Glad the d9 patch for v3 was mainlined and we'll be using that for another year.

[jmolivas]

I decided to give a try this weekend and took one of the the examples at the GraphQL module to programmatically register queries and resolvers and so far I was able to:

• Write a DataProducer Plugin that take advantage of a Plugin Derivative to register multiple DataProducer plugins depending on the number of ContentTypes on a site
• Programmatically register queries (singular & plural) for multiple ContentTypes
• Programmatically register resolvers for Connections on those ContentTypes
• Programmatically register resolvers for multiple Fields within a ContentType using addFieldResolver and providing support for composing Producers.

NOTE: I did hardcoded an Array containing the data that was used for the automation mentioned below :point_up:

What will be next?

• Find a way to get all the required data from Drupal via _entitytype.manager, _entityfield.manager and/or from the SDL definition using directives.
• Take advantage of some code at _socialgraphql from OpenSocial and use it programmatically.

[jmolivas]

Yay! first functional PoC for the GraphQL v4 module providing support for automatic schema generation.

At this very moment only string, string_long field types, and some other default Drupal Node fields as uuid, title, uid are registered.

NOTE: A new Gatsby Drupal Source plugin it's been also being developed on parallel using Gatsby GraphQL toolkit in order to take advantage of Gatsby v4 features.

[iBobik]

@jmolivas Great to see that. Just a one request, because it is hard to change later:

Could you please convert field names to camel case? V3 module does it and we all have queries already written like this.

There also could be another thinks about naming things what could be backward compatible.

[jmolivas]

@iBobik I fixed that already, thanks for the comment

[iBobik]

Nice. There are also "field" prefix, entity types and bundle are concatenated (NodeArticle) and queries names are different.

Should I document it all now, or it will be better time later?

[jmolivas]

It will be great to have all those documented somehow to provide some kind of backward compatibility.

We can probably provide something like a v3 compatibility mode and try to generate the schema as close as posible to that version to avoid to much disruption on current implementations.

The main goal of the schema generation in this work, is to provide a GUI (a YAML file for the initial phase) to allow users to define how they want to name things and what to expose.

[jmolivas]

It took me a couple of extra beers/hours than expected and had to write a new DataProducer plugin but I got Media Images resolving automagically using Drupal and GraphQL v4 module

[jmolivas]

New updates: Field prefix added to types (Node, Paragraph) to avoid type collisions. Support for Paragraph schema generation providing similar capabilities as the Node schema generator.

[jmolivas]

Initial support for Unions was done.

Union name was done using ContentType+FieldName to avoid collisions but a generic naming could be done without much rework.

[laurencefass]

I am new to Drupal GraphQL. I dont really understand why a wide variety of producers and provided, but resolvers are not, but reading all the comments here: would it be more useful to offer a CLI or new module to provide selective, customisable schema generation?

[Kingdutch]

I dont really understand why a wide variety of producers and provided, but resolvers are not

This may be slightly off-topic for this discussion but it's important to know that "resolvers" are a function that "resolve a value for a field" and "DataProducers" are "a Drupal Graphql concept to use Drupal's plugin system to build resolvers". So while a resolver is not necessarily a DataProducer, a DataProducer is always a resolver (through the DataProducerProxy).

My DrupalCon NA talk may help clarify the differences.

[jmolivas]

More work done on the GraphQL Compose module,

Support for MediaImages and Users added, yes SDL and resolvers are no longer hardcoded within the module.

Related to Media Images, at this very moment only url, width and height are exposed but a more advanced custom producer could be configured.

And finally support for automatic schema generation for Media Images that allow multiple number of values entity_reference.

[jmolivas]

Another big update for today. Initial support for automatic schema generation for Paragraphs entity_reference_revisions.

[jmolivas]

@laurencefass

would it be more useful to offer a CLI or new module to provide selective, customisable schema generation?

Coming from a big fan of CLI ... Drupal Console creator here o/

Having a CLI to provide the generation will limit the number of user that can take advantage to only users that feel confortable using the CLI.

Instead of that I am writing the GraphQL Compose module to allow that selection and customization via the Drupal GUI.

[jmolivas]

Initial release for automatic schema generation for GraphQL v4.x is here via the GraphQL Compose contributed module.

Some of the main features are:

• Support for Content types
• Support for Paragraphs
• Support for Media Images
• Support for (entity_reference & entity_reference_revisions)

Use Composer to give it a try

composer require 'drupal/graphql_compose:^1.0@beta'

More info on the d.o project page => https://www.drupal.org/project/graphql_compose

[jmolivas]

Happy to show the DX improvements for Drupal GraphQL integration in this video, where you can see how to use the automatically generated GraphQL fragments on your queries using the Drupal Compose module.

[jmolivas]

A little more detailed information about the generated fragments.

As you can see on the images, entity_reference (user, media, language, taxonomies, etc..) and entity_reference_revisions aka paragraphs are supported and fully functional.

[Kingdutch]

GraphQL Compose now exists as a module to solve this issue 🎉 so I'm closing this :)

[ayalon]

We had a deeper look at the code of the Drupal Graphql Compose module. We decided to not go this route. Because we heavily depend the auto generated schema of GraphQL 3, we decided to implement our own schema generator for GraphQL 4. It does not rely on intermediate "twig template" generation and things like this. It is fully configurable with an UI. You can choose which entity to expose and which fields. It support a lot of special cases and related related modules and can expose not only entites but also field configuration and other things. You might want to have a look at it here: https://www.drupal.org/project/graphql_core_schema