Auto-inserting Blocks

[ockham]

• Part of https://github.com/WordPress/gutenberg/issues/41236.

Tasks list

• [x] https://github.com/WordPress/gutenberg/pull/49789
• [x] https://github.com/WordPress/gutenberg/pull/50103
• [x] https://github.com/WordPress/gutenberg/pull/51294 (alternative approach)
• [ ] Explore the approach where we don’t update anything in the editor initially, but users see a banner informing that the content is different on the front end because of the auto-inserting blocks. Users could preview it, inject them to edit the content, etc.
• [ ] Explore auto-inserting blocks in the editor on the client just after parsing the HTML but before initializing the editor.
• [x] 🛠️ #51449 (Explore modifying the REST API response by parsing HTML and auto-inserting blocks and then serializing it back to HTML.)

What problem does this address?

There are plugins in the WordPress ecosystem that, when activated, auto-append a Login/out link to any Navigation menu present on the site (via the wp_nav_menu_items filter):

These no longer work if an FSE theme is active, where all the presentational elements are provided by blocks rather than PHP code. And while there even is a Login/out link block in WordPress (see e.g. https://github.com/WordPress/gutenberg/pull/29766), it has been argued that adding it manually isn’t on par with the user experience offered by simply activating a plugin. So in the following, we’ll assume that this is the baseline UX that we want to retain.

We cannot simply add the wp_nav_menu_items filter to the Navigation block, as it would allow arbitrary modifications of the block HTML – which could potentially cause it to be no longer parseable. It has thus been requested to add a counterpart filter to the Navigation block that allows for more “controlled” modifications.

However, another problem remains: Any (PHP) filter that modifies the rendered markup (of a dynamic block) on the frontend doesn’t allow for those modifications to be edited by the user inside the (FSE) block editor. Frontend/editor parity is an important tenet in Gutenberg, so we’ll add that to our constraints. Gutenberg contributors have thus been wary about adding such a filter, suggesting that in a block theme world, the ideal counterpart to a pre-FSE filter might not be another filter, but a different kind of extension mechanism altogether.

In a previous discussion, @mtias had clarified one more constraint: While a plugin should be able to add a block upon activation that would be both visible on the frontend and modifiable in the editor, that block should not be serialized. Instead, there would need to be a mechanism that would allow the user to either save the block to whatever template they’re editing, or dismiss it. I adopted the term “Ghost Block” to refer to this kind of blocks that are there but aren’t 👻

What is your proposed solution?

A mechanism that would vaguely consist of the following pieces:

• Add a filter that allows changing the allowlist of inner blocks for a given parent block.
• Add a filter that allows adding a ghost block to a relative position, specified e.g. by a parent block and a "first"/"last" argument, or a sibling block and a "before"/"after" argument.
  • In the frontend code, we’d need to modify Core’s block rendering logic to dynamically render a ghost block in the correct position. @youknowriad suggested using the render_block hook for this.
  • In the editor, we could add code that would insert the ghost block after loading the content, thus dirtying the editor state. This would have the desired effect of prompting the user to either save the ghost block, or dismiss it.
• Provide an insert_ghost_block() PHP function to plugin authors that utilizes the above filters and that they can hook into plugin activation.

We'd obviously also need some UI in the editor in order to alert the user to the automatically added block that needs saving or dismissing, but this seems like the lesser problem for now.

Alternatives

• It was suggested to add the ghost blocks at REST API layer (e.g. when requesting a wp_template CPT). This turned out problematic, since it wouldn’t cover the case where a user newly inserts a Navigation block.
• Matías advised against introducing a collection of filters to create the desired behavior, but to rather use Global Settings instead, since this would also allow adding (and dismissing) ghost blocks at theme, site, or user level.

Open questions

• It was brought up that while inserting the block into all parent blocks of a given type might be good enough for some blocks (such as the Login/out block inside of the Navigation block: in a lot of sites, there will be only one Navigation block anyway, likely present in one reusable template part), we might need to go more fine-grained in general, i.e. allow inserting only into parent blocks that are in a certain template or template part. For this, we’d need some sort of selector syntax.
• Riad brought up the question whether a parent block should still show its "placeholder" state, even if it contains ghost blocks only. (He was leaning against, as the user could remove those manually to reset the parent block to its placeholder state.)

(Most of the discussions mentioned above happened at an IRL meeting with @mtias, @youknowriad, @priethor, @poliuk, @luisherranz, @michalczaplinski, @mburridge, @c4rl0sbr4v0, @SantosGuillamot, and @DAreRodz.)

[mtias]

There are many vectors to this challenge. From an extender perspective we want to make it trivial to create blocks that can show up in their semantic places in both the editor and the front-end. Easy to use, easy to discover without user intervention, but also easy to modify. I don't like the term "ghost" to refer to them because in the front-end they are as real as any other block. It's only in the editor where the implications are different, because we don't serialize them, and because neither the theme nor the user has decided on it. In the editor, a user should be able to relocate one of these blocks, so the initial placement is both a suggestion and discovery mechanism for a plugin / block but not an imposition.

My suggestion is not necessarily to use global settings but to use configuration to drive the behaviour, which is expressed in both block.json and theme.json. This could take the shape of an autoInsert: [ 'core/navigation' ] property. We touched upon the limitations of this in terms of narrowing down to specific template areas and so on, which we could extend through syntax like core/template/header/navigation or refine through theme.json. This property would also need to be able to express some basic positioning like before / after / innerFirst / innerLast.

It could also be that autoInsert works upon the first encountered block of the specified type, so it doesn't get repeated if there are multiple navigation blocks in a page. Or maybe this is also a flag like useOnce. Ultimately, it has to be possible for a user to remove the block a plugin adds if they don't like it there, and obviously the block should not return, so we need to discriminate between these two absenses (default state where a template doesn't include a given block and a case where the user doesn't want a block there) from an operative point of view.

Another part of the challenge is that we need to describe a loose contract between templates and blocks (and templates are also blocks, so between blocks and blocks). I say loose because ideally a theme doesn't need to mark these hookable areas with any additional mechanisms because they are implicit in the semantics of the blocks its using.

When it comes to the UX, there are different ways we might want to present auto-inserted blocks to draw a distinction, but that is a slightly separate consideration.

[mtias]

Fleshing out this idea a bit further, I think we could instrument something where every block that has an inner blocks area with allowed-blocks: * would support loading blocks that declare themselves as being for that region.

From the perspective of the child block:

// End and after only work if the specified block has unlocked inner blocks
{
    'load': [ 'core/navigation', 'before' | 'start' | 'end' | 'after' ],
}

The before and after could be implicit positional hooks for all blocks (i.e. if I have a plugin that wants to add a "scroll to top" thing after the site title).

[nerrad]

@senadir (and @ralucaStan) - there's some overlap here with what you and your team has done for extensibility in the WooCommerce Cart and Checkout blocks. There could be some opportunity here to contribute towards a solution in Gutenberg that would be beneficial here?

[mtias]

I had another realization here that I believe simplifies things greatly: the auto-insert behaviour should work with file based templates, not with saved templates. If a user wants to remove the auto-inserted block, they remove it and save; since the template becomes a saved template at that point, the user choice would be honored. If the user wants to restore the block they can insert it or revert the template. The same goes for moving it elsewhere, since we don't need to calculate whether the block is already inserted we just honored what was stored. I think this contemplates most of the possible scenarios pretty elegantly.

The one case we need to consider is when a user has a customized header already, and then install a plugin / block with auto-insert behaviours since those won't kick in. I think this is ok and we should separately work on ways to help surface blocks that may be trying to auto-insert themselves on a saved template by exposing it in the UI somehow (i.e. "there are 5 blocks that could be shown here") and to allow the user to quickly restore or interact with them if they want to.

[nerrad]

The one case we need to consider is when a user has a customized header already, and then install a plugin / block with auto-insert behaviours since those won't kick in. I think this is ok and we should separately work on ways to help surface blocks that may be trying to auto-insert themselves on a saved template by exposing it in the UI somehow (i.e. "there are 5 blocks that could be shown here") and to allow the user to quickly restore or interact with them if they want to.

This is the primary concern I had as well. Once a template is customized and saved, the discovery process for auto-insertion of content any plugins activated after the fact want to do, is a critical piece imo. Especially for templates that may not be visited often by users (i.e. a checkout template for a commerce application). I think we'd need to think through discovery beyond just within the template itself.

[luisherranz]

I was thinking about this as well. I haven't had time to settle on these thoughts yet, but I guess they are worth sharing, just in case.

Global Styles are a way to overwrite block.json properties. If Gutenberg embraces @mtias' section's proposal, sections will have "Global Styles" capabilities and, therefore, the opportunity to overwrite the block.json properties. We could leverage that to save whether an autoInsert is active or not on each section.

Imagine this block.json configuration.

// my-org/my-block block.json
{
  "autoInsert": {
    "core/navigation": {
      "placements": ["after"],
      "templatePartCategories": ["header"],
      "active": true // <-- This is the default, so it doesn't need to be included.
    },
    "core/buttons": {
      "placements": ["after"],
      "templatePartCategories": ["header", "footer"],
      "...": "There may be more filtering options"
    }
  }
}

When an auto inserted block is present, the UI should show a button to remove it. That user intent would be stored in the closest section's Global Styles (Section Styles?), turning active: false inside that block configuration. If each Navigation block is a section (can save Global Styles overwrites), then this solution works even if:

• There are multiple Navigation blocks in the same template.
• There are multiple "header" template parts.

// Closest section (parent navigation block)
{
  "my-org/my-block": {
    "autoInsert": {
      "core/navigation": {
        "active": false // <-- Overwrite block.json.
      }
    }
  }
}

In this example, the user removed the auto-inserted block inside one of the Navigation blocks, but not the ones in other Navigation blocks or templates parts.

As it is unclear what is the user intent when they click the "remove" button, a prompt could be presented to make them chose between:

• Remove it only here (save active: false in the closest parent section).
• If it's inside a template part: Remove it in this template part (save active: false in the template part's Global Styles (Section Styles?)).
• Remove it everywhere (save active: false in Global Styles).

If a user wants to revert these changes, it can do so in the corresponding Global Styles or Section Styles UI. This won't be super intuitive, but at least there will be a place to do so.

This also works with HTML templates: if a theme creator knows about an autoInsert and wants to prevent its insertion, it can add active: false in the correct Section Styles. Imagine a Woo theme that has two menus (MainMenu and SubMenu) but only wants the mini-cart button added automatically to the MainMenu. It can explicitly opt-out using:

// Section Styles of the "SubMenu" Navigation block
{
  "woocommerce/mini-cart": {
    "autoInsert": {
      "core/navigation": {
        "active": false
      }
    }
  }
}

Or they could add active: false to the template part and explicitly opt-in using active: true in the MainMenu navigation, which will ensure that the mini cart won't be auto inserted on other navigation blocks added by the user.

---

Two other unrelated and very unsettled thoughts:

• I'm also wondering how plugins/blocks could enable/disable the auto insertion programmatically if we end up using block.json properties.

  For example, If you want to add a block automatically but only when the user selects a specific option in my plugin. Imagine that the WooCommerce plugin only wants to add a Login/Logout button to the navigation when creating an account is mandatory to make an order.

  I'm not sure if you can modify the block.json properties of a block via filter, but I guess that could be one way to do so.

  EDIT: It seems like this would be possible using block_type_metadata, so maybe it could be something like this:

  add_filter( "block_type_metadata", function ( $metadata ) {
  if ( "my-block/my-org" === $metadata[ "name" ] ) {
    $isAutoInsert = my_plugin_is_autoinsert_active();
    $metadata[ "autoInsert" ][ "core/navigation-block" ][ "active" ] = $isAutoInsert;
  }
  
  return $metadata;
  });

• I'm also wondering how blocks could reposition slots or create new ones.

  @mtias proposed these placements: 'before' | 'start' | 'end' | 'after'

  Imagine the default placement in the navigation block. I guess it would be something like this?

  These placements comments are not serialized, I'm just using them to show their positions in relation to the rest of the block.

  <!-- before wp:navigation (default) -->
  <!-- wp:navigation -->
  <div class="wp-my-nav-block">
  <ul class="wp-my-nav-block__desktop-menu">
    <li>Link 1</li>
    <li>Link 2</li>
  </ul>
  <div class="wp-my-nav-block__mobile-overlay">
    <ul class="wp-my-nav-block__mobile-menu">
      <li>Link 1</li>
      <li>Link 2</li>
      <li>Extra Mobile Link</li>
    </ul>
  </div>
  </div>
  <!-- wp:navigation /-->
  <!-- after wp:navigation (default) -->

  What if the block wants to redefine the placements? Even make it appear in multiple places? Or add new ad-hoc placements?

  <!-- wp:navigation -->
  <div class="wp-my-nav-block">
  <ul class="wp-my-nav-block__desktop-menu">
    <!-- before wp:navigation (custom) -->
    <li>Link 1</li>
    <li>Link 2</li>
    <!-- after wp:navigation (custom) -->
  </ul>
  <div class="wp-my-nav-block__mobile-overlay">
    <ul class="wp-my-nav-block__mobile-menu">
      <!-- before wp:navigation (custom) -->
      <!-- before-mobile wp:navigation (new) -->
      <li>Link 1</li>
      <li>Link 2</li>
      <li>Extra Mobile Link</li>
      <!-- after-mobile wp:navigation (new) -->
      <!-- after wp:navigation (custom) -->
    </ul>
  </div>
  </div>
  <!-- wp:navigation /-->

  My only vague idea would be to use <AutoInsert> components (or whatever this thing is finally called).

  const MyBlock = ({ attributes }) => (
  <div {...blockProps}>
    <AutoInsert name="before" />
    <h1>{attributes.title}</h1>
    <button onClick={open}>Open modal</button>
    <AutoInsert name="after" />
    <div className="my-block__modal">
      <AutoInsert name="before modal">
      ...
      <AutoInsert name="after modal">
    </div>
  </div>
  )

  Maybe this could even replace some of the "Slot" needs for some plugins, but I have barely started thinking about that yet.

---

PS: I'm not sure about the difference between the before/start and after/end placements. @mtias, @ockham: would you mind clarifying?

[mtias]

@luisherranz Suppose we have the following block tree:

block-a
block-b
    block-d
    block-e
block-c

and we are interested in block-b as the anchor point, the options would be start / end:

block-a
block-b
    // -> insert here with "start"
    block-d
    block-e
    // -> insert here with "end"
block-c

And for before / after:

block-a
// -> insert here with "before"
block-b
    block-d
    block-e
// -> insert here with "after"
block-c

The first case requires block-b to have an inner blocks area. The second doesn't.

[luisherranz]

Ohh, got it. Thank you, Matías!

[gziolo]

While we work on the general solution, https://github.com/WordPress/gutenberg/pull/37998 adds a way to modify inner blocks for the Navigation block with a PHP filter. This is going to be included in WordPress 6.1.

[ockham]

Since I'm not sure it's fully captured in any of the above comments, I'll add another use case that @mtias brought back to my mind recently:

A plugin wants to add a "Like" button below the post content block, but only in the single template.

(This means that we'll need some way and syntax to limit auto-inserting blocks to specific templates.)

[gziolo]

@ockham, are there any prior PRs that explored the potential solution for this feature?

I know about #37998, which addressed the same issue for the Navigation block explained in detail in https://github.com/WordPress/gutenberg/issues/37717. However, the approach taken is based on WP hooks that can't be generalized for every possible block and can't be visualized in the editor.

By the way, I plan to work on this feature for some time and build a prototype with some basic functionality to gather more feedback as we learn about the implications of surfacing auto-inserted blocks in the UI.

[ockham]

@ockham, are there any prior PRs that explored the potential solution for this feature?

None that I'm aware of!

I know about #37998, which addressed the same issue for the Navigation block explained in detail in #37717. However, the approach taken is based on WP hooks that can't be generalized for every possible block and can't be visualized in the editor.

Yeah, that's kind of the "old-school" approach, with the downsides that you mention. Not really the long-term/generic solution that we're trying to conceive of in this issue 😊

By the way, I plan to work on this feature for some time and build a prototype with some basic functionality to gather more feedback as we learn about the implications of surfacing auto-inserted blocks in the UI.

Sounds good! LMK if you need any pointers or help 😄

[gziolo]

I opened https://github.com/WordPress/gutenberg/pull/49789 to explore some ideas. At this moment, there isn't much included, but I wanted to share the branch early for visibility.

[gziolo]

I closed #49789 with the following findings documented in https://github.com/WordPress/gutenberg/pull/49789#issuecomment-1516007658:

I recorded a narrated video to share the progress of the exploration. In the first part, I'm talking about the issue and the importance of the comment https://github.com/WordPress/gutenberg/issues/39439#issuecomment-1150278043 from @mtias where he says:

I had another realization here that I believe simplifies things greatly: the auto-insert behaviour should work with file based templates, not with saved templates. If a user wants to remove the auto-inserted block, they remove it and save; since the template becomes a saved template at that point, the user choice would be honored. If the user wants to restore the block they can insert it or revert the template. The same goes for moving it elsewhere, since we don't need to calculate whether the block is already inserted we just honored what was stored. I think this contemplates most of the possible scenarios pretty elegantly.

I also explain the basic idea explored in this branch that led me to apply the current changes to the branch and how I wanted to keep track of the block names that were auto-inserted in the block editor.

https://user-images.githubusercontent.com/699132/233314793-9c6a4906-7fbd-4fcb-88d4-c746c10ceb6b.mov

In the second part, I presented how I use the modified E2E test plugin to verify how the changes applied in this branch impact the list of blocks loaded in the block editor.

https://user-images.githubusercontent.com/699132/233318524-589853f5-5d04-40e6-9c67-19696e71df17.mov

The learning so far is that the approach taken doesn't scale well despite my initial enthusiasm shared in https://github.com/WordPress/gutenberg/pull/49789#issuecomment-1512959490. As briefly discussed with @youknowriad in https://github.com/WordPress/gutenberg/pull/49789#discussion_r1171377177, it might be very difficult to bend the current approach to integrate seamlessly with the undo/redo, but also with other parts of the editing workflow like switching between the code and the visual editors. I'm inclined to try next the idea brought by Riad to change the list of blocks earlier in the flow – during initial block parsing on the client or even run that logic on the server. In particular, auto-inserting blocks on the server is appealing because we need it anyway for site visitors, so maybe we can integrate a similar logic with REST API for post content (that would apply to posts, pages, CPT, reusable blocks, template parts, and templates).

The important note is that since we need to auto-insert blocks on the server if the site admin never opens the block editor, it can't work out of the box with static blocks that depend on the save method. We need to limit the application of this approach to dynamic blocks that can be fully assembled on the server.

[gziolo]

Still to clarify

There are still a few open questions that we need to discuss while we work on prototypes for the auto-inserting mechanism. I summarized below everything I discussed recently with @artemiomorales and @ockham during video calls.

Format for the auto-inserting mechanism

1. Figure out what would be the best format for the auto-inserting mechanism inside the block.json file. We will need more flexible API that being able to define the block name we target, example: "autoInsert": { "after": [ "core/quote" ] }.
2. Decide where the auto-inserting blocks can integrate by default. Is it every type of editor (site, widgets, post) and content (template, template part, reusable block, pattern, post, page, CPT)?
3. How can developers narrow down the places where blocks get auto-inserted? Example: first navigation block in the template, 5 times on the page, etc. Do we want to cover it in the first iteration or leave it for another time?

Possible scenarios to cover

Here is the list of possible scenarios for the cases of how people would interact on the site with auto-inserting blocks. The assumption is that someone has just installed a plugin with auto-inserting blocks on the existing website. Is that list a good baseline for further explorations?

1. Someone never opens the block editor.
   1. Blocks get automatically inserted when serving the site to visitors.
2. Someone opens the block editor for unmodified content, and they don’t apply any changes to the content.
   1. Blocks get automatically inserted when editing the content, but they don’t cause the editor to show the need for saving changes.
   2. Blocks get automatically inserted when serving the site to visitors.
3. Someone opens the block editor for unmodified content, they apply any change to the content, and then saves it.
   1. Blocks get automatically inserted when initially editing the content.
   2. When saving the content, all auto-inserted blocks still present get saved to the database. There also needs to be a way to remember which block types were auto-inserted.
   3. Block types remembered for the content no longer get automatically inserted when serving the site to visitors.
4. Someone opens the block editor for the modified content with some auto-inserted blocks, they apply more changes to the content, and then saves it.
   1. Blocks no longer get automatically inserted when editing the content.
   2. When saving the content, nothing changes for the persisted list of auto-inserted blocks.
   3. Block types remembered for the content still don’t get automatically inserted when serving the site to visitors.
5. Someone installs another plugin with new auto-inserting blocks.
   1. Newly installed blocks that support the mechanism get automatically inserted when serving the site to visitors.
6. Someone installs another plugin with new auto-inserting blocks. They open the block editor for the modified content with some different previously auto-inserted blocks, apply more changes to the content, and then save it.
   1. Only newly installed auto-inserting blocks get automatically inserted when editing the content.
   2. When saving the content, all newly installed auto-inserted blocks still present get saved to the database. There is a way to remember old and new auto-inserted block types.
   3. Old and new block types remembered for the content no longer get automatically inserted when serving the site to visitors.

Considering the assumption that we only support auto-inserting for dynamic blocks, removing the block from the site in all scenarios means they shouldn’t display anymore, or only a static HTML fallback saved to the database would remain visible for site visitors.

Static vs dynamic blocks

We could potentially seek ways to integrate an auto-inserting mechanism with Template Parts, Block Patterns, or Reusable Blocks. There is a concern that we might have 3 different ways to do a similar thing with one API of auto-inserting blocks. We better limit the scope to the blocks for now but keep in mind that it could be expanded to other existing APIs that block themes use to bring more flexibility and overcome the limitation of not having a way to run JavaScript on the server that is required for the save method.

New ideas for integration with the editor

We discussed exploring in the next step alternative ways to modify the HTML for the block editor:

1. We don’t update anything in the editor initially but present a banner informing users that the content is different on the front end because of the auto-inserting blocks. Users could preview it, inject them to edit the content, etc.
2. We auto-insert blocks on the client just after parsing the HTML but before initializing the editor. The user would see a banner informing that the content contains auto-inserted blocks, and they could remove them with a single click.
3. We modify the REST API response by parsing HTML and auto-inserting blocks and then serializing it back to HTML. The rest would work the same way as in Option 2.

The remaining question is whether we should dirty the block editor state for options 2 and 3 when opening an unmodified template, template part, etc.?

[nerrad]

Should there be some sort of visual indicator in the editor to signal which blocks are auto-inserted to the user?

[ockham]

Should there be some sort of visual indicator in the editor to signal which blocks are auto-inserted to the user?

@nerrad Yeah, we'll definitely need some UX design around this. For now, we're focusing on the engineering side, since we think that we first need to solve some technical problems. We will need to keep some UX aspects in mind already (especially e.g. with regard to whether auto-inserted blocks should dirty the editor state or not, or if we need a whole new "contains auto-inserted blocks" state), but most of the design work should probably start only once we've settled on a few basic principles.

[ockham]

Blocks get automatically inserted when serving the site to visitors.

We have a very, very basic prototype in https://github.com/WordPress/gutenberg/pull/50103. All it does is demonstrate automatic insertion of a block right after a given block, and as the last child of another given block, via render_block_data and render_block filters. All these things are currently hardcoded -- the blocks that are being inserted, the block that they're being inserted next to, and the relative position.

As a next step, I'd like to add logic to limit auto-insertion to a given template (and/or maybe template part). I think that this is a logical next step, as we can then build upon this and find a mechanism to determine if the relevant template was modified by the user to persist or dismiss the block -- see https://github.com/WordPress/gutenberg/issues/39439#issuecomment-1150278043.

Information what (block) template is being rendered doesn't seem to be easily available during block rendering, so I met with @gziolo today to discuss a few ideas, among them:

1. keeping a “breadcrumb trail” via the parent block arg passed to the render_block_data filter
2. keeping a dedicated parent field on each block instance
3. or using a template rendering related hook to
   1. set a global with that information
   2. or even conditionally add the auto-insertion related filters

I tried option 1. but it seems like the generated "tree" of parents might not be quite complete (or at least it doesn't seem to go all the way up to the current template).

While I don't love globals, there's some appeal to option 3.1, as it'd allow us to introduce a function to determine the current template (which might come in handy for other use cases as well).

We also discussed if it would be sufficient to provide a set of attributes for an auto-inserting blocks, or if we’d need something like a block pattern to cover a wider range of blocks (including static blocks and inner blocks), and where block authors would need to put those (separate file, referenced from block.json?). Furthermore, this raised the question if it's going to be auto-inserting blocks or patterns that we'll offer (also with regard to how they'll be distributed -- through the block or pattern directory). Anyway, we agreed that for the early stages of this experiment, specifying only attributes for a given auto-inserting block might be sufficient.

[mtias]

@ockham I am not sure about prioritizing a design implementation that accounts for template / template part limiting. It's a bit abstract and very unlikely for a plugin to govern because it won't know ahead of time what template parts may or may not exist. I think we should avoid that entirely.

[gziolo]

@ockham, excellent progress on server-side rendering for Auto-inserting blocks. I really like where it is heading. Some work is still necessary to shape the initial API for block types and potentially block patterns. When it’s ready, it would be great to start an experiment in the Gutenberg plugin to gather feedback from Woo, Jetpack, and other plugins that might benefit from this feature and expressed interest in verifying the prototype. There is also the whole story about the block editor, but it seems like it is optional initially to run early tests.

Figure out what would be the best format for the auto-inserting mechanism inside the block.json file. We will need more flexible API that being able to define the block name we target, example: "autoInsert": { "after": [ "core/quote" ] }.

My current thinking is that we should start simply by limiting the autoInsert configuration to placement (after, before, first-child, and last-child as in #50103) and the name of the block type(s). We can extend it later if necessary by allowing non-string values, example core/quote becomes { "name": "core/quote", "attributes": { "foo": "bar" } }. In practice, though, it would also be interesting to extend the application of the API to block patterns, to make it possible to cover custom block attributes. inner HTML, and even inner blocks like in the example in #50103 when integrating Social Icons blocks with:

<!-- wp:social-links -->
<ul class="wp-block-social-links">
    <!-- wp:social-link {"url":"https://wordpress.org","service":"wordpress"} /-->
</ul>
<!-- /wp:social-links -->

How can developers narrow down the places where blocks get auto-inserted? Example: first navigation block in the template, 5 times on the page, etc. Do we want to cover it in the first iteration or leave it for another time?

One aspect to think about is that block types already have an existing mechanism for controlling where the block can be inserted with parent, ancestor and Block Supports' option multiple (offers a way to insert a block only once, but we could tweak it for a maximum number, too). We can also further extend options for how blocks integrate with the inserter and use that as a control mechanism with Auto-inserting blocks.

[ockham]

I am not sure about prioritizing a design implementation that accounts for template / template part limiting. It's a bit abstract and very unlikely for a plugin to govern because it won't know ahead of time what template parts may or may not exist. I think we should avoid that entirely.

Alright, I can focus on something else first. (Thinking to detect whether the current template has user modifications or is file-based to determine whether or not to auto-insert blocks, per your comment.)

@mtias Just to clarify, I thought that one use case we had was something like "Auto-insert a given block below the Post Content block, but only when on a page that uses the Single Post template to render." Did I get that wrong, or would you rather express that "location" differently? 🤔

I'll add that while it's true that we can't always know what template parts are available, we could still limit auto-inserting blocks to a given template, since it's guaranteed that there will always be one template used to render a given page. (We'd likely need to apply template resolution to that specified destination template.)

[mtias]

but only when on a page that uses the Single Post template to render

I think this doesn't necessarily need to be expressed by the block API itself and can be handled by a plugin at the time of registering the block or through conditionally filtering the attribute.

[ockham]

Update:

The experimental PR (https://github.com/WordPress/gutenberg/pull/50103) now supports auto-inserting blocks on the frontend based on a block.json autoInsert field.

I've become increasingly skeptical that auto-inserting a block is sufficient and laid out my arguments here. As a consquence, I've filed an alternative PR to auto-insert block patterns instead: https://github.com/WordPress/gutenberg/pull/51294

Finally, I started exploring auto-insertion in the editor, via the REST API: https://github.com/WordPress/gutenberg/pull/51449. I'm quite happy with the state of this latter PR, as it now supports insertion in the editor and on the frontend, via the same underlying mechanism. That PR's description has more details on the underlying technical intricacies.

[simison]

I've become increasingly skeptical that auto-inserting a block is sufficient and laid out my arguments https://github.com/WordPress/gutenberg/pull/50103#issuecomment-1579060416. As a consequence, I've filed an alternative PR to auto-insert block patterns instead: https://github.com/WordPress/gutenberg/pull/51294

Auto inserting template parts might be another good way to look at these APIs (pardon if this was already discussed). Use cases I have in mind are injecting a template part containing a "subscribe to this blog" modal or a GDPR cookie banner.

Since the modal should appear on all posts, or cookie banner on all pages and posts, it's better to keep them in template parts injected to other templates. Template part is handy for linking and allowing customer edit the modal/cookie banner in focused template rather than as part of the whole page's template.

[youknowriad]

Auto-inserting patterns/template parts is interesting. How do you detect that the particular "pattern" has been already edited and is present in the content to avoid inserting it twice?

[joshuatf]

Just throwing in my 👍 for this feature. The new WooCommerce product editor uses a template and we've explored various APIs to allow this template to be extended by third party plugins. This feature would eliminate the need for our own custom API.

Some of our earlier discussions went with a similar pattern of inserting before or after specific blocks, but we found this to be somewhat problematic for a couple of reasons:

1. The plugin needs to know about a specific field and know that it will be present in order for their field to be added. This may not be the case in all of our templates, whereas in our case "sections" (parent blocks with inner blocks) we can trust to be present. However, the current API seems to only allow inserting at the start or end of a parent block.
2. If multiple plugins were to add fields after the same existing field, the order they are added to the template is presumably up to how they are added at runtime.

To better demonstrate this, we might have this template:

block-a (section)
  block-b (field)
  block-c (field)
  block-d (field)
block-e (section)

If we wanted to insert a block (field) from a plugin between c and d, we would need to insert after c or before d using the current API.

block-a
  block-b
  block-c
  block-f (plugin field)
  block-d
block-e

If another plugin adds a field after c, I'm speculating that the result would look like this if the code is executed after the first:

block-a
  block-b
  block-c
  block-g (2nd added plugin field)
  block-f (1st added plugin field)
  block-d
block-e

In our POC we opted for a order or priority to create a more declarative API to insert in the correct position without worrying about async behavior or order of execution used on the server-side. This also means that the plugin only needs to know about the parent block existence and not the child in the event that block-c from above was removed in some of the templates.

Curious to know your thoughts on this @mtias @ockham.

[mtias]

@ockham I replied on the PR regarding the patterns vs block distinction. I think the examples used are problematic in that they are not common nor natural. Patterns are problematic if they are instance based (like themes do with wp:pattern) given they can become a static instance if the user interacts with them, which means there's no way to properly anticipate duplication unless you always reference a pattern wrapper (i.e. an actual core/block or core/template-part block).

I'd stay with the block-first API. Including the ability to provide different sets of default attributes is interesting, though I think too preemptive. We should see if there's an actual demand for it from real use cases.

[ockham]

@joshuatf Thank you for getting in touch, and really appreciate the feedback! First off, it's exactly use cases like yours that we'd like to cover with this -- I'd go as far as to say that if we don't manage to make it fit for your needs, we would have kinda failed our task 😅 I'd thus be more than happy to collab on this to make sure that it will be useful for y'all!

The plugin needs to know about a specific field and know that it will be present in order for their field to be added. This may not be the case in all of our templates, whereas in our case "sections" (parent blocks with inner blocks) we can trust to be present. However, the current API seems to only allow inserting at the start or end of a parent block.

That's correct. Can you give me an example how the kind of insertion y'all need goes beyond first/last child insertion? Is this what you describe in your example (inserting before or after a given block that's also a child block of another given block, i.e. defining the inserting position by means of two "anchor" blocks rather than just one)?

If multiple plugins were to add fields after the same existing field, the order they are added to the template is presumably up to how they are added at runtime. [...] In our POC we opted for a order or priority to create a more declarative API to insert in the correct position without worrying about async behavior or order of execution used on the server-side.

I know that @mtias doesn't like priority args for APIs like this (and Gutenberg has so far avoided them e.g. in client-side filters -- as opposed to WordPress' server-side ones). I guess this approach implies the line of thought that anything that uses a filter -- or, in our case, an insertion position -- needs to be okay with not controlling that other blocks might render before it.

At first glance, if I imagine e.g. a customizable form block (like an address?) with a number of different fields, that seems okay to me: If two different plugins compete for a spot, say, before the "Country" field, neither of them will be guaranteed which one will render before the other, so they will have to make provisions for either case 🤔 Can you give me a concrete example where the order matters?

This also means that the plugin only needs to know about the parent block existence and not the child in the event that block-c from above was removed in some of the templates.

FWIW, the way we're covering this is that blocks are only auto-inserted into unmodified templates (see) -- the moment a template is modified and saved by the user, we stop auto-inserting. This allows us to respect e.g. the user manually removing the auto-inserted block (instead of continuing to auto-insert it), or to avoid auto-inserting it after the user has already saved it in its suggested position, both of which would otherwise be a considerably harder problems.

---

I'd be curious to test the code in #51449 with some practical examples that you're working on or aware of @joshuatf -- they could serve as a benchmark for the viability of the present approach, and to discover what features are missing! I'd love to land #51449 (or a variant of it) as experimental in Gutenberg fairly soon; but I'd also be happy to first file a PR against any repo you're working on to try it out with those blocks 😄 Maybe you can point me to some candidate repos and/or blocks?

[joshuatf]

I'd go as far as to say that if we don't manage to make it fit for your needs, we would have kinda failed our task 😅 I'd thus be more than happy to collab on this to make sure that it will be useful for y'all!

Really appreciate this, @ockham! Happy to share some of our use cases so we can work towards finding something that works (or invalidate some of the assumptions made in Woo around this if they're wrong).

That's correct. Can you give me an example how the kind of insertion y'all need goes beyond first/last child insertion? Is this what you describe in your example (inserting before or after a given block that's also a child block of another given block, i.e. defining the inserting position by means of two "anchor" blocks rather than just one)?

Sure! It's not necessarily that we need two anchor points and in fact it's probably more akin to not having any anchor points at all. I think the below question and example illustrate this.

Can you give me a concrete example where the order matters?

If we take a look at the WooCommerce Brands plugin as an example, this plugin adds a brand taxonomy selection which should get added to all product types. Let's say that this gets added directly after the pricing fields in the "General" tab.

Then another plugin, like Tiered and Dynamic Pricing wants to add a field at the same spot. Ideally, the pricing fields are adjacent and the brands taxonomy comes after. But since we can't guarantee the order, the order may be Regular Price | List Price | Brands Taxonomy | Dynamic Pricing

The brands plugin chose to insert next to that field because of its order and not necessarily relevance. This problem is further compounded if we have some product types or custom product types that remove the pricing fields and we no longer have an anchor point to attach to.

I know that @mtias doesn't like priority args for APIs like this (and Gutenberg has so far avoided them e.g. in client-side filters -- as opposed to WordPress' server-side ones).

We had also tried to avoid this, but did not find a pattern that would allow us the flexibility needed. @mtias could you expand a bit on your primary reasons for avoiding the use of an order or priority argument? That will help guide us and maybe invalidate our assumptions around why we need one.

FWIW, the way we're covering this is that blocks are only auto-inserted into unmodified templates (https://github.com/WordPress/gutenberg/issues/39439#issuecomment-1150278043) -- the moment a template is modified and saved by the user, we stop auto-inserting.

That makes sense, thanks for the explanation! The case in the product editor is interesting because at least with its current and foreseeable future, we do not plan to allow merchants to make any modifications to the templates. However, areas in WooCommerce Blocks checkout would benefit greatly from this as I know there have already been workarounds made to achieve similar behavior.

I'd be curious to test the code in https://github.com/WordPress/gutenberg/pull/51449 with some practical examples that you're working on or aware of @joshuatf -- they could serve as a benchmark for the viability of the present approach, and to discover what features are missing!

Will give this a spin this week!

[mtias]

The brands plugin chose to insert next to that field because of its order and not necessarily relevance.

I think this is the problem. The example showcases why we shouldn't be relying on priorities for ordering. Relevance should be expressed semantically, so that extending "price" always makes sense and doesn't need to account for other extensions itself.

When you need to provide utmost flexibility, your should look at the entire template that's laying these blocks out and allow changing that at a higher level.

[joshuatf]

Relevance should be expressed semantically, so that extending "price" always makes sense and doesn't need to account for other extensions itself.

Agreed with the sentiment here, @mtias. I think in practice there are many cases where the highest relevance is the parent block, but the extending plugin wants to insert it at some position other than first or last.

When you need to provide utmost flexibility, your should look at the entire template that's laying these blocks out and allow changing that at a higher level.

This is definitely the case for the product editor as we need more flexibility and I don't think this auto insertion API provides the level of flexibility needed for those types of templates.

I also understand that user-edited templates may make the need for an order property a little less important since the user has the option to re-order blocks after it has been auto-inserted. However, I can certainly see cases where 3PDs anchor to a sibling element to place their element in what they deem the expected place (order) for it to render only to have it not show up because the user has previously removed the sibling anchor block.

[mtias]

Yeah, this specific API is being designed as a low footprint way of bringing a block into an editor context automatically while allowing users to remove or relocate as needed. It's not really about composing a list of several blocks, organized in specific ways, which is a higher level abstraction.

[ockham]

Auto inserting template parts might be another good way to look at these APIs (pardon if this was already discussed). Use cases I have in mind are injecting a template part containing a "subscribe to this blog" modal or a GDPR cookie banner.

Since the modal should appear on all posts, or cookie banner on all pages and posts, it's better to keep them in template parts injected to other templates. Template part is handy for linking and allowing customer edit the modal/cookie banner in focused template rather than as part of the whole page's template.

Apologies for not getting back to you earlier, @simison. The mechanism currently explored in #51449 applies the same technique to templates and template parts: Blocks are auto-inserted into the response from the /templates REST API endpoint, as long as the relevant template (or template part, respectively) doesn't have any user modifications. In the walkthrough video that I posted to that PR, it's actually the TT3 theme's "Comments" template that the block is being auto-inserted into; this seems to be working well enough. If you get a chance, it'd be great if you could give it a try and see if it fits your need 😊

[ockham]

Auto-inserting patterns/template parts is interesting. How do you detect that the particular "pattern" has been already edited and is present in the content to avoid inserting it twice?

Sorry for the delay in my reply, @youknowriad. Per @mtias' comment, we're only auto-inserting into templates, template parts, and patterns that don't have any user modifications (i.e. that come straight from a theme or plugin-supplied template file). This solves a lot of problems around the complexities we'd otherwise face in detecting user modifications, even though it incurs some drawbacks (that we've considered acceptable though):

The one case we need to consider is when a user has a customized header already, and then install a plugin / block with auto-insert behaviours since those won't kick in. I think this is ok and we should separately work on ways to help surface blocks that may be trying to auto-insert themselves on a saved template by exposing it in the UI somehow (i.e. "there are 5 blocks that could be shown here") and to allow the user to quickly restore or interact with them if they want to.

[ockham]

A little update for the folks following along at home:

I've added a little demo video to the PR and opened it for review since I've felt it's in good enough shape for that. For now, this means that there's:

• No ability to specify attributes or inner blocks for auto-inserted blocks (block slug only; it's the block's job to provide defaults that work sensibly for auto-insertion).
• No visual highlighting of auto-inserted blocks in the editor (we'll likely add that later).
• No UX alerting users that a plugin would auto-insert a block into the current template (or template part) if it didn't have any user modifications already. (Also likely to be added later.)

Please give that PR a try and leave your feedback on it!

[joshuatf]

@ockham Out of curiousity, will this auto-insertion API ever be usable with the comma delimited template format?

array(
            array( 'core/image', array(
                'align' => 'left',
            ) ),
            array( 'core/heading', array(
                'placeholder' => 'Add Author...',
            ) ),
            array( 'core/paragraph', array(
                'placeholder' => 'Add Description...',
            ) ),
        )

I know the templates REST API is probably expecting the comment delimited format (e.g., <!-- wp:image ...) and the auto-insertion filters before the REST API response, but I'm not seeing a clear way to parse the above comma separated template into the comment delimited version. But that could be a misunderstanding on my part of when each template format should be used.

[ockham]

Update: I've merged https://github.com/WordPress/gutenberg/pull/51449, which means that auto-inserting blocks should become available as a Gutenberg Experiment in GB 16.4.

As for next steps, I'm thinking of the following:

• [ ] #52969
• [ ] Check if auto-insertion works with synced patterns
• [ ] Add mechanism to limit to certain templates/post types, e.g. via a filter (we continue to get requests for this kind of feature, see e.g. https://github.com/WordPress/gutenberg/pull/51449#pullrequestreview-1529686353 and https://github.com/WordPress/gutenberg/pull/51449#issuecomment-1648795994)
• [ ] Add test coverage. The way we have to implement this is quite heterogenous -- e.g. auto-insertion into template parts didn't work for a while. It'd be good to cover a matrix of possible scenarios:
  • Frontend/REST API (editor)
  • Template/template part/pattern
  • Unmodified/modified

Curious to hear from @mtias if you'd like to add other items to the above list, and/or prioritize differently 🙂

[ockham]

@ockham Out of curiousity, will this auto-insertion API ever be usable with the comma delimited template format?

@joshuatf Apologies for the late reply, I missed your comment.

TBH that different format hasn't really been on my radar. Reading the reference you linked to, it seems to be used for a given page's (or, more generally, post type object's) template argument? It seems like some kind of function should exist to transform one format into the other 🤔 Can you provide some context what you'd need that format for?

[joshuatf]

Reading the reference you linked to, it seems to be used for a given page's (or, more generally, post type object's) template argument?

There appear to be 3 different template formats and unfortunately none of them are very well documented or compared. This particular template type is very useful for not only post type templates, but also for crafting inner block templates.

It seems like some kind of function should exist to transform one format into the other

I thought the same, but I didn't have any luck finding utils to transform between the comma delimited type and the other two. My best guess as to how these are used is that the parsed associated array version (with innerBlocks, innerContent, attrs, blockName properties) and the comment delimited version (<!-- wp:group ... -->) are more representative of instances of blocks, while the comma delimited type is more "template-like" in nature and represents a structure for how blocks should be instantiated.

On a side note, I did attempt to write some utils that handle parsing between these formats that works okay structurally, but misses innerContent or the HTML that should fall between the tags in the comment delimited version in blocks that contain inner content. I believe this is pulled from the save method and may be challenging to replicate in utils on the server. Without this, block validation will fail in the client.

Can you provide some context what you'd need that format for?

Sure! We use template types on the post object for the new product editor that is block-based. Each product type contains its own template defining how the editor should be laid out.

WooCommerce Blocks also uses this format quite often to craft inner blocks for many of its checkout blocks. https://github.com/woocommerce/woocommerce-blocks/blob/f8ce88888bd7bc80629c0849cdd5b22b73d843b2/assets/js/blocks/cart/inner-blocks/cart-totals-block/edit.tsx#L20-L25

Also worth noting that WooCommerce Blocks contains its own useForcedLayout method to try and handle inserting newly added blocks when the layout has not been modified. I think the auto-insertion API may be a good candidate for those areas as well instead of spinning up adhoc solutions in each of these repos.

[ockham]

Update:

Development continues in https://github.com/WordPress/gutenberg/pull/52969. Here's the current state, which allows some basic block insertion via enabling a toggle:

More work is needed to reliably insert and remove a block, and to have the toggle accurately represent if an auto-inserted block is present or not; there's some related discussion starting at this comment. This will very likely require the introduction of a global auto-inserted block attribute.

[ockham]

Update: I've now opened https://github.com/WordPress/gutenberg/pull/52969 for review. While it isn't a feature-complete implementation of the toggles yet, it implements most of the desired behavior as described by @mtias here. I think it makes sense to have the PR reviewed (and hopefully merged) already, as it's a good starting point; missing features can be implemented iteratively in follow-up PRs.

The PR description contains a short demo video I made, and a TODO list for those follow-ups.

[johnstonphilip]

This would allow a plugin to auto insert a block, like a filter hook, but can it also remove a block when the plugin is deactivated, like a filter?

[ockham]

This would allow a plugin to auto insert a block, like a filter hook, but can it also remove a block when the plugin is deactivated, like a filter?

Yes, that's kind of how it works -- at least if the containing template or template part has no user modifications.

[ockham]

Closing in favor of https://github.com/WordPress/gutenberg/issues/53987.