Create an API for defining blocks in the editor.

[BE-Webdesign]

This can serve as a discussion for creating an API for the editor. Share your thoughts :smile:

[BE-Webdesign]

Here are things I think the API should do/have:

• Be extensible
• Be easy to use
• Code should be easy to read and understand.
• Have great documentation/tutorials
• Provide a way to register types of blocks
• Create a separation between the blocks and their actual presentation. ( Blocks should be plain ole JavaScript objects that could then render themselves into what is needed. Most likely TinyMCE and DOM nodes. )

Addendum ( Block Types should most likely be registered server side and exposed to the client. )

[aduth]

Related controls discussion: #16

Taking a few of the current mockups as a basis, I've identified some/all of the pieces a developer should expect to be able to implement:

• Label
  • To be shown in insertion dialog
• Icon
• Block controls
• Inline controls (optional)
  • In the case of the quote mockup, one should expect to be able to format text. Is this merely supporting nested plain text block?
  • It's possible that some blocks may want to support only a subset of formatting options
• Children
  • Given sub-note of inline controls, perhaps blocks could allow nested block descendants
  • Future support of a "Columns" block?
• Custom editor-specific form
  • In the case of the quote mockup, the quote and citation may be separate fields
• Transform support
  • Mockup
  • e.g. change plain text to heading or quote
• Advanced settings popover ("Inspector")
  • Shown in early mockups for image settings, assumed to be the gear icon in the current image mockup

Below are a few of my early prototypes which include rough implementations of some or all of these requirements:

• https://github.com/aduth/wp-block#usage
• https://github.com/WordPress/gutenberg/blob/2a5a9f2/tinymce-per-block/src/blocks/text-block/index.js#L17-L53

Given discussion at https://github.com/aduth/wp-block/issues/1, I've also been wondering if we could optionally / alternatively support server-side block registration options, with the following goals in mind:

• Sanitize block attribute values based on defined types
• Scaffold UIs based on defined block attribute types
  • These would admittedly not be as great a user experience, but seems to be an easy way for developers to make quick early iterations of a block, or for freelancers/agencies who can't necessarily justify creating extremely polished block UIs
• Allow blocks to be exposed through REST API or elsewhere in PHP, as current proposals expect registration to occur within browser client

We could look to Shortcake and JS Widgets for inspiration on implementing / consolidating server-side registration if it's a path we want to pursue (cc @westonruter @mattheu @danielbachhuber )

• https://github.com/xwp/wp-js-widgets/blob/f261e9f/core-adapter-widgets/text/class.php#L97-L114
• https://github.com/xwp/wp-js-widgets/blob/fd8c3c4/php/widgets/class-wp-js-widget-text.php#L66-L130
• https://github.com/wp-shortcake/Shortcake/blob/master/dev.php

Less obvious, but if we decide to add a new syntax for blocks, we'll also want to consider how to add backwards compatibility support for e.g. captioned images (currently shortcode) which should either be supported by or transparently upgraded to the new recommended block. This might need to be a separate issue.

[aduth]

Getting into a little more detail, for both block and inline controls, we'll have the following requirements (some assuming we follow proposed block comment syntax):

• Label
  • Used as tooltip / title
• Icon
  • Simplest would be a string reference to icon name of either the Dashicon or Gridicon (whatever we settle on)
  • Optionally supporting image URL, similar to register_post_type's menu_icon
  • Perhaps too complex for the sake of optimization, but in my latest prototype I'd made this a reference to a virtual DOM component primarily to avoid over-including unused icons
• Is selected?
  • Could either be associated with some attribute of a block, or simply passed a function and expected to return a truthy value (arguments include block attributes, children, ...)
• Click/tap behavior
  • For inline controls, it's likely we'd need to execute some command on the TinyMCE instance (regardless of whether we take approach of single or multiple instances)
  • For block level controls, this may either change content of the block or affect its attributes

[westonruter]

We could look to Shortcake and JS Widgets for inspiration on implementing / consolidating server-side registration if it's a path we want to pursue

Just to note here that JS Widgets 0.4 was recently released and it includes full integration with Shortcake so that all widgets defined as “JS widgets” are automatically also available as Shortcake post elements (content blocks). Likewise it introduces icons for widgets (current Dashicons).

The server-side registration ensures that that the widgets (blocks) can be exposed in the REST API and that the widget instance data is guaranteed to be JSON-representable so that the data can be manipulated directly with JS (see WP-CORE#35574, WP-CORE#33507). The widget also then defines the interface for manipulating the data, sanitizing the data, and rendering the data with PHP. The shortcode on the other hand only addresses the last piece, the rendering of the data. The Shortcake plugin was developed to implement the shortcode UI which essentially is a JS-driven version of what widgets historically handled with PHP (WP_Widget::form() and WP_Widget::update()), and thus JS Widgets is able to provide the plumbing for how Shortcake manages a widget shortcode's state.

[jasmussen]

In the case of the quote mockup, one should expect to be able to format text. Is this merely supporting nested plain text block?

We should probably not tackle nested blocks for the editor focus. This will absolutely become necessary for the customization focus, but probably best as something to keep in mind but not work on.

Incidentally you pointed out an error in the mockups. Yes, you should be able to format quotes. The omission of controls was in error. Here's an updated mockup:

It's possible that some blocks may want to support only a subset of formatting options

Absolutely. Headings probably shouldn't have any formatting at all. I'm sure there will be other blocks with unique requirements.

[Lewiscowles1986]

Are there plans for what to do with oEmbed content like if someone posts a link to a YT in a paragraph?

I'd think it's safer to

• simply split the paragraph into two paragraphs with an oEmbed in-between if oEmbed link not at start or very end of content
• simply end the paragraph before the oEmbed if there is no text after the oEmbed link
• prepend the oEmbed if it's at the start of a paragraph

[jasmussen]

Are there plans for what to do with oEmbed content like if someone posts a link to a YT in a paragraph?

Some discussion here: https://github.com/WordPress/gutenberg/issues/81#issuecomment-281371015

See also https://github.com/WordPress/gutenberg/issues/24#issuecomment-281971314

[aduth]

Are there plans for what to do with oEmbed content like if someone posts a link to a YT in a paragraph?

I was going to mention that in complementing discussion in #81, it would be nice to support blocks registering as being associated with a particular oEmbed URL pattern, such as to allow a YouTube block to enable the user to configure features like start time and autoplay. However, in looking at a few examples, it's not commonly supported to customize the embedded URL to affect these changes. In YouTube's case, you'd have to modify the markup of the embedded frame returned, or for Vimeo passing as arguments to the oEmbed request itself. Which is to say, neither of these services would allow you to control their embed behavior by the URL we'd expect to be saved in post content.

[aduth]

Stepping back for a moment, I thought it could be useful to outline some requirements of a block API for both a block implementer (plugin author) and the editor itself, as well as some ideas I've been exploring as conclusions of these requirements.

Requirements

Plugin Author:

At it's most basic, the idea of a user inserting a block is...

• Inserting some markup into post content
• In all but a few cases, the markup will need some customization (think "state" of a block)
• Some overlap exists, but ideally an editor representation of a block could be distinct from what's shown on the front end of a site (think empty states for an embed, placeholders for cite/caption)

There are a few more nuanced requirements or desires, a few highlighted here:

• If we need to define complex behaviors for editable fields, it would be good for these to be generalized in such a way for reuse (e.g. allowing only certain inline markup in image caption)
• I may need to define some default styles for my block's appearance on the front-end in addition to within the editor
• As a block implementer, I only need to be concerned with the logic of my own block, never with the editor context or sibling blocks

Editor:

Ideally, the editor...

• Can render a block at any point in time given its known state (for the editor or for post content) without a round-trip to the server
• Can make available the presence and ideally state and structure of a block from PHP / REST API (so as to enable third party clients including mobile apps to make sense of them)

Conclusions / Ideas:

Block Demarcation Syntax

There have been some explorations of a block demarcation syntax, such as that in the "Coda" section of the original technical overview post. Here it was proposed to use comments to mark the bounds of a block:

<!-- wp:image -->
<figure>
  <img src="/">
  <figcaption>A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

The post grammar explorations took this further to consider encoding some attributes within the comment syntax itself:

<!-- wp:image src:"/" caption:"A picture is worth a thousand words" -->
<figure>
  <img src="/">
  <figcaption>A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

The idea of serializing a block's state entirely within the comment is nice because it can be easily parsed and exposed, even server-side and made available through the REST API. It also enables a block to be more easily rendered because the block's rendering logic becomes a function of its current state (if you're familiar with React paradigms, much like the idea of "components" and "props"). It lessens worries of markup becoming becoming out of sync because we treat the encoded state as the ultimate source of truth, we know what to expect and, if we find that the markup differs, we can provide fallbacks. Finally, it avoids the plugin author needing to manually determine their own block's state by parsing the DOM or traversing an AST.

The most obvious downside is the amount of duplication that occurs between the attributes and the markup itself. I think there are a few options:

• Bite the bullet and accept the duplication (see also: post revisions)
• Encode only a few attributes to be used as hints (this sacrifices most of the benefits)
• Store state elsewhere outside of the markup itself, but keep block demarcators and create a reference of a block's markup to its associated state

To this last point, I think an idea that @danielbachhuber had considered for secret data (https://github.com/aduth/wp-block/issues/2#issuecomment-277096025) could be useful for all cases. If the state of a block is only truly necessary in its raw form in the context of the editor and from the REST API, it seems reasonable to store this elsewhere as post meta. So the markup becomes:

<!-- wp:image 841 -->
<figure>
  <img src="/">
  <figcaption>A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

Where 841 is a meta_id of wp_postmeta whose meta_value is the serialized form of block state.

Edit: Maybe this becomes simpler if the type itself is moved to post meta, where comments are generalized e.g. <!-- wp:block 228 -->

Block Schemas

While the above syntax can enable us to more easily make the state of a block available to other clients such as mobile apps, it's still not sufficient to enable them to present any sort of editable form. To support this and inspired by previous discussion in https://github.com/aduth/wp-block/issues/1, there may be value in allowing the minimal representation of a block to be defined as a schema server-side.

register_block( 'image', [
    'schema' => [
        'type' => 'object',
        'properties' => [
            'src'     => [ 'type' => 'string' ],
            'caption' => [ 'type' => 'string' ]
        ]
    ]
] );

The benefits here would be:

• Consistency with REST API controller schemas
• Combined with above syntax explorations, enables access to both state and structure of a block outside of the browser context
• Provides an approachable interface for creating block prototypes, which can be progressively enhanced in the context of the editor, i.e. complementary JavaScript APIs to enhance block editor experience to achieve desired UX

Componentization of Complex Behaviors

While it's yet to be decided how far we take the idea of React's "UI a function of state", it leads naturally to componentizing not only blocks, but common behaviors between blocks. In the current multi-instance prototype, @youknowriad has applied this as far as toolbars being the responsibility of a block to render, which is not as difficult as it seems when masked as a simple ToolbarControls component.

To my own earlier example, if one needed to control complex editable behaviors such as allowable markup, these could act as arguments (props) to a shared editable interface.

Thinking more concretely and permitting the exploration of React's JSX syntax, a blocks editor render logic could amount to roughly:

function ImageBlock( state, onChange ) {
    return (
        <figure>
            <Toolbar />
            <Media
                src={ state.src }
                onChange={ ( src ) => onChange( { src } ) } />
            <Editable
                inline
                allowedTags={ [ 'strong', 'em', 'a' ] }
                value={ state.caption }
                onChange=( ( caption ) => onChange( { caption } ) ) />
        </figure>
    );
}

[youknowriad]

Thanks @aduth This is really a great summary of the block API needs.

As a block implementer, I only need to be concerned with the logic of my own block, never with the editor context or sibling blocks

I just have some concerns regarding this. I agree it's ideal if we could limit the block API to the current block itself but I think it's not always possible. How do you think switching block types (we have this on the current prototypes) could work without knowledge about the block types to be switched to/from? Do you imagine a block abstraction like block category or something?

The same concerns could apply while "merging" blocks, merging happens if we hit backspace at the beginning of a block.

[Lewiscowles1986]

@youknowriad AFAIK the oEmbed has similar problems currently. Surely that is something for another issue? Perhaps one referencing this one.

[joyously]

What happens when the raw editor is used to modify the HTML? In the example of an image that has a postmeta reference containing the src, it will no longer match if I type over the HTML src tag to show a different image. Is there a way to recover from that sort of thing? Or what about typing over the figure tag to make it a paragraph? Or mangling the quotes or brackets? How will the block editor fail gracefully in these situations?

testreminder

[ellatrix]

@youknowriad makes a good point. There needs to be logic somewhere to transform a paragraph (or a collection of paragraphs and lines with line breaks) into a list and back. Atm the single prototype handles this by passing the node, and the block handles the transformation to and from this "base" state.

Similarly a block such as a block quote or a custom section block will need to be able to wrap a (collection of) blocks and unwrap them again.

[aduth]

I agree it's ideal if we could limit the block API to the current block itself but I think it's not always possible. How do you think switching block types (we have this on the current prototypes) could work without knowledge about the block types to be switched to/from?

@youknowriad This is good to raise, and is indeed an inevitable interaction between blocks. With the ideal in mind, we could at least position this such that a block defines what it can be initialized from (as an explicit opt-in to the behavior) rather than what it can be transformed to.

What happens when the raw editor is used to modify the HTML?

@joyously Yes, this could be a problem if we move state of a block out of the markup, because while we've significantly improved developer experience of block implementer by removing the concern of parsing state, the implications are that edits can only be performed on the state data directly, never on the resulting markup. These scenarios are easy enough to detect, if we find that the markup differs from what we'd expect given the current state. In these cases we may be forced to downgrade to a more generic block, i.e. editing src manually would lose distinction of it being an image block and become something like a "raw HTML" block.

[sirbrillig]

i.e. editing src manually would lose distinction of it being an image block and become something like a "raw HTML" block.

Another possible UX could be that the block detects that its markup is different from what it would generate and asks the user what to do (it doesn't have to be able to parse the markup, it can just diff it with the current output). Something like, "The HTML of this block has been manually modified. Do you want to convert this into an HTML block or restore the last version of this block before it was modified?"

[jasmussen]

Something like, "The HTML of this block has been manually modified. Do you want to convert this into an HTML block or restore the last version of this block before it was modified?"

This is pretty much exactly what Matt proposed very early on. He called it an I know what I'm doing block:

so we parsed a post, found some sections we know exactly what to do with, and some that we have no idea [...] essentially an "I know what I'm doing" block/section :slightly_smiling_face:

[westonruter]

Yes, this could be a problem if we move state of a block out of the markup, because while we've significantly improved developer experience of block implementer by removing the concern of parsing state, the implications are that edits can only be performed on the state data directly, never on the resulting markup.

@aduth another problem with storing the block state in postmeta is that the state is then no longer tracked in post revisions with the content. Maybe this would be solved when revisions support is added for postmeta (#20564), or maybe also there would to be potential for the content and the corresponding postmeta to get out of sync.

[aduth]

Ah, thanks so much for raising that @westonruter , that's a very good point and could have bitten us hard later down the road 😬 Thanks also for the link to the ticket. Reading through and perusing the latest plugin code, it appears to be a good fit for what we'd need in tracking meta with each revision (cc @adamsilverstein as owner).

At worst and while certainly not ideal, storing state in its serialized form in the comment itself doesn't seem out of the question. Doing so would sacrifice some of the benefits, namely: sending more to the front-end than necessary and losing ability to store secrets.

[azaozz]

Few problems with this approach:

1. Removes post_content as "the source of truth". Basically the HTML in post_content becomes non-editable and something like pre-rendered variant of the user input.
2. Because of the above, the HTML editor becomes useless/pointless. Even a small variation in the HTML (for example adding an attribute to a tag) will invalidate it and make the block non-editable in the "visual" editor. This would be a pretty dismal user experience. Thinking about admins or editors trying to do their job after a contributor or an author creates new post, etc.
3. Storing the "block state" in post_meta sounds like a good idea. However imagine a post with ~50 blocks (not uncommon) having 20-30 revisions (also not uncommon). To save such post we will need ~1500 rows in the DB! The post_meta table will literally "explode" :)
4. What would be the "block state" when the user pastes big chunk of HTML, for example: a 500 cell table from Excel? Do we duplicate that whole table and save it separately?

The idea of serializing a block's state entirely within the comment is nice because it can be easily parsed and exposed, even server-side and made available through the REST API. It also enables a block to be more easily rendered because the block's rendering logic becomes a function of its current state...

As far as I see most blocks in a post will be (big) chunks of HTML. They will only have one state (i.e. will be stateless) and have no settings. As HTML they are "ready to go" and will not need parsing or rendering. The "block state" would be useless for them.

On the other hand this is a good idea for "dynamic blocks". In the current editor the "state" of dynamic blocks (a.k.a. wpviews) is the text string used to create them (usually a shortcode or URL). This works pretty well in the HTML editor too as the actual block content is removed when switching to it and replaced with the "block state", then the "block" is rendered again on switching back to visual. Of course this can be extended for use with the block demarcation.

[Lewiscowles1986]

At the point post_content is no longer the universal source of content for core post-types, does it not beg the question, why are we not using a document db, or storing the bulk of the post as a document or with document semantics?

I'm not sure a JSON field in MySQL is such a good option (although it's query-able), but for some systems that live on top of WordPress there are already a litany of fields jammed into the post meta table(s)

[aduth]

@azaozz Thanks for the feedback. You make good points. I'll try to address them:

Removes post_content as "the source of truth"

For the purposes of the post's appearance on the front-end, this is not true. The markup is always saved in post_content in the exact way we'd expect it to appear on the front-end. That said, your statement is true in the context of an editor. Or at least, an editor is fully capable of displaying a block's content, and even allowing direct edits to the HTML, but if this were to occur it loses all distinction of being a block. That seems fine with me. If an editor opts in to providing behaviors supporting detecting and treating the markup as a block, they can continue to allow edits wherein the the underlying state is the source of truth. My proposals above support this for any editor, not just wp-admin, via schema structure and block parsing.

Because of the above, the HTML editor becomes useless/pointless. Even a small variation in the HTML (for example adding an attribute to a tag) will invalidate it and make the block non-editable in the "visual" editor. This would be a pretty dismal user experience.

Has it ever been a desirable user experience for them to need to make edits in the HTML editor? I'm not going to dispute the claim that the above proposal more heavily emphasizes edits made in the visual editor, with graceful degradation for edits made in the HTML view. The initial negative reaction to this is obvious, but I'm inclined to think they're based on current habits and workflows which require one to inevitably make some edits in the HTML editor. It should be our goal to eliminate the need for those workflows.

Storing the "block state" in post_meta sounds like a good idea. However imagine a post with ~50 blocks (not uncommon) having 20-30 revisions (also not uncommon). To save such post we will need ~1500 rows in the DB! The post_meta table will literally "explode" :)

Yes, it's a lot of data. I'll address this with alternatives later in my comment.

What would be the "block state" when the user pastes big chunk of HTML, for example: a 500 cell table from Excel? Do we duplicate that whole table and save it separately?

We'll already need some sort of "transformations" API to support block type switching that's been shown in the current mockups. This seems like it might be of the same sort; needing to detect format of some pasted content and parse it into most applicable block type.

As far as I see most blocks in a post will be (big) chunks of HTML. They will only have one state (i.e. will be stateless) and have no settings. As HTML they are "ready to go" and will not need parsing or rendering. The "block state" would be useless for them.

I think part of what might be leading to this conclusion is that content we've supported 'til now and block types supported in the prototypes are very basic in nature. But even for basic blocks I see the value in thinking of them in terms of state. Take the image block: it's src and caption (and sure, a few other properties). Presenting it this way to the block implementer more easily enables them to describe the structure of their block given the state at any particular point in time. This becomes more compelling when understanding that by the nature of an editor these values will change over time, and a change is merely updating the value of the state property.

By contrast, what's the alternative? Offering a node or markup where the developer is expected to attach their event handlers and update the markup in response to the user making changes? This works, but is fragile at best. We also lose the ability to make meaningful observations about a block's markup in anything except the browser (i.e. how does the mobile app know how to do anything with Plugin X's special image block?).

On the other hand this is a good idea for "dynamic blocks".

I've avoided distinguishing between static and dynamic blocks 'til now, because I don't think we need it yet. The observation is sensible though, because we already think of dynamic content in terms of properties and values (shortcode attributes). In practice though, even with proposals as I've made earlier, it's likely (for the near term anyways) we'll just save them to post_content as shortcodes:

<!-- wp:gallery {"ids":[48,23,91]} -->
[gallery ids="48,23,91"]
<!-- /wp:gallery -->

Which at a glance appears a huge waste in its duplication, 'til one realizes that while this is the format in which it's saved, the presentation in the editor is much different, so having access to its raw data is quite valuable. We do this already in the editor by parsing the shortcode itself. I find this to be no different between shortcodes or static content, so the proposal quite intentionally generalizes this behavior to the block level.

---

Let's consider alternatives though. I'm still relatively convinced that the previous explored proposal covers most bases well in terms of storing content as just markup, making sense of blocks beyond just the editor in browser, and of course hitting user expectations on front-end and editor interactions. It does this while at the same time instilling confidence in a block implementer through obvious APIs for working with the underlying state of a block. And I'd encourage to not think of state as any particular "data" representation, but instead simply as the values by which the markup for a block would vary.

Delegate State Parsing to Developer

The most obvious complaint will be storage requirements if we were to serialize this state in its raw data form. We could mitigate this slightly by removing developer conveniences around not needing to perform any parsing and instead delegate some of this responsibility to them to determine state based on markup of a block. In the case of an image block, this could look something like:

function parse( figure ) {
    return {
        src: figure.querySelector( 'img' ).src,
        caption: figure.querySelector( 'figcaption' ).innerHTML
    };
}

There's a few downsides here:

• There are a number of markup variations for the image block I need to consider. What if there's no caption? We'd probably not want to render a figcaption at all in that case. If this above logic were to then be run, it would throw an error when attempting to access the innerHTML property of a non-existent node.
• This parsing logic can only be performed in the context of a browser. We can't make sense of a block on the server or in mobile apps.

To the second point, we could use a formal grammar available both in the browser and on the server, expecting the developer to traverse it themselves, or provide conveniences for accessing values by path (XPath, anyone? 😄 ).

I've yet to see a viable solution here.

Forgo Block Distinction for Common Elements

An idea that works particularly well with the single instance approach and aligns with the current editor is to not think of content as a composition of blocks but rather give blocks special treatment as an optional add-on to default editable behaviors. We already have sensible defaults for managing paragraphs, headings, and other base elements, and these would have been the worst offenders for duplicated content of serialized state anyways. If only these special blocks need state consideration, there'd be much less to store overall.

There might be a way to extend this to a multi-instance approach to allow a stateless block to be defined as a particular tag name, for example:

registerBlock( 'wp', 'text', {
    nodeName: 'P'
} );

Where the editor knows to render a node of this type and apply a reasonable contenteditable default.

[Lewiscowles1986]

By contrast, what's the alternative?

single meta containing data that requires a checksum field, storing all parsed block-data. It's n+1 from a very zoomed out high-level of post entry + single meta field (not computer instructions or even PHP instructions).

Saves time processing each time editor is loaded, and can be confirmed with a property containing a hash. If that hash matches the hash of post_content, use the unserialised meta data. It's not a worse case enhancement, but it could save complex parsers time initially deciphering, and if it could be baked into the editor it could remove the need to have comments (something PHP used to use).

As for serialization, IDK it's a difficult area that even mainstream desktop apps like libreoffice and MS office that have many hours of management of rich types. They still fail to come to elegant solution on complex block types like this, it's not exactly an elegant problem having potentially n types of block.

[westonruter]

Storing all the blocks' data in a single postmeta field would indeed mitigate the explosion of DB rows in revisions.

Question: has wp_posts.post_content_filtered been considered for storing anything? It's a field that doesn't get much attention or use.

[mtias]

Interesting conversations here, thanks @aduth for igniting it.

I want to take a small step back and think about the different layers we are interacting with: data source, data interface, editor output, viewer output, etc.

As discussed in the post where we introduce the idea of HTML comments as demarcation, I believe data source is important to be in post_content for any static block (including blocks that have fixed fields like captions and cites). It is less important for dynamic blocks that need to pull resources from different places to be rendered.

We only started with simpler blocks because we need to have a solution for them that doesn't fragment its reliability as editable HTML content. I want us to have a robust approach here that allows us to absorb post formats and move on to more complicated blocks.

That is not to say that the more complicated blocks should have exactly the same way of storing content in HTML. The data interface, which represents how the developer of a block interacts with the state of the block, is separate from where and how we store that data. I see a lot of value in treating blocks as state for the editor experience, and presenting them as state to the block developer, but not in storing it as state separate from post_content, at least for static blocks.

So let's look at what you see as the data state layer; as a developer I would request for my block:

type: "image", { 'img.src', 'img.alt, 'caption' }

That's what I need to build my UI and to render it in the editor as content to be manipulated. How we get this info into that structure is a concern of the editor architecture, and whether we infer it from HTML source or retrieve it from post-meta doesn't matter at that point to the developer.

It does matter for us creating the editor internals and its API. So let's consider the image with caption source in the HTML:

<!-- wp:image -->
<figure>
  <img src="/">
  <figcaption>A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

All the data is expressed in regular HTML, so to manipulate the caption separately you need to know about the shape of the markup. That can be as implicit as saying caption: "figcaption" when I define my block data requirements and we'll get the value for that DOM element to you. Or we can add some additional tools to explicitly label these fields like:

<!-- wp:image -->
<figure>
  <img src="/">
  <figcaption data-block-field="caption">A picture is worth a thousand words</figcaption>
</figure>
<!-- /wp -->

This is one possible way of storing the relevant pieces to be extracted into the consumable structure for the developer. By default a developer request for a field with name could try to match data-field="name", otherwise it could match an HTML_Tag with name: "name".

Here's a grammar parser output for a simpler version without a caption:

[
  {
    "type": "WP_Block",
    "blockType": "image",
    "attrs": {},
    "startText": "<!-- wp:image -->",
    "endText": "<!-- /wp -->",
    "rawContent": "<!-- wp:image -->\n<figure class=\"\">\n  <img src=\"/\">\n</figure>\n<!-- /wp -->",
    "children": [
      {
        "type": "Text",
        "value": "\n"
      },
      {
        "type": "HTML_Tag_Open",
        "name": "figure",
        "attrs": {
          "class": ""
        },
        "text": "<figure class=\"\">"
      },
      {
        "type": "Text",
        "value": "\n  "
      },
      {
        "type": "HTML_Tag",
        "name": "img",
        "attrs": {
          "src": "/"
        },
        "startText": "<img src=\"/\">",
        "endText": "</figure>",
        "children": [
          {
            "type": "Text",
            "value": "\n"
          }
        ]
      },
      {
        "type": "Text",
        "value": "\n"
      }
    ]
  }
]

We may not need to mark the image src as a data-field because we can rely on the semantics of <img> and the block developer knows there will be a single img element in the block returned by HTML_Tag with name img and get its attributes from the attrs object which we expose as name.attr.

I am okay with having references to data that exists elsewhere for blocks that need the dynamic nature of that data, and it's ok to optimize for that. As long as we design the different parts of the puzzle properly, and the abstractions work at the same level, I don't think we need a single source of truth across the board.

For complex and dynamic blocks this is a possibility and we don't expect the source of truth of these posts to exist within this HTML.

<!-- wp:post-list {"query": {} } -->
... Fallback HTML ...
<!-- /wp -->

[aduth]

So let's look at what you see as the data state layer; as a developer I would request for my block:

type: "image", { 'img.src', 'img.alt, 'caption' }

How do I as a developer even come to conclude that img.src is the correct path to access? There's a very opaque cloud of abstraction assumed here that I find only works well enough to garner a "Yeah, I suppose that makes sense" retrospective response and lacks an obviousness that solicits the confidence of "I know how to do this on my own".

[aduth]

Another problem I think we've overlooked 'til now is that while we've been discussing options for universal understanding between browser and server of block state and state shape, we've not put the same consideration toward generating block markup. So even if the server or mobile app is able to understand and perhaps even update the raw state values, as of now we have no consensus on how to update the markup accordingly.

As an addendum to my earlier comment, it should be a requirement of the editor that a client is able to update its markup in response to user input without a round-trip to the server.

Off the top of my head, we may have a couple options, neither of which I'm particularly happy with:

• A block's markup rendering is implemented in JavaScript, meaning any client (like the mobile app) must have access to a JavaScript runtime in order to generate the markup (WebView, JavaScriptCore). The server would not be able to generate markup unless Node.js or V8js are detected (we should assume most hosts will not support these).
• A block's markup rendering is implemented in PHP, but in a template language that can be understood and mimicked in a client. An obvious candidate is sprintf (sprintf.js), which is not a joy to work with for complex markup, might have sanitization implications, and would not provide us with good patterns for sharing common complex behavior among blocks.

Does anyone have ideas for how we might tackle this?

[westonruter]

In an ideal world, the client and server would be able to render a block. But as you note, this is really only feasible if both the server and the client are running the same language (e.g. JavaScript)… or if there is a sufficiently powerful templating language that has implementations in both JS and PHP, and I don't think Mustache is powerful enough.

I'm inclined think it is not reasonable to expect that the arbitrarily-complex rendering logic on the frontend (in PHP) to be able to be run on the client by JavaScript (in the editor) as well. I think the approach that Shortcake took by rendering shortcodes on the server to be a reasonable baseline pattern to follow with rendering blocks in the editor. Otherwise, a block could provide a basic rough/raw rendering of a block for previewing in the context of the editor, such as is presently done with image galleries.

[sirbrillig]

I ran into this same challenge when I was exploring these ideas at the end of last year. Mostly I had the same thoughts as you @aduth and @westonruter: shared markup language (that's probably insufficient), or server-rendered only (makes offline impossible, slow to update, and sucks for poor connections). I was only able to come up with two conclusions of my own:

• It could be possible to write a rendering function in both PHP and JavaScript for each static block. This would be really annoying for a developer, but could possibly work if the tooling/templating was really good.
• Since the editor is always client-side (JS or native running JS in a web view) we can always generate markup on the client in JS and save it to the server. This is just what you were describing above, but I don't see a big problem with requiring native clients to support JavaScript. I would note, though, that dynamic blocks will likely need to be rendered server-side, so they will need to be written in PHP. I suppose my question is: why does the server need to be able to generate static block markup?

[aduth]

This is just what you were describing above, but I don't see a big problem with requiring native clients to support JavaScript.

Do you have anything specific in mind for an approach here? I've been thinking about this a bit and was curious if we could either treat the block effectively as an iframe via WebView (with some communication to the parent native context), or generate a scaffolded form from a block's schema and, upon change, execute the JavaScript for generating markup given a particular set of block attributes. This thinking might be a little naive since I've not worked much in a native environment. Those more familiar could indulge us with possible gotchas, particularly around loading scripts from a user's own site.

[BE-Webdesign]

To throw in my two cents:

I agree with azaozz's sentiments. To me the editor should only be concerned with manipulating the post_content which is then sent back to the server and saved. The server would send back out the previously saved post_content pretty much like what WordPress already does. The client would parse the post_content into a state tree which can be manipulated and re rendered, or not; as TinyMCE could essentially handle that for us. By keeping the flow the same we most likely will have 100% backwards compatibility with the current WP Editor. The new one will be a lot shinier though and provide a more seamless inline editing experience.

Proposed Flow:

post_content in DB (mostly just HTML) --render-to-editor-> Editor parses post_content into the editable UI (should be handled by TinyMCE) --save-to-server-> The post_content markup is saved in the same fashion WP currently handles everything. The more I think about this project, the more I think the focus in some ways should be centered around making an awesome single TinyMCE instance, that has an inline block editing UI. After looking over some of the things that can be done with TinyMCE, a lot of our problems can be solved by hacking on the APIs it provides.

On the "secret/private" data idea:

This type of data should/would exist only on "dynamic blocks" of content. We should make use of the shortcodes API already present in WordPress to handle that kind of stuff and improve that experience via iseulde's idea. Any reference data that needs to be used would be accessed via the shortcode/resource-id to prevent the sensitive data from being displayed in the actual markup publicly.

Block Registry/Definition API:

The API for defining/registering blocks should be very simple and only be concerned with defining/registering. As blocks are registered there should be someway for TinyMCE to pull data from the registered blocks and add that to the interface for the editor. The registering of blocks could happen server side, and could be pulled into the JS realm when the editor is being initialized. It could also happen client side and forego PHP, however the more I think about it the less I like this approach. There should probably be an association between a block and its controls, so that TinyMCE can dynamically generate the appropriate controls of a selected block based on the block types registered to the editor.

Server vs Client:

I do not see any benefit to trying to have the server render a block via static state, that should be handled via JavaScript. The way in which WordPress interacts with PHP/HTTP is stateless. The initial render and the previously saved output of the editor are one in the same. Adding an extra layer where PHP is rendering static state data as HTML seems like an unnecessary step. Use the server to serve what is already saved and let the client interpret the saved post_content and provide a GUI to manipulate that interpreted state. "Dynamic Blocks" should be rendered via the server in some form or another, (for now shortcodes somewhat work) or at the bare minimum the server should expose what controls/state is necessary to render a dynamic block client side.

Environment:

By contrast, what's the alternative? Offering a node or markup where the developer is expected to attach their event handlers and update the markup in response to the user making changes? This works, but is fragile at best.

This is basically what TinyMCE does and although I agree it is fragile, it is also probably the most practical approach moving forward. I believe digging into the flexibility of TinyMCE to enable it to understand blocks properly is probably more important than worrying about rendering state to markup. A state -> markup system can always be created further down the road if it is ever considered that TinyMCE won't be used anymore.

We also lose the ability to make meaningful observations about a block's markup in anything except the browser (i.e. how does the mobile app know how to do anything with Plugin X's special image block?).

Plugin X's special image block type could be registered to expose data that could be used in a non browser environment. The HTML could be parsed as the state and after it is manipulated then rendered as new HTML in a non browser environment. The exposed block type data could be used in tandem with the HTML in a non browser environment. I think veering away from HTML as the main data format of representing post_content, although technically appealing, would be impractical in the context of WordPress and TinyMCE.

how far we take the idea of React's "UI a function of state"

Make the UI a function of post_content. Essentially a state tree and the post_content are just a different representation of the same thing.

tl;dr:

Don't over complicate things and veer too far away from the flow of data WordPress already uses. I think the Block Type Defining API should be server side in PHP. The work on the editor should mainly be focused on making a Single TinyMCE instance understand how to parse, interpret and modify blocks, which are then saved as post_content in the typical WordPress fashion as a mix of HTML and shortcodesque things.

[aduth]

@BE-Webdesign Good stuff, I can always get behind efforts to avoid introducing unnecessary complexity.

That said, let's not trivialize making sense of the markup nor working with a DOM. In some ways we've done ourselves a disservice by starting with very simple block types. As far as implementing blocks go, headings and paragraphs are solved problems. The more interesting cases are galleries, contact forms, sliders, maps, repeaters, testimonials, subscribe forms, logo soup, plans comparisons, etc. We should be thinking about all of these block types as we work through the APIs that developers will use to build them.

Editor parses post_content into the editable UI (should be handled by TinyMCE)

This is easy for a single element like a paragraph tag, but TinyMCE won't be able to make sense of more complex markup on its own. The step from post_content to editable UI is a big one, because the markup needs to be transformed from one which is appropriate for published content to a UI which makes the most sense for edits, all the while preserving underlying values back and forth.

TinyMCE can dynamically generate the appropriate controls of a selected block based on the block types registered to the editor

Dynamically generated fields don't make for a good UX. When I'm adding a map block, I should see a map, not just a text input.

[...] the developer is expected to attach their event handlers and update the markup in response to the user making changes [...]

This is basically what TinyMCE does and although I agree it is fragile, it is also probably the most practical approach moving forward

Kindly I disagree. For anything but the simplest elements, working with and keeping the DOM in sync with our understanding of state quickly becomes very difficult. Introducing conventions or tools is perfectly fine if the cost of becoming familiar with them is strongly negated by an increased sense of overall confidence on the developer's part to create sound implementations.

---

Now a few points to concede on my part:

• Shortcodes would be a reasonable practical solution to a need for secret data. It's not a great overall solution if we were to ever want to consolidate the idea of a shortcode into a single unified block pattern, and shortcodes seem better suited for dynamic content (in my example, "Gravatar" would be a static block).
• Maybe keeping state in the DOM is fine, but we still need patterns around making sense of the underlying values. We could do this by mapping data- attributes like @mtias described, assuming the implementation of a parse function, or perhaps another solution that's yet to be explored. Despite all its downsides, the one thing "storing in meta" had going for it was being totally transparent to the block implementer.
• We have a choice on just how opinionated we want to be with patterns. The block proposal in #300 is less opinionated in that by providing access to a raw DOM node to work with in the editor, a developer can choose to use whichever tools they're most comfortable with. By contrast, my own in #302 is admittedly more opinionated with the introduction of React, but argues in doing so that it's the best and most succinct fit for the requirements.

[BE-Webdesign]

@aduth,

I think for the most part we are on the same page, and I probably am not articulating my ideas too well. My ideas could also be terrible 😄 . I am hoping there can be a way forward that somehow the new block editor can be jump started by post_content that a WordPress install already has, that could include shortcodes. As the block editor is used the block demarcators will be added and store the state into the post_content, so on the next rendering of the editor, the information for the blocks will be there. Getting TinyMCE to do that will certainly be a pain, but I think it could potentially lead to a smooth transition from the current editor to a new block based editor.

The step from post_content to editable UI is a big one, because the markup needs to be transformed from one which is appropriate for published content to a UI which makes the most sense for edits, all the while preserving underlying values back and forth.

Definitely, and this is what TinyMCE kind of does for us or at least closes that gap a bit. I think it is possible to get TinyMCE to understand "blocks" from the post_content and render the appropriate UI based solely off of information stored in post_content.

Dynamically generated fields don't make for a good UX. When I'm adding a map block, I should see a map, not just a text input.

I was talking about the controls for a block, a map will definitely be there for a map block. So for a "map block" TinyMCE will generate a control that can open a modal or something to switch up the map settings. I am probably not explaining my ideas well 😄 .

Kindly I disagree. For anything but the simplest elements, working with and keeping the DOM in sync with our understanding of state quickly becomes very difficult. Introducing conventions or tools is perfectly fine if the cost of becoming familiar with them is strongly negated by an increased sense of overall confidence on the developer's part to create sound implementations.

You could have React components inside the TinyMCE instance ( or whatever you want really ) to ease the handling of a more complex block.

[androb]

Re @BE-Webdesign

You could have React components inside the TinyMCE instance ( or whatever you want really ) to ease the handling of a more complex block.

This is exactly what the TinyMCE Core team are looking at the moment. We want to be able to support "blocks", "components", "widgets" or whatever to coexist both "in" TinyMCE as well as outside of it. We are trying to get to some sort of 'best of both worlds' approach... but appreciate this project is moving very fast. We're trying to bring the TinyMCE team's ideas in :)

[ellatrix]

Experimenting a bit with some of the ideas @aduth had with having a render function and parsing, but without touching the DOM. It would look something like this, where the received content is JSON and h has helpers for this content. Instead of using arrays, you can also use h( name, { ..attrs }, ...children ) and if the developer wants, they can use JSX to generate it. render will need to handle empty props to insert an empty version (placeholder) of the block.

function render( props ) {
    return (
        [ 'figure', { contenteditable: false },
            ( ! props.src ?
                [ 'div', { onClick: onClick },
                    [ 'svg', ... ],
                    [ 'p', 'Pick image' ]
                ] :
                [ 'img', { alt: props.alt, src: props.src } ]
            ),
            [ 'figcaption', {
                contenteditable: true,
                placeholder: 'Write caption\u2026',
            },
                ...props.caption
            ]
        ]
    );
}

function getProps( content, h ) {
    return {
        alt: h.getAttribute( h.find( content, 'img' ), 'alt' ),
        src: h.getAttribute( h.find( content, 'img' ), 'src' ),
        caption: h.getChildren( h.find( content, 'figcaption' ) )
    };
}

function onClick( props, callback ) {
    filePicker().then( function( files ) {
        ...
        callback( render( { src: src } ) );
    } );
}

registerBlock( {
    name: 'image',
    ...
    controls: [ ... ],
    render: render,
    getProps: getProps
} );