Improve placement of AMP validation errors in the block editor

[jauyong]

Feature description

Consider moving the warning notice to the sidebar since inline it will cause problems based on where the block is located in the editor. The block could have some error outline so that when clicked it will show the block sidebar with the validation error information in an expanded panel.

This was taken out of #3664

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• Make sure that the aspects in #5304 are accounted for.

Implementation brief

QA testing instructions

Demo

Changelog entry

[westonruter]

Related: We need to provide non-technical users with an escalation path with technical administrators to get assistance with validation errors when they occur. For users who have developer tools turned off, showing the full error details should not be done by default but rather a way for them to ping an administrator with the details.

[jwold]

Spent some time today playing with a concept for this. Here is what it looks like right now when you get an inline error:

The idea then is to highlight the block where the error is occurring, and if you select the block, flag the notice at the block level, but only by showing it in the sidebar:

[westonruter]

Yes, that is better. However, the warning icon probably can't be placed along the side since it would cause the same layout problems as the inline notice. So can the button to show the validation errors be moved to the block toolbar? Actually this may not even be needed since as soon as a block is selected the block validation errors panel would appear in the sidebar. Maybe that wouldn't promote discovery enough and an additional toolbar button could focus on that panel and expand it. Not sure what the best practice is there.

[jwold]

I like that idea. There are also scenarios where someone may not have the sidebar open, so the [i] info icon could direct a user to click it and have the sidebar open with the relevant error message. Here's an updated prototype:

[jwold]

We just had a discussion about this, and it may make sense to remove the (!) icon from the top right, to simplify the work, since manipulating the DOM may be challenging.

[johnwatkins0]

Wanted to make a suggestion here. In addition to notice fatigue in the editor, a lot of sites have an excess of panels in the editor Document sidebar. It's easy for individual plugin panels to get visually lost or confused with core panels, as they all look the same.

I suggest using a PluginSidebar instead. We should be able to tailor the sidebar icon to the severity of the issues (I'd need to test this to make sure), and then all the AMP features would be in one place, instead of mixed into the block editor or the main document sidebar in a less controlled manner.

[jwold]

@johnwatkins0 that's a great idea. I've been wondering how we would fit a third tab in the Document/Blocks section. A plugin sidebar makes more sense.

[westonruter]

Would a PluginSidebar work well to display both document-level and block-level validation errors? Or would block-level errors still show up in the block settings sidebar when the block is selected?

[johnwatkins0]

I might be going to extremes here, but I think as long as there is an indicator that there are issues with blocks in the post, we can handle everything inside a plugin sidebar.

Benefits of this:

1. No need to override blocks' internal behavior, which is still a messy process with the block editor. I think we can passively collect the invalid blocks in the sidebar and allow them to be removed from there.
2. When opening the AMP plugin sidebar, there is no confusion for the user that they are working with the AMP plugin and not something from core or some other plugin.

We can even mix in some of the plugin's own styles into a sidebar. Personally, this sort of thing is very anxiety inducing:

In a sidebar, we'd have a bit more freedom around the interface.

[johnwatkins0]

@westonruter Have we ever considered validating individual Gutenberg blocks' output from within the editor instead of only on post save? I think it could be done, and would be a nice bonus to the UX updates. We could capture validation issues before the post is saved, surface them while the user is editing, and then also have a PluginPrePublishPanel section.

[westonruter]

In a sidebar, we'd have a bit more freedom around the interface.

I'd love to see some mocks for how this would look for both document-level and block-level errors.

Have we ever considered validating individual Gutenberg blocks' output from within the editor instead of only on post save?

Yes, we have. However, we determined it won't work reliably. In particular, it wouldn't catch validation errors that only occur in the frontend context (e.g. when render_block or the_content filters are manipulating the content). There also may be some such filters that behave differently when amp_is_request() and so we'd have to somehow force the blocks to render the AMP versions in the editor: in particular, it also catch validation errors from the PHP output of dynamic blocks, right? We'd have to somehow force ServerSideRenderer to render the AMP version, but then since Bento AMP is not yet available, rendering AMP components in a React context will not work.

Lastly, if we did try to do a quick client-side validation in the editor without waiting for server-side validation to be performed upon save, there will often he scenarios where the client-side validation fails but then the server-side validation would pass, and this is because of all the HTML conversions that the sanitizers perform. So to be consistent we'd have to replicate PHP sanitization logic in JS, which would not be feasible in the short term.

[johnwatkins0]

It does sound very complicated to try to do AMP validation client-side and I wouldn't want to take that on, but how about something like:

1. When a new autosave or revision is created (definitely not on every editor state change), make a request to the custom REST endpoint for validating URLs, passing in the preview URL.
2. Send back the results and parse these results into the UI for a plugin sidebar.

I'm sure it would end up being much more complicated than that, but it would allow us to essentially take snapshots of the post's validation state during editing. Of course I am assuming that post previews are processed by the plugin the same as published posts, which I could have wrong.

[westonruter]

This should actually be current plugin behavior. Whenever a draft is saved (including autosaves) the validation is performed. (This is not done if DevTools is turned off or if async validation is implemented per #2069.)

I thought this was the original behavior, but it doesn't seem to be working anymore. Note in https://github.com/ampproject/amp-wp/issues/2069#issuecomment-530954697 that it is running during DOING_AUTOSAVE requests. (Note the the Classic Editor may be the context of that issue.)

See also https://github.com/ampproject/amp-wp/issues/2069#issuecomment-530961900:

The ideal is that the user is provided feedback as soon as possible after they have added something that is not AMP compatible. The idea behind autosave is that, especially in the context of Gutenberg, warnings can be added shortly after a block has been added which does something not valid in AMP.

I'm not sure why this is broken now. Perhaps it is broken specifically in the block editor because autosaves are now managed via a separate /autosaves endpoint? The cause could be that the amp_validity REST field is not being added to the autosaves endpoint, but rather to the main posts endpoint:

https://github.com/ampproject/amp-wp/blob/34e098ca0b773138ec33cdbf3228ccaedcaad64d/includes/validation/class-amp-validation-manager.php#L689-L706

Or perhaps the issue is that requesting the frontend doesn't have the autosave revision is not being applied for the post preview when loopback request is being made?

Nevertheless, it may be disorienting if validation errors all of a sudden show up when an autosave happens. At least with the current (broken) behavior, the validation errors only show up once the user hits Save Draft or Publish. The downside is that the validation errors may only be caught once the post is published, if the user never hits Save Draft or adds AMP-invalid content before an autosave happens. Then again, if a loopback request is made for every single autosave, then this also adds server load.

If/when we restore the autosave validation checking, we'd want to make sure that the UI update for a newly-detected validation error doesn't adversely disrupt the user's workflow, or make the appearance of that error notice overly disconnected from the point at which the user first added the AMP-invalid content. Putting the results in a plugin sidebar should help with that a lot.

[johnwatkins0]

@jwold I started doing some testing with a plugin sidebar and can confirm the following:

1. In the screenshot below, where I currently have the button that says "AMP" (which I've hardcoded for testing), that component has access to all the editor state and can be adjusted at any time within an editing session -- e.g., to show a warning state if there are errors, or to show an "OK" state if there are no errors. What I'm thinking for this button is we could have one or more SVGs that in combination with CSS can indicate whether there are validation issues needing attention.
2. Similarly, that panel that shows "Hello" also has access to editor state and can, e.g., respond as blocks are edited or removed. I'm going to do more experimentation with this part.

Note that when the user clicks that Star icon (which indicates whether the plugin is pinned), the AMP plugin sidebar button is moved into that three-dot menu. When the plugin is unpinned, it may become challenging to surface validation errors, although we could do something like add a line to the editor status panel.

More things I want to test:

1. Listing out validation errors in the plugin sidebar.
2. Opening/closing the sidebar from elsewhere in the editor.

[jwold]

Great work. I like where you're going. I'm wondering if following how Yoast does this might make sense. Or, just a simple badge.

[westonruter]

Oh yes, I like that.

The dots in the Yoast button refer to the Readability analysis and SEO analysis?

Nevertheless, they have a panel in the Document tab in addition to having their own plugin sidebar. That “Improve your post with Yoast SEO” is a shortcut button that opens the plugin sidebar. We could do the same, where we have “Review validation issues (X)” instead. This can also be where we put the AMP Enabled toggle.

In terms of the icon, we shouldn't rely on color alone as Yoast is doing. This circles back to the issues discussed in depth in https://github.com/ampproject/amp-wp/issues/3726. We should move toward aligning with the AMP Validator extension, but it's icons aren't yet ideal either: https://github.com/ampproject/amphtml/pull/30011#pullrequestreview-477225336. So perhaps a simple error count in red is good. Maybe the icon should have 3 states:

• AMP Enabled (plain AMP logo)
• AMP Enabled but there are unreviewed validation errors (AMP logo with red count)
• AMP Disabled due to validation error-causing invalid markup being kept (AMP logo with a slash through it?)

If the "AMP Enabled" toggle is off, then the plugin sidebar should be disabled/removed entirely.

We also need to keep in mind what the admin bar shows on the frontend, and align as closely as we can with how dashicons and colors are used there: https://github.com/ampproject/amp-wp/pull/4986.

[jwold]

Two ideas I've been playing with:

[westonruter]

I like those a lot. I think Option B is preferable.

[johnwatkins0]

Agree with Weston. As a user, the number is going to catch my eye much more. It just needs to be easy to get the number resolved down to 0 -- like the notices within the sidebar should have an equivalent of, "I don't care about this. Just make it go away."

@jwold, I'll keep going with this branch and start to pull the notices into the sidebar without spending time on the design part of it. I'll share my progress as I get further along, but let me know any time if you have ideas for the design aspects.

[jwold]

I'm not sure if I'm tracking the actions fully, so I'd love feedback. I created a prototype that goes through the icon types.

1. No error
2. Error with sidebar closed
3. Error with sidebar open
4. No error after removing block
5. Error with sidebar closed and block dismissed

[johnwatkins0]

@jwold I might not be fully following what this one represents. Can you explain what you have in mind?

[jwold]

@johnwatkins0 I started to write up an answer to this, but left it out in favor of just showing the prototype initially.

I'm not actually sure when that should occur. Further above Weston described this as:

AMP Disabled due to validation error-causing invalid markup being kept (AMP logo with a slash through it?)

But I'm not sure how it would be triggered

[johnwatkins0]

@jwold I see. I think there is a question about what we want to do in the sidebar. Step 1 is moving all the block notices into the sidebar, as shown in your animation. A couple things to keep in mind with that:

1. In the sidebar, the notices can look how they do in the editor (the core-style yellow), or they can look how we want.
2. In the sidebar, we should have a way to highlight and scroll to the related block. I've tested this on my prototype branch and there is a core Gutenberg function that does it, so the question is how to represent it in the UI (e.g., a "Select block" button.
3. A question is whether we want to show each block validation error separately in the sidebar, or group them together by block (as they are now).

Beyond the block validation notices, my next question is whether we want to move a version of this UI into the editor sidebar as well:

It could be done but would obviously add complexity. It sounds like what is shown in my screenshot above (kept errors) is what would correspond to the broken-AMP icon. Currently this is not obviously surfaced in the editor, however. Even if I set those two errors as "Kept," they still show up in the editor:

And when I leave the URL validation screen in this state:

I'm honestly pretty confused about why the editor for that post now looks like this:

I feel there's probably some way to tie it all together.

For a first step, though, I think we just need to move the AMP-related editor notices into the sidebar.

[westonruter]

As for the plugin sidebar icon states, this is is what they'd represent:

Editor Icon	Current Admin Bar Icon	State
		No validation errors or all removed and marked as reviewed.
		There is at least one unreviewed validation error “which's” invalid markup is being removed.
		AMP is disabled due to invalid markup being kept for at least one validation error. (The unreviewed count should probably be included here too.)

Maybe there are opportunities to better align the icon usage in these two contexts.

Naturally the screenshot with the black background just indicates the pressed state:

Please cross-reference with the screenshots and various scenarios in this PR: https://github.com/ampproject/amp-wp/pull/4986.

A couple things to keep in mind with that:

It would be good to list out all validation errors in the plugin sidebar in the same way that they are all listed out on the validated URL screen, but screen real estate is indeed a concern. But the ultimate goal would be for the user to not have to leave the post editor to view and moderate these validation errors. Perhaps an abbreviated view can be shown when the sidebar is shown, and then eventually show full details if the sidebar is expanded to take up the whole page? (Not even sure this is a capability.)

1. In the sidebar, the notices can look how they do in the editor (the core-style yellow), or they can look how we want.

We wouldn't want to present the information in a warning notice like we have today. There's no need to have the error details collapsed like we have in the inline warnings.

The key information needing to be displayed in the editor context is:

1. What the error is (e.g.invalid script).
2. The theme/plugin responsible for the error (if available).
3. (Optional) What action the plugin is currently taking (usually “removed”).
4. (Optional) Whether the validation error has been reviewed already.

All the other information that is currently displayed on the validated URL screen can be integrated later, including the Source Stack information and the UI to manipulate whether the invalid markup is kepty/reviewed and whether it is reviewed.

Again, ideally the whole UI on the validated URL screen could be packaged up into a set of components which could be integrated into the edit post screen. There should be no need to leave the edit post screen to review the validation errors and take action on them.

2. In the sidebar, we should have a way to highlight and scroll to the related block. I've tested this on my prototype branch and there is a core Gutenberg function that does it, so the question is how to represent it in the UI (e.g., a "Select block" button.
3. A question is whether we want to show each block validation error separately in the sidebar, or group them together by block (as they are now).

When a given block is selected, it would be great if the validation errors caused by that block could be highlighted. Maybe that means having all other validation errors collapsed or filtered out of view. But it should be easy to also see the list of all validation errors as well.

Currently this is not obviously surfaced in the editor, however. Even if I set those two errors as "Kept," they still show up in the editor:

Yeah, that was probably not the best call. But in lieu of having the plugin sidebar, showing them in the inline notice seemed the quickest way to show blocks which had validation errors. Currently the warning notice shows up for any block quick has unreviewed validation errors or validation errors with invalid markup kept. These are the two scenarios which need user attention because either they need review or they are responsible for blocking AMP for the page.

[westonruter]

The editor icons should also be displayed in a block toolbar when a block is selected, specifically if there are any validation errors that are occurring in the block? Clicking that block toolbar button could then expand the AMP plugin sidebar and highlight the block's own validation errors in that list.

[westonruter]

We need to also account for the Classic Editor case. This probably means a metabox that will contain the same components that would be displayed in the AMP plugin sidebar in the block editor.

[johnwatkins0]

Following up on our call with this @jwold @westonruter, I think we should start with the bottom-up approach with this, which would be:

1. Create the sidebar.
2. Move the block validation notices into the sidebar -- for now, still linking to the URL validation screen for the complete validation functionality.
3. Implement interlinking between the sidebar and the blocks in the UI, with Weston's suggestion just above about the block toolbar button + a button on the sidebar notices to highlight the relevant block.

Does this sound right? This update could be for the 2.1 milestone, and then the larger top-down change -- redesigning the URL Validation components and making them reusable anywhere -- would follow longer-term.

[amedina]

For the sake of consistency, the icon for "There is at least one unreviewed validation error “which's” invalid markup is being removed." should be the AMP logo with a yellow-highlighted exclamation bubble overlapping it. This keeps the association of "yellow ==> error not reviewed, removed, valid AMP, possible functional/visual problems"

[westonruter]

@johnwatkins0 Yes, and also move the AMP Validation notice in the Classic Editor from an admin notice at the top down to a metabox.

[jwold]

Wanted to share an update with progression for discussion.

[jwold]

Here's an updated prototype:

Video: https://v.usetapes.com/url0fZX6h0

It includes audio as I share the thinking behind how it works.

[johnwatkins0]

@jwold While I think what you show in the video is great from a design perspective, I see that the drawers for the errors are including actions to Keep/Remove errors and to mark them as reviewed. I think at this stage we don't have the backend infrastructure in place (REST endpoint etc.) to handle those actions dynamically through JS operations (correct me if I'm wrong @westonruter). I would suggest that we keep the overall approach in your video but save those actions for a later time. The MVP for this is we move the notices about invalid blocks into the sidebar (without the "Remove block" functionality, which we don't need) and continue to direct users to the URL Validation screen for the URL in order to take any actions.

We can add the REST endpoint, etc. as part of this PR, but that would be significantly larger than the MVP of simply redesigning the UI with the plugin sidebar.

[westonruter]

I think at this stage we don't have the backend infrastructure in place (REST endpoint etc.) to handle those actions dynamically through JS operations

Correct. We have classic WordPress admin requests currently for manipulating validation error states. No REST API endpoints yet. I agree this can be deferred for a later release. Most important now is to improve the presentation of the readonly information.

[westonruter]

Otherwise, I really like the prototype! Just remove the dropdown and checkbox, as @johnwatkins0 said:

This will simplify the implementation because then we don't have to figure out how to present the preview capability as well as saving the state changes for the validation errors, and whether or not these changes include/exclude changes to the edited post as a whole.

[westonruter]

Also, instead of:

I suggest something like:

✖️ Clear block selection

Or maybe “show all errors”.

Also, when a block is selected, let's keep the original icon and not replace it with the checkmark.

So instead of:

Keep always it as:

The selection indicator is redundant anyway, since the list of errors has already been filtered down.

While uncommon, it is actually possible for a block to have validation errors that are coming from multiple sources (plugins, themes). So we should retain the original icon, probably depicting both the source (e.g. plugin icon) along with the kind of error (e.g. HTML, CSS, JS).

[jwold]

I'm running into an issue, and would like feedback. Navigating from the block icon to the isolated error sidebar instance feels straightforward. That's great. I think we've worked out how that would behave. (Example).

However, navigating from the AMP sidebar to the same isolated instance raises some usability issues. I've been thinking of many ways to do this, and wanted to share a few ideas that could work:

Option 1 User opens AMP sidebar in WordPress Editor, sees list of errors, expands an error to learn more, and then isolates the error alongside related errors. A and B show two ways we could do that:

The challenge with this approach is confusion about states. Why would I bother isolating the error when I can just expand it?

Option 2

User opens AMP sidebar in WordPress Editor, sees list of errors, then isolates the error alongside related errors and sees expanded details.

The challenge with this approach is it breaks Gutenberg conventions. Normally you'd expand a sidebar issue.

[jwold]

I just stumbled across this interaction and kind of like it. The initial list of Google flights can be expanded or isolate. Expand on the right, or isolate on the left.

And here's a prototype (not complete, but getting closer) showing how that could work for us:

[johnwatkins0]

@jwold How about if the entire header of a panel (the icons, the headline, and the down arrow) open the error details, and inside the error details is a button/link to see all errors related to that block?

[jwold]

That's a great suggestion. Here's an example of what that would look like. I've added numbers to the screens to help with following the flow.

[westonruter]

I like that. But something I'm realizing is that clicking "Clear block selection" will show all errors but it will also (naturally) remove the selected block from the editor. This may not be desired. So what if clicking "Clear block selection" would not remove the selected block from the editor, but rather just reveal all validation errors. If the user wants to get back to the filtered view of just the validation errors for that block, they can either click the button in the block toolbar, or re-expand a validation error to click that link to just show the errors for that block.

For that matter, perhaps the "See other (2) block errors" link wouldn't be strictly needed because the user could click the button in the block toolbar to get to the same place. Although I suppose having the way to filter the list of errors would be useful.

Did we consider having a selected state applied to the validation errors in the list? If I select a block in the editor, what if any validation errors directly related to that block were also shown as being selected with perhaps some blue outline or something. Any time the selected block in the editor changes, then any errors occurring in that block would get highlighted. When a validation error does have an associated block, then there could be a "Select related block" link when the error is expanded. This would then show block in the editor and highlight any other related blocks as well. If we did that, perhaps we wouldn't need to filter the list down at all? We would have to make sure that the first validation error for the block is scrolled into view, however.

[johnwatkins0]

clicking "Clear block selection" will show all errors but it will also (naturally) remove the selected block from the editor.

@westonruter Do you mean that clicking "Clear block selection" would unselect the block, or that it would delete the block? If the latter, I'm confused and would need you to expand on that.

Did we consider having a selected state applied to the validation errors in the list?

I like this. We could just give the errors related to the selected block, say, a light blue background and not have the filtered view at all.

[jwold]

Did we consider having a selected state applied to the validation errors in the list?

I thought a bit through this direction last week, and believe it could be a valid option!

[westonruter]

Do you mean that clicking "Clear block selection" would unselect the block, or that it would delete the block? If the latter, I'm confused and would need you to expand on that.

Not delete, no. Unselecting the block is the current behavior depicted when "Clear block selection" is clicked. I don't think this should be the case.

I thought a bit through this direction last week, and believe it could be a valid option!

When expanding, there can also be a link to select the associated block, if there is one. If that associated block is then selected, then the appearance would be the same as if the user had clicked the AMP button in the block toolbar.

[johnwatkins0]

If that associated block is then selected, then the appearance would be the same as if the user had clicked the AMP button in the block toolbar.

Great point. Selecting the block is effectively the same as highlighting the block's errors. Block selected === the block errors are highlighted.

[johnwatkins0]

@jwold Another design request for this (not blocking me from moving forward). What should happen when there are no errors? Obviously a "no errors found" type of message, but I suggest something more fun and congratulatory sort of like the final screens of the onboarding wizard. What do you think? No rush -- I can handle this last.

[jwold]

Here's an initial idea we could run with for now. Feedback welcome! cc @johnwatkins0

[westonruter]

There should still be a link to view the validated URL screen as well, since there will be additional information available there, namely the stylesheet data. This link should appear regardless of whether there are errors or not. So this message would be shown below that link. I think perhaps a simple ✅ with simple “Page at permalink is valid AMP” message would suffice.

[jwold]

Oh good point. Agreed!

[johnwatkins0]

@jwold @westonruter About 75% done with this. After working on it a while, I have some suggestions that may warrant one more design round if you think I'm going in the right direction.

Current state of the sidebar below. You'll see I've deviated from the design:

I think we want the sidebar to give the following information in order of importance:

1. Whether AMP is broken at the URL.
2. If AMP is broken, which errors are breaking AMP.
3. Whether there are new errors in need of review and what they are.
4. What errors exist that have been reviewed.

All of this is captured in what I have right now in the screenshot.

1. The broken AMP icon shows when any of the errors at the URL are being kept. (Here's the icon when AMP is not broken: )
2. The kept error is indicated with the text "Invalid script: jquery.js kept."
3. & 4. I have broken Reviewed and Unreviewed errors into two sections. And the number on the plugin icon corresponds to the number of unreviewed errors. I think this is the right approach because there may be errors that have been reviewed and which the user might not want to be bothered with, even if they remain for now. Getting that badge number down to 0 should be easy.

What do you think about this direction?

Other things

Block icon

The icon on the left is the JS/CSS/HTML icon. The icon on the right is the block icon taken from the block settings. This is a departure from the design (where the block icon has a gray background with white SVG paths) because these block icons can be almost anything, so we shouldn't try to manipulate them beyond adding that border and the border radius.

Panel content

I haven't been sure what to include in the panel content:

We should probably include more than this but less than what's on the URL validation screen. Any suggestions?

Orange line

I'm not sure I'm linking this orange line around the block containing an error, because as a new user I might not know what it is:

But I wouldn't fight to remove it. Users will figure out what it is after a little while.

Block toolbar icon

The block toolbar icon (which can't be placed all the way to the left as in the design) works well:

But with my current approach every block has this icon, even if the block has no errors:

I'm not sure that this is a good idea. Should we perhaps only show the icon on blocks that have unreviewed errors?

[johnwatkins0]

Quick addition to the above comment:

I would like to explore creating a small REST endpoint to allow these errors to be marked as reviewed from the editor. As a user, this would allow me to get that badge number down to 0 without leaving.

The REST endpoint can also allow these errors to be marked as Kept/Reviewed.

This can be a small REST endpoint, likely to be adapted into a more full-featured endpoint in the future.