Inserter Constrained tabbing doesn't work in Edit mode

Description

Likely broke in https://github.com/WordPress/gutenberg/pull/32824

The main block inserter is supposed to constrain tabbing with the Tab key within it. However, tabbing is not constrained when the editor is in 'Edit' mode. It only works when the editor is in 'Select' mode.

Note: I think this is not limited to the inserter and potentially affects any component with constrained tabbing that lives in the DOM immediately before (and I guess also after) the Writing Flow component.

Actually, WritingFlow is responsible for this wrong behavior. Specifically, useTabNav kicks in when tabbing away from the inserter and breaks the expected constrained tabbing behavior.

Why it only breaks in 'Edit' mode?

WritingFlow uses two 'focus capture' elements. They are two div elements with a tabindex attribute=0: <div tabindex="0"></div>
Aside: I would like to note that focusable div elements with no semantics / labelling / interaction are less than ideal for accessibility but let's pretend to ignore this for now, as that's not the root cause of this issue.
The tabindex attribute is set only in Edit mode. It is removed in Select mode.
On the other hand, the Inserter constrained tabbing detects what the next and previous 'tabbable' elements are outside of the Inserter.
In Edit mode, the next element happens to be the first 'capture focus' empty div <div tabindex="0"></div>.
When this empty focusable div receives focus, it triggers the writing flow useTabNav behavior, which breaks the constrained tabbing.
More specifically, I think what breaks the constrained tabbing is preventScrollOnTab which was added in https://github.com/WordPress/gutenberg/pull/32824
See the keydown event listener used by preventScrollOnTab. It's a 'global' event attached to the window, which is inherently risky. Things to check here:
- preventScrollOnTab runs and searches all tabbable elements at any Tab key press on any part of the page, which is also a performance concern.
- preventScrollOnTab prevents the browser default action, while constrained tabbing relies on it after https://github.com/WordPress/gutenberg/pull/34836
- When the keydown event target is one of the two 'capture focus' elements, preventScrollOnTab moves focus to the next or previous 'tabbable' element, which is supposed to be the first or last 'tabbable' element within the editor area.
- It appears it's also a matter of timings: at some point, both constrained tabbing and preventScrollOnTab try to move focus to some elements but I'm not sure there's anything that guarantees which runs first and which last.

Step-by-step reproduction instructions

Edit a post.
Observe the editor is initially in 'Edit' mode (see the 'Tools' button in the top bar shows a pencil icon).
Use the Tab key to navigate to the 'Toggle block inserter' button in the top har.
Press Enter to open the block inserter.
Focus is moved within the block inserter.
Use the Tab key to navigate through the controls within the inserter.
Observe that when tabbing away from the last control within the inserter, tabbing is not constrained: focus goes to the post title.
Expected: tabbing to be constrained within the inserter and focus to move to the first control within it.
Observe that, when focus goes to the post title, the editor switches to 'Select' mode (formerly called 'Navigation' mode). See the 'Tools' button in the top bar shows the 'select' icon.
Click the 'Toggle block inserter' button.
Focus is moved within the block inserter.
Use the Tab key to navigate through the controls within the inserter.
Observe that this time tabbing is constrained within the inserter.

Aside: when tabbing within the inserter, at some point focus is on a visually hidden 'Close block inserter' button that shows only its tooltip. This isn't ideal, will create a separate issue.

Screenshots, screen recording, code snippet

To clarify the 'focus capture' behavior:

The two 'focus capture' elements in Edit mode, with a tabindex=0 attribute:

The two 'focus capture' elements in Select mode, with no tabindex attribute:

No response

Environment info

No response

Please confirm that you have searched existing issues in the repo.

Yes

Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

WordPress / gutenberg