fluid-lab / gamepad-navigator

GSoC 2020 project
Other
6 stars 10 forks source link

Add ability to interact with content from external iframes... #155

Closed duhrer closed 4 months ago

duhrer commented 5 months ago

It looks like ally.js does support looking for focusables in iframes. Basically, you pass the iframe as the context.

Probably the solution here is to update the "update tabbables" method to look for content within iframes and interleave that with the focusables in the body of the document. For that to work, we'd need to come up with a way to figure out where the iframe should appear in the list of focusables, and place the focusables for the iframe in the right spot.

As a sanity check, we need to first confirm whether there are options simply using ally.js with different options, for example, using the "all" focusable detection approach that ally provides. Some testing is needed.

If that doesn't pan out, one sloppy way to do this might be to:

  1. temporarily make any iframes focusable that were not already focusable, keeping track of whether they were
  2. calculate the tabbables using ally.js
  3. Replace each iframe element with its focusable children
  4. make any iframes that were not focusable before non-focusable.

A cleaner approach would be to:

  1. Find the normally focusable items, possibly without sorting by tabindex
  2. Find the iframe elements
  3. Order all by tab index using something similar to the previous "tabindex" sort function.

In any case we'd need a good test fixture for this and also to review the handful of outliers that calculate tabbables themselves (the modals, I think?).

duhrer commented 5 months ago

Another option is to replace ally.js outright with our own code that includes iframes in the initial list of items to inspect, and then expands those as needed.

duhrer commented 5 months ago

We might also order them by onscreen position by default, using first the vertical and then the horizontal position. This could be used as a starting point for the work on smarter scrolling (#124) and a simulated mouse pointer (#139).

We'd then reorder the list based on any tabindex values that shift position.

duhrer commented 4 months ago

Looks like the case where I encountered this (embedded vimeo videos) highlights that this is yet another CSP issue.

An iframe within the same domain will have its contentDocument property populated, and we can interrogate that to find focusable elements.

An iframe that points to another domain will have null for that property, and no further information is available from the containing document (where our code lives).

The src property of the iframe is a page that can be opened in a new window, so we can make the iframe itself focusable and allow the user to navigate to the destination as though we were following a link.

It seems like we can't focus on an iframe itself, so we'd need to wrap the iframe in our own focusable element, which should include some kind of tooltip or other hint to indicate that clicking it will open the content .

duhrer commented 4 months ago

I just confirmed with the dev console that I can:

  1. create a wrapper element using something like createElement("div").
  2. Position the wrapper by appending it inside the parent of the iframe, which is its parentElement.
  3. Move the iframe inside the wrapper using append.

Another thought I had is to place a sibling before the iframe that's just a link. This would be simpler in some ways, but I'm afraid it would break up the look of the page too much, especially when there is a tight layout surrounding the element.

Combining these approaches, we could wrap the iframe in an anchor tag that:

  1. Is styled as a block element.
  2. Points to the src of the iframe
  3. Has an alt attribute that indicates that clicking will open the iframe in a new window.

On that last point, if the iframe has a title, we should use that as part of the alt attribute.