Pre-proposal: standardize setting alt text from `alt` metadata in display_data

[minrk]

Problem:

alt text is required for image accessibility, but there's currently no documented way to set alt text on displayed images.

Proposal:

populate alt attribute from the alt metadata field in display_data messages, if defined. Alternative: use a non-standard but more specific name, such as html-alt, or alt-text.

There are a few places this info can come from, such as ImageDescription EXIF data, but we should have a protocol-level place for users and libraries to be able to easily populate these fields.

Another alternative comes from the fact that the Jupyter protocol mime-bundles already are multiple representations of the same object, and should always include plain text. So another valid choice is for User Interfaces (JupyterLab) to populate the alt text for images with the text/plain representation of the same output. However, in practice that is very rarely a good choice for alt text for an image, since it describes the displayed object, not the image itself. I'd need to hear from accessibility folks if that's better or worse than no alt text at all. But maybe this should lead us to make better text representations in text/plain, too!

While using the text/plain output may be a reasonable default when alt is unspecified, I still think it makes sense to pass through the alt metadata so that users and libraries can more easily set alt on the destination element, regardless of what we choose for possible sources of default values in the HTML.

Still to iron out:

• example or API for how to get alt text for e.g. matplotlib figures. Computing alt text for figures is tricky, and not likely to be a property of the Figure object itself, so we should show an example of what it might look like, e.g. with the work here

Affected packages:

• image publishers (ipykernel, other kernels)
• notebook renderers (JupyterLab, nbconvert)

references:

• https://courses.cs.washington.edu/courses/cse493e/23au/projects/matplotalt.html

Suggested reviewers:

• @isabela-pf
• @gabalafou

[bollwyvl]

Great stuff!

alt is a fine starting point, and the assistive technology can use all kinds of things from the aria-* attribute family with the biggest impact being aria-role.

Specifically for images, things get more complex: a Figure can be mapped to a <figure>, which can optionally have a <caption>, which is distinct from alt (more explain, less describe).

Having a robust, schema-constrained data model for this content in the Jupyter Messaging Spec would be a great way forward, but ideally it would come from somewhere else that already had tools, documentation, and test suites to run against.

:bell: @tonyfast

[tonyfast]

great stuff. i'm glad there is momentum for things like this. i agree this should be standardized, but i don't know if the nbformat is the place for it.

like @bollwyvl says this is a good starting point, and, also, a can of worms. alt creates an aria-label, sometimes the title attributes does this, or maybe caption, or figcaption. that said the accName computation is complicated. i really do believe that we should have supplementary schema for features like alt, title.. its important to consider that when we go in this direction we start to say that this schema is for accessible notebook documents, a union of whatwg HTML5 and nbformat. thats a lot to take on, but someone should eventually.

i could imagine a supplementary schema that says something like:

[properties.cells.items.properties.outputs.items.metadata.properties."image/png"]
type = "string"
required = true

[properties.cells.items.properties.outputs.items.metadata.properties."image/jpeg"]
type = "string"
required = true

[properties.cells.items.properties.outputs.items.data.properties."text/markdown"]
description = "check for empty markdown links"
type="string"
pattern = ".*\[\s*\](\S).*"

so now the question is what if this fails for lack of alt text? does the notebook save with a failure? can we not save until we satisfy the schema. i think the point is that notebooks need to be written accessibility for the schema validate any aria labels. generally, it is really hard to do accessible things in data models others than html because that is where WCAG is defined.

@blois introduced changes for "specifying alt text in for IPython.Image", but it still seems that generally the alt text rarely makes it to html including lab and nbconvert. i think this approach is a good starting point and we should try to see that propagates to the html where it can be measured.

• example or API for how to get alt text for e.g. matplotlib figures. Computing alt text for figures is tricky, and not likely to be a property of the Figure object itself, so we should show an example of what it might look like, e.g. with the work here

in the context of authoring a notebook, inverse models for inferring alt text could be a distraction. in a notebook, an author is going to know the context of their figures as preliminary objects like arrays and dataframes. when we have live states of data we can create better forward models for generating alt text rather guessing with inverse models on lossy data representations.

matplotlib figures with subplots really stretch the idea of encoding alt text or how it is encoded. the area element can be used to have multiple alt text annotations for the figures. for these use cases we'd have to encode shape information alongside any alt text in the schema.

[minrk]

Thank you! I thought I had seen something about this before, but was looking at frontends instead of ipykernel.

A supplementary schema for accessible documents is a really great and interesting idea. Do you think that's where we should start, or is a smaller scope appropriate for a first pass?

Figure vs Image is a good question, and we could open that here, or we could consider that separately. I think it's a valid choice to say that frontends should or may display image outputs as <figure>, e.g. when certain metadata is present. It's not clear to me what that should be (e.g. image/png with metadata figure: true vs a separate figure.jupyter.org/v2+json mime-type with its own schema (🤞 for existing figure jsonschema we could use)).

The scope of what I originally aimed for here was encoding what are typically accessibility attributes on an img tag, e.g. alt, title, aria-* that can be understood as straight pass-throughs for Image -> html img attribute (i.e. if you set metadata field X, we set HTML attribute X on the resulting img element). Is that naïve or a potential impediment to a more comprehensive approach in the future? Should we namespace these as specifically html_attrs in metadata? My first thought is that alt is well understood enough (title less so), and aria- are specific enough that they can be defined clearly at the top level and wouldn't cause trouble. I think it's also appropriate to acknowledge that notebooks, while not exclusively HTML, really are HTML-first, so inheriting definitions from HTML for things like title that may be unclear in a hypothetical situation absent any other context, is still sensible and reasonably clear in our situation.

For the matplotlib plots, I did not mean to pick them to start a discussion of how to compute alt for charts, but rather as an example where kernels register the display function that produces an image (the matplotlib inline backend), but the alt text is likely to come from somewhere else (e.g. directly written by the author). i.e. what does it look like in ipykernel to display a figure with user-created alt text? So assume the author has the alt text and Figure, how do they publish them together? This is less about matplotlib specifically, and more about attaching author-controlled alt text to something that may already have an image repr, to make sure it's manageable for authors without needing to re-register custom display formatters for everything.

The area point does suggest that perhaps the metadata structure should be quite a bit more complex. My initial thought on the area approach is that it should be outside the Jupyter protocol (at least for now), and rely on html outputs, which can include all the alts and other fancy things, etc. as-is without the protocol or frontend needing to understand them.

[tonyfast]

i.e. what does it look like in ipykernel to display a figure with user-created alt text? So assume the author has the alt text and Figure, how do they publish them together? This is less about matplotlib specifically, and more about attaching author-controlled alt text to something that may already have an image repr, to make sure it's manageable for authors without needing to re-register custom display formatters for everything.

right now, software and accessibility wise we are missing this. there is no recommendation for including alt text because there is no implicit way to do it. we really need this. nothing really seems to work right and it should. in the near term alt needs to wind up in the text/html output with the matplotlib data or the caption in a pandas table. i don't know if schema is going to fix the author's responsibilities, and it is hard to own that without accepted accessible authoring practices.

A supplementary schema for accessible documents is a really great and interesting idea. Do you think that's where we should start, or is a smaller scope appropriate for a first pass?

i personally would choose trying to extend the schema. we've introduced two JEPs that focus on extensible schema for the notebook format and rallying around those would be a good experiment.

cf https://github.com/jupyter/enhancement-proposals/pull/108 cf https://github.com/jupyter/enhancement-proposals/pull/97

Figure vs Image is a good question, and we could open that here, or we could consider that separately. I think it's a valid choice to say that frontends should or may display image outputs as <figure>, e.g. when certain metadata is present. It's not clear to me what that should be (e.g. image/png with metadata figure: true vs a separate figure.jupyter.org/v2+json mime-type with its own schema (🤞 for existing figure jsonschema we could use)).

i always caution measuring accessibility outside of html, or removing the image from its context. ARIA10:Using aria-labelledby to provide a text alternative for non-text content is a technique for alt text that relies on the document. i know i keep giving reasons not to, but i think that is a scope creep potential for putting accessibility directly in the notebook format. it is an entire world in itself. i think that nbconvert-a11y could potentially take on some of these schema concerns though.

I think it's also appropriate to acknowledge that notebooks, while not exclusively HTML, really are HTML-first, so inheriting definitions from HTML for things like title that may be unclear in a hypothetical situation absent any other context, is still sensible and reasonably clear in our situation.

i'd really love for this to be more explicit, it is still implicit. we should have assistive notebook representations that generated tagged pdf. then we want some accessible epub. our goal is the same nbconvert targets all along, but accessible. that requires retrofitting accessibility which has proved very challenging in jupyter. i think outside of immediately improving the quality of html output we should be adding testing like axe and the w3c/validator to more pipelines; this is the robust part of wcag. i think the schema will become more clear when we have testing infra around our accessibility.

[blois]

Also see discussion in https://github.com/ipython/ipython/pull/12864

I would also love to see an easy/automatic way for alt text from matplotlib images.

[minrk]

in the near term alt needs to wind up in the text/html output with the matplotlib data or the caption in a pandas table....i don't know if schema is going to fix the author's responsibilities, and it is hard to own that without accepted accessible authoring practices.

I don't view this as taking any responsibility from the author. I think computing the accessible info is going to remain their responsibility (and in some cases where possible, a library). I mainly want to get to the point where if an author has the accessibility information, they can publish and it will be displayed. For alt, @blois has added the API already, and I want to acknowledge the metadata it in the frontends.

I think the one Jupyter-specific point is that all of our displayed images actually do come with text representations in the text/plain output. So a potential default implementation is to use the text/plain representation as alt text (in the absence of other explicit alt text). However, this is a representation of the displayed object, not the displayed image, so might be worse than not having alt text. I don't know enough about the relevant topics to say if that's good or not.

i personally would choose trying to extend the schema.

To be clear, do you mean adding to the existing nbformat schema, or defining an extended "accessible notebooks" schema, which derives from the existing one?

How is this for a proposal:

• define an "accessible notebook" schema. Perhaps its origin/development should be in jupyter/accessibility, or nbconvert-a11y
• start the new schema which extends the notebook schema and starts with a definition for the alt field in metadata
• add the schema to https://github.com/jupyter/schema
• support alt metadata in frontends as a first step

then we have a place to advance the accessible outputs schema going forward, owned by the experts in the area, and are starting with the low-hanging fruit of handling the alt metadata @blois has already added in ipykernel. How many of these things does nbconvert-a11y already do?

My original plan was just the last bullet point, but it seems like having a dedicated place to grow more of these metadata definitions can facilitate things going forward.

• advantage of separate schema: accessibility work can proceed without being slowed down by Jupyter as a whole
• disadvantage: by not being in base nbformat, risks less uptake by implementations and perception of reduced importance

i'd really love for this to be more explicit, it is still implicit.

I agree. In this context, I mean it mostly in terms of naming fields, e.g. when we need to pick our own key for "text that can be substituted for an image," we use alt because HTML defines it. As much as possible, I like to appeal to authority in naming things, so when folks know one thing, they know both. I'd defer to the accessibility team on what standards and fields those should reference.

we should have assistive notebook representations that generated tagged pdf

This is where things go way beyond my understanding, but presumably will inform the schema development. How much can an accessible PDF be generated from what is assumed to be already accessible HTML? Or is this only feasible with higher-level structured representations of an image and metadata?

i think outside of immediately improving the quality of html output we should be adding testing like axe and the w3c/validator to more pipelines; this is the robust part of wcag.

Yes, that would be great!

[tonyfast]

the proposal you put forth sounds great. we can work on schema in salient projects with the end goal being the jupyter schemas repo. conveniently, the front end work on accessible image descriptions concurrently while these other activities are happening.

How many of these things does nbconvert-a11y already do?

nbconvert-a11y just started taking on outputs. it is nascent work with the goal of providing more semantically meaningful outputs for dataframes and arrays.

the main contribution of nbconvert a11y so far is the reference template for the notebook that uses semantic html5 to construct an ideal representation for assistive technology. it could be used as implemenation, but is primarily a reference for testing out accessible designs. here is an example of the lorenz notebook with outputs.

• advantage of separate schema: accessibility work can proceed without being slowed down by Jupyter as a whole

i think another advantage is that schema work can be introduced into testing pipelines of projects trying to adopt this enhancement. this is another potential mode for uptake.

As much as possible, I like to appeal to authority in naming things, so when folks know one thing, they know both. I'd defer to the accessibility team on what standards and fields those should reference.

my suggestion for this key with accName. all roads lead to accName, title alt caption figcaption. this choice dovetails nicely into further descriptions with accDesc.

How much can an accessible PDF be generated from what is assumed to be already accessible HTML?

the theory is some accessible html could be made in accessible pdf with pdf.js, but we still need to experiment and test.

[KaiNylund]

Hi all, I'm the developer of the project mentioned in the first comment, but definitely not an expert on Jupyter.

Rather than transferring over complex html attributes, is there a way to render matplotlib outputs or other figures in html by default? (also see #14482) For instance, the way matplotalt renders figures together with alt text is by displaying the base64 encoded image in an <img> tag:

import matplotlib.pypplot as plt
from PIL import Image
from IPython.display import display, HTML

# Get matplotlib figure as a base64 string
fig = plt.gcf()
fig.canvas.draw()
pil_img = Image.frombytes('RGB', fig.canvas.get_width_height(), fig.canvas.tostring_rgb())
data_url = "data:image/png;base64," + pillow_image_to_base64_string(pil_img)

# stop matplotlib from rendering figure
plt.close()

# render as html with alt
display(HTML(f'<img src="{data_url}" alt="{alt_text}"/>'))