Accessibility

[VisenDev]

It might be worthwhile to consider whether or not we want to support accessibility functionality. Accessibility usually only gets considered very late into open source development as an afterthought. We should consider

• a) Do we want to support any accessibility features?
• b) What features do we want to support?
• c) What is the pathway to properly integrate those features into dvui?

[david-vanderson]

Excellent questions. Accessibility functionality is very important, and we will most certainly work towards support in dvui.

To be clear, because dvui is a small, new, open-source project, the features and functionality that are added come from people showing up, asking for them, and being willing to help.

Here's what I currently understand about accessibility-related functionality:

• Language Support
  • dvui currently supports left-to-right single-glyph stuff (very simple layout)
  • need multi-character glyph for some languages and emoji
  • need right-to-left support for some languages
  • do we try to support mixed (left-to-right and right-to-left) in the same context?
  • these affect rendering, cursor/selection logic and require shipping unicode tables and bigger fonts
• Language Input
  • dvui has some pieces in place to support IME (Input Method Editor) support, but untested so likely missing something
• Visual Output
  • need a high contrast theme (do we pair it with large fonts or different fonts?)
  • screen magnification is hopefully handled by the OS, as well as cursor size
• Audio Output
  • need to produce a text description of everything (for sending to an OS text-to-speech thing I hope)
  • this interacts with focus and scrolling
• Keyboard Input
  • dvui supports keyboard navigation and interaction, but we are likely missing it in some places

Most of these are going to require separate work per backend, and I expect support will vary by backend.

My plan here (like most things), is to wait for someone to show up who understands one of these well and can help us figure out the requirements and help test solutions.

To anyone reading this - if I forgot to list something please reply so I can add it.

[david-vanderson]

As a concrete example, making a high contrast theme is something we could implement quickly. But how would we know if we did the right thing?

So we need someone who knows what a high contrast theme (themes?) should be, and can help us test and tweak it. We don't need perfect requirements, but personally I don't know enough to do this satisfactorily.

[VisenDev]

As a concrete example, making a high contrast theme is something we could implement quickly. But how would we know if we did the right thing?

Initially, we could work on copying an existing high contrast color theme, like https://github.com/agude/vim-eldar

Audio Output need to produce a text description of everything (for sending to an OS text-to-speech thing I hope)

Yes, this is one of the main things I was thinking of. It will probably take some work figuring out how to integrate this properly with operating systems.

I am the most familiar with this type of accessibility because one of my younger brothers is dyslexic and screen readers have been a big help to him.

We could also look into adding fonts like OpenDyslexic. Actually this should be fairly quick to add, I'll try to work on that once my current pr is done.

[david-vanderson]

Great! Copying an existing theme sounds good (especially because you are now the theme expert).

For text-to-speech it's a relief to have someone familiar with it. Can you open an issue dedicated to that and put whatever you currently understand about it there?

Thanks!

[JosephABudd]

As a side note. I really miss Opera version 9 which implemented VXML. I wrote my cms web site for it. Each web page was a 3D colletion of information. I wrote my browser application which read my favorite web sites to me as those 3D objects (not as a linear stream of html like dragon did). Those were the days.

[VisenDev]

OS Specific TTS integration

• Mac
  • We can use Speech Synthesis for the speech generation.
  • We should be able to somehow query whether the user has a screenreader enabled using this
• Windows
  • Windows has a built-in screen reader called narrator, but I can't figure out how to integrate that into custom apps (maybe its DOTNET specific?). So alternatively we could also use the TTS api
  • Linux: accessibility support is a bit of a mess as far as I can tell. This will take some research to figure out how to properly do TTS

[david-vanderson]

I'm watching https://www.youtube.com/watch?v=Jf0cjocP8Wk and just found out about https://webaim.org/resources/contrastchecker/ which might help with deciding how much contrast is enough for normal and high contrast themes.

[iacore]

About IME -- fcitx (ibus compat)

• dvui has some pieces in place to support IME (Input Method Editor) support, but untested so likely missing something

Our SDL works (missing system font):

https://github.com/user-attachments/assets/f08179be-e8b7-4ed0-9a85-582dcea0fd18

Our raylib doesn't. I tested it. IME input events are treated as normal key input events.

glfw is stuck in time.

https://github.com/raysan5/raylib/issues/1945 https://github.com/glfw/glfw/pull/2130 <- Unless we use this forked version of GLFW, our raylib backend is not going to get IME support now.

[iacore]

Talking about fonts, we need a set of "system UI fonts" to fallback to. Firefox has their own list of fonts. On Linux, fc-list can be used to query which font supports which codepoints. https://www.1a-insec.net/blog/11-fonts-where/

[david-vanderson]

Thanks very much for trying this!

I can see from your video that the IME temporary window is nowhere near the dvui textEntry. I've filed https://github.com/david-vanderson/dvui/issues/132 for this.

I'm less sure how to deal with fonts, but I think you are right that we'll have to connect to some system fonts somehow, while keeping the ability to have only compiled-in fonts present. I've filed https://github.com/david-vanderson/dvui/issues/133

[BratishkaErik]

There's also AccessKit with generated C bindings, although it's written in Rust and sadly I don't know how to integrate cbindgen in build.zig.

[david-vanderson]

That looks promising!

[iacore]

There's also AccessKit with generated C bindings, although it's written in Rust and sadly I don't know how to integrate cbindgen in build.zig.

VoiceXML was literally better. This need us to build a document tree too.

[iacore]

I think we will just use AccessKit.

I looked into IME support as well.

dbus -> AT-SPI dbus -> ibus dbus -> fcitx

dbus is at every point.

The SDL implementation is here: https://github.com/libsdl-org/SDL/tree/main/src/core/linux

Maybe we can copy SDL's code into dvui.

About dbus libraries, there are currently two.

• libdbus (the c library)
• zbus (the Rust library)

The complexity of DBus is really ..., and we are stuck with it. See busctl --user.

AccessKit and SDL deals with DBus for us.

Further readings:

• Using busctl to send DBus message: https://a.exozy.me/posts/switch-virtual-keyboard/
• DBus type system: https://dbus.freedesktop.org/doc/dbus-specification.html#type-system
• Tutorial using dbus in C: https://www.linuxjournal.com/article/7744

[DataTriny]

Hi, AccessKit dev here. Happy to see this project considering accessibility seriously.

Since dvui appears to mostly use SDL as its windowing layer, here is an example app showing how to integrate AccessKit C bindings with SDL.

I am not aware of any project written in Zig currently integrating AccessKit and I don't know if our C bindings would be enough for you, but I can help if dedicated Zig bindings are needed.

You might also want to look at egui, an immediate mode UI toolkit which uses AccessKit.

[david-vanderson]

Thank you very much! That's very helpful, I'll take a look at those.

I'm pretty new to this stuff. Can you recommend what a first-step integration goal would be? Would it be to read a description of a single UI element?

[DataTriny]

@david-vanderson If you want to be able to inspect the accessibility tree, you will first want to hook AccessKit into your windowing provider. The example program I linked above should contain all you need for this.

After that, you will need to come up with a way to identify your UI elements: these IDs must be stable across frames. You can then try building accessibility nodes with accesskit_node_builder, populating it with the information you already have. Collect all the nodes and push them into an accesskit_tree_update. If you already support keyboard navigation, you can then include this in the tree update.

We are still figuring out how to best document AccessKit. The C header file doesn't contain much so the Rust documentation will be more helpful. Most of the node's properties come from the ARIA web standard so you can also use this specification as a reference.

[iacore]

egui code for filling accesskit node info from widget info: https://github.com/emilk/egui/blob/7bd6f83f18af7317f75e7a32413a9a9eb747db7f/crates/egui/src/response.rs#L1010

---

@DataTriny What's the minimal code in C that constructs the following tree and send it to accesskit?

• container A, bounds ...
  • button B, text "B", bounds ...
  • label C, text "C", bounds ...

What's the code if it changes to

• container A, bounds ...
  • button B, text "B", bounds ...
  • label D, text "D", bounds ...
  • label C, text "C", bounds ...

How do I submit the changes? Do I need to calculate the delta or does accesskit do it and announce it to the screen reader?

It would be nice if you can provide us with this code in C or Zig so we can use it as a starting point.

---

I am not aware of any project written in Zig currently integrating AccessKit and I don't know if our C bindings would be enough for you, but I can help if dedicated Zig bindings are needed.

Yes, it's mostly the Zig-specific pointers that provide ergonomic. We will likely have our own Zig extern fn definition file (like C header), so we will need to cooperate on making the ABI stable. This one file will probably live in the dvui repo first, and later move to the accesskit repo once stable.

---

Question about the build system.

accesskit-c need CMake and Cargo to build. Is it better for accesskit to ship the prebuilt binaries or how are we going to build it?

[DataTriny]

Tree updates

The very first tree update is a bit special as you must provide information about the root window and possibly the name of your application. It is not shown here, please consult the SDL example program linked above to see how it is done.

Second tree update (could be part of the first tree update if the whole UI is available as soon as the window is shown):

// Giving the container an ID of 1, assuming 0 refers to the root window.
accesskit_node_id CONTAINER_ID = 1;
accesskit_node_id BUTTON_B_ID = 2;
accesskit_node_id LABEL_C_ID = 3;

accesskit_node_builder *container_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_GENERIC_CONTAINER);
accesskit_node_builder_set_bounds(container_builder, ...);
accesskit_node_builder_push_child(container_builder, BUTTON_B_ID);
accesskit_node_builder_push_child(container_builder, LABEL_C_ID);

accesskit_node_builder *button_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_BUTTON);
accesskit_node_builder_set_bounds(button_builder, ...);
accesskit_node_builder_set_name(button_builder, "B");

accesskit_node_builder *label_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_LABEL);
accesskit_node_builder_set_bounds(label_builder, ...);
accesskit_node_builder_set_name(label_builder, "C");

// The first update would need to include information about the root window, it is not shown here.
// We always have to give information about the focus. Here we put it on the button, but it could be set to the root window if we don't have something better.
accesskit_tree_update *second_update = accesskit_tree_update_with_capacity_and_focus(3, BUTTON_B_ID);
accesskit_tree_update_push_node(second_update, CONTAINER_ID, accesskit_node_builder_build(container_builder));
accesskit_tree_update_push_node(second_update, CONTAINER_ID, accesskit_node_builder_build(button_builder));
accesskit_tree_update_push_node(second_update, CONTAINER_ID, accesskit_node_builder_build(label_builder));
// second_update is now complete and can be pushed to a platform adapter.

Third tree update, if you can't persist any state:

accesskit_node_id LABEL_D_ID = 4;

accesskit_node_builder *container_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_GENERIC_CONTAINER);
accesskit_node_builder_set_bounds(container_builder, ...);
// The order in which children IDs are pushed is the order in which they will be exposed in the tree.
accesskit_node_builder_push_child(container_builder, BUTTON_B_ID);
accesskit_node_builder_push_child(container_builder, LABEL_D_ID);
accesskit_node_builder_push_child(container_builder, LABEL_C_ID);

accesskit_node_builder *button_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_BUTTON);
accesskit_node_builder_set_bounds(button_builder, ...);
accesskit_node_builder_set_name(button_builder, "B");

accesskit_node_builder *label_d_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_LABEL);
accesskit_node_builder_set_bounds(label_d_builder, ...);
accesskit_node_builder_set_name(label_d_builder, "D");

accesskit_node_builder *label_c_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_LABEL);
accesskit_node_builder_set_bounds(label_c_builder, ...);
accesskit_node_builder_set_name(label_c_builder, "C");

accesskit_tree_update *third_update = accesskit_tree_update_with_capacity_and_focus(4, BUTTON_B_ID);
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(container_builder));
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(button_builder));
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(label_d_builder));
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(label_c_builder));
// third_update is now complete and can be pushed to a platform adapter.

AccessKit will compute the diff between the two updates, only add or remove nodes as necessary and emit the appropriate events.

Alternative tree update if you can retain some state, this is obviously more efficient but not practical for immediate mode UI toolkits:

accesskit_node_id LABEL_D_ID = 4;

// Since container C's children have changed, we must re-create it.
accesskit_node_builder *container_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_GENERIC_CONTAINER);
accesskit_node_builder_set_bounds(container_builder, ...);
// The order in which children IDs are pushed is the order in which they will be exposed in the tree.
accesskit_node_builder_push_child(container_builder, BUTTON_B_ID);
accesskit_node_builder_push_child(container_builder, LABEL_D_ID);
accesskit_node_builder_push_child(container_builder, LABEL_C_ID);

accesskit_node_builder *label_d_builder = accesskit_node_builder_new(ACCESSKIT_ROLE_LABEL);
accesskit_node_builder_set_bounds(label_d_builder, ...);
accesskit_node_builder_set_name(label_d_builder, "D");

// We can still put the focus to button B even if it is not in the update, AccessKit still remembers it.
accesskit_tree_update *third_update = accesskit_tree_update_with_capacity_and_focus(2, BUTTON_B_ID);
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(container_builder));
// Button B and label C haven't changed, we don't need to include them in the update.
accesskit_tree_update_push_node(third_update, CONTAINER_ID, accesskit_node_builder_build(label_d_builder));
// third_update is now complete and can be pushed to a platform adapter.

Build system integration

I currently know nothing about typical Zig build systems but we offer some pre-built binaries here. They are generated in our CI so you can count on them being available. We recently extracted the bindings into their own repository, which explains why there are so few releases in there.

We also plan to add support for Meson, I don't know if that would be of any help.

I can also point you to the work-in-progress integration of AccessKit into the Godot game engine. They are currently using our pre-built binaries as well.

[david-vanderson]

Thank you for the detailed explanation and code samples!

[iacore]

The SDL example contains platform specific code. https://github.com/AccessKit/accesskit-c/blob/48061861ebe14873a29b65488728ce2a1160f72b/examples/sdl/hello_world.c#L72-L92

[DataTriny]

There is just a bit more platform-specific code @iacore, but it is nothing compared to what you would have to write yourself if you decided to implement accessibility for each platform from the ground up, the stacks are quite different.

We do our best to keep the core schema platform neutral; the way you build AccessKit nodes and tree updates is completely decoupled from the platform.

[iacore]

Could this be a library, or is it like platform glue code that each project have a copy of and customize it? I'm not sure how to treat it.

I think maybe accesskit-c could use a backends/ folder like imgui: https://github.com/ocornut/imgui/tree/master/backends

[DataTriny]

AccessKit is an entire library, it's just that we don't provide integrations for popular windowing providers such as SDL, SFML or GLFW. We officially support the most popular Rust windowing provider though, which you can find in the platforms/winit directory of the main repository.

If people show interest in using AccessKit with windowing providers written in other languages than Rust, maybe we will spend time building such components. Help from the dvui team would be welcome!

[Paul-Dempsey]

• do we try to support mixed (left-to-right and right-to-left) in the same context?

Pretty much a hard requirement. There are no pure right-to-left writing systems. All real-world usage is bidirectional (BiDi). In Arabic and Hebrew, numbers are most commonly "western" digits (Arabic-Indic, technically) and written left-to-right embedded in the text. Same with foreign words used in a sentence, which are written in their native directionality.

Of course, BiDi and IME are Internationalization, and not Accessibility, the subject of this issue. Really everything under "Language Support" is mostly orthogonal to accessibility.

[DataTriny]

I agree that this issue's scope is probably too big. It should be turned into a tracking issue instead.

If the project's maintainers agree that AccessKit is the answer to the accessibility tree part, I can open a dedicated issue for that where this topic can be discussed.

[david-vanderson]

If the project's maintainers agree that AccessKit is the answer to the accessibility tree part, I can open a dedicated issue for that where this topic can be discussed.

Yes please - that would be wonderful. Could you put in the description what pieces of accessibility integrating with AccessKit will help with? Is it more than screen readers? Thanks!

[DataTriny]

@david-vanderson I've filed #151. I haven't expanded on actionable steps yet, but I have described the motivation. I've elaborated a bit on what assistive technology means. If you didn't know, the automation testing aspect might interest you as implementing AccessKit would mean that dvui users could rely on existing testing software for the QA of their apps.

[david-vanderson]

Thank you!