uswds combo box with well over ten thousand `<select>` options

[dwinston]

I've used the component code at https://designsystem.digital.gov/components/combo-box/ for a related project (see here).

The corresponding page, i.e. https://helioweb.polyneme.xyz/funnel_authors, is slow to load, and interactive selection of co-authors is high-latency. I assume this is because e.g. I load over 18,000 <option>s into eachselect[name="coauthor"] element.

@cferdinandi can you help me in coming up with a solution to this that (1) allows continued use of the uswds design system (as you've done for this project), and (2) does not sacrifice accessibility? I imagine this will be some progressive enhancement that fetches partial matches via debounced ajax requests.

"as soon as possible" would be highly appreciated, as https://helioweb.polyneme.xyz is "live" for a conference this week.

[dwinston]

Also, this work will absolutely be relevant for this project. example: the concept scheme I am using for helioweb has 65,073 concepts, which I reduced to 4,423 by only including concepts currently associated with authors in the backing database. I definitely need to be able to quickly select concepts from very large concept schemes.

[cferdinandi]

Ohh neat! Is this different from the project we've been working on?

You have two options, generally, with a pattern like this:

1. Load the "combobox" UI by default. Don't populate the dropdown options until the user starts typing. You do this by querying some type of search API that dynamically returns results.

2. Load a default input empty text field by default. Asynchronously get the list of options with JavaScript, and "upgrade" the UI once it's available.

Based on what I know about the codebase and existing setup, I think option 2 is probably the best path forward, but there's also likely stuff I don't know. Option 2 would also be the fastest to implement.

[dwinston]

The other project should eventually have this project as its “basis”, but I wanted to get something up to demo this week.

For option 2, do you mean to get the list once, based on, say, a minimum of two (three?) letters typed in, hoping the list is never more than a few thousand options, and then load that in? Or is there dynamic re-fetching?

[dwinston]

I better understand option 1 at the moment, but I’m interested in understanding a faster-to-implement option. :)

[cferdinandi]

@dwinston For sure, so with option 2 the initial markup looks like this...

<label for="first-concept">First Concept</label>
<input type="text" id="first-concept" name="first-concept">

After the page loads, a JS fetch() call will request the full list of items to use for the combobox. Once it's returned, then the JS will update the UI to include that hidden <select> menu that provides the typeahead functionality you have today.

This provides a functional UI quickly, and progressively enhances it into something better once available.

With Option 1, the markup you have now is there from the start, but without any options in the <select> menu. Those get populated live as the user types.

It can work, but requires a fast-responding API to back it up. I'd also need to research if there's anything needed with the design system to make sure it works properly.

[dwinston]

I see. So option 2 would indeed put all (tens of thousands of) options in the DOM. I think that would be too much for a typical user's browser-allocated RAM, no? So it seems like option 1 is the necessary approach for gigantic sets of options?

If that's right, then please go for option 1 @cferdinandi .

I can try to provide a fast-responding API, but I think it would also be prudent to have a (re)loading indicator when fetching new options based on the current value state of the user's text input.

[dwinston]

perhaps related: would a <datalist> be helpful for this kind of thing, where there may be multiple select elements on a page with the same options?

[cferdinandi]

@dwinston Can users input any value, or only from a pre-defined list of items.

[dwinston]

only from a pre-defined list of items.

this.

[cferdinandi]

@dwinston Ok, in that case, we probably want some type of search + select component where we prevent users from submitting until they've selected a valid item. There's a bit more accessibility work around this as a result. Pushing it live this week will probably be challenging, but we can try!

[dwinston]

sounds good. thanks!

[cferdinandi]

@dwinston Is there a codebase I should be looking at for this? An API structure I can use for reference?

[dwinston]

@cferdinandi I just pushed a reference demo for this at /concepts-search. 20s demo video here.

[cferdinandi]

@dwinston Cool, I'll see what I can put together. This particular component might benefit from a a more traditional JSON-based API rather than HTML responses... but maybe not!

I'll see what I can come up with.

[cferdinandi]

@dwinston Initial findings...

1. The USWDS combobox can't be used for this. Because of how it loads, you can't dynamically update options later. We'll have to code up our own. I can do that.

2. I noticed that if you type just 1 character, you get back tens of thousands of options and the page crashes. Any thoughts on how to add some safety rails so that this doesn't happen?

   My gut reaction was to require a minimum of several characters before you search, but that could be artificially limiting as well.

[dwinston]

1. The USWDS combobox can't be used for this. Because of how it loads, you can't dynamically update options later. We'll have to code up our own. I can do that.

Darn. Thanks!

2. I noticed that if you type just 1 character, you get back tens of thousands of options and the page crashes. Any thoughts on how to add some safety rails so that this doesn't happen?

I just pushed a change to truncate the response to at most 50 concepts.

[cferdinandi]

@dwinston Awesome, thanks. I should be able to wrap this up tomorrow.

[cferdinandi]

@dwinston I've got a working branch setup at https://github.com/polyneme/heliokos/tree/typeahead-datalist

This...

1. Let's you configure the endpoint to use and length of delay (with a default of 500ms)
2. Calls the API and dynamically updates a list of options
3. Sets a hidden field with the ID while visually displaying the text version
4. Validates that the provided text is in fact one of the allowed values

One change I'd recommend: have the API return a JSON object, an array of options with the id and value. This will decouple the data from the UI display, and make it easier to update how the content is rendered later if needed.

[
    {
        "id": "1234"
        "value": "Saturn"
    },
    {
        "id": "5678"
        "value": "Rings of Saturn"
    }
]

[dwinston]

awesome. looks and reads great. I'm curious about two things, but closing regardless:

1. do we need to await the response before checking for response.ok?

// If the response is bad, throw error
if (!response.ok) throw response;

// Otherwise, get response
let data = await response.text();

2. what is handleEvent about? I can't find it in the MDN docs under HTMLElement.

/**
* Handle event listeners
* @param  {Event} event The event object
*/
handleEvent (event) {
    this[`on${event.type}`](event);
}

[cferdinandi]

@dwinston Want me to create PR you can merge? Make any updates to the API response (to use JSON instead of HTML)? I'll update the Web Component accordingly.

do we need to await the response before checking for response.ok?

I am, here:

// Query the API
let response = await fetch(this.endpoint, {

The response.ok property is a boolean on the response, no asynchronous aspects to it.

what is handleEvent about? I can't find it in the MDN docs under HTMLElement.

Apparently a method on the HTMLElement that's been around since the early 2000s that I just learned about. Makes managing events while maintaining the correct context for this a lot easier: https://gomakethings.com/the-handleevent-method-is-the-absolute-best-way-to-handle-events-in-web-components/

Also, we can pull HTMX back out, since it's no longer needed.

[dwinston]

oh right, this is not a PR. Yes, please create the PR. 😄 And yes, I updated the API response to be json in the form you suggested, so please so update the Web Component accordingly.

[cferdinandi]

Updated, with PR: https://github.com/polyneme/heliokos/pull/30

[dwinston]

closed by #30