Improve relevance and ordering of search results in Link UI and Navigation block

[annezazu]

Update: I've revised this Issue to be focused on the relevance of the search results. I've spun out other concerns into a separate Issue.

In summary the Post search results shown are sometimes counterintuitive and we may need to refine the REST API search endpoint to improve this.

Original Issue description

## What problem does this address? This week, [Anna Morris shared the following feedback](https://wordpress.slack.com/archives/C015GUFFC00/p1642529259080700?thread_ts=1642526603.077800&cid=C015GUFFC00) in the FSE Outreach Program's channel: >But mainly, selecting what page to add feels really hard. So, finding the News page (for a client site) for example took ages, it brings up everything with News in. If I wasn't totally sure of the name of a page (actually it was called "blog" I actually had to go and find the page and check, and come back and search. I've never had to do that before. I think there's something with the hirachy of info it shows that's off maybe? Also not being able to go... well it's a page... what pages do I have in a list (like before) is somehow hard for me cognitively perhaps. Finally, all that info of listing pages/posts/cateogries etc with the seached term in in a little dropdown thing was hard, it's fiddly and small and exhausting At the core of this is the experience of what appears upon searching for something to add to the navigation block. Right now, it's a bit odd what's displayed. Here are two very simple examples: I search for "test" and "testing duotone" comes up before the post actually titled "test": I search for "contact" and the one with the worse permalink (/contact-2) appears first: It seems we might be overly biasing for recently updated rather than name matches? ## What is your proposed solution? I'm not quite sure! This is too high level and obvious to say but add more granularity to the search functionality. I imagine this problem will likely ease when one can bulk add items/see more at once: https://github.com/WordPress/gutenberg/issues/31667

[talldan]

Thanks for bringing this to attention. There have been some relevant discussions before. You mentioned the bulk linking interface which is one.

I remembered @jasmussen also had some proposals in this issue - https://github.com/WordPress/gutenberg/issues/34041#issuecomment-948597379.

[jasmussen]

The URL dialog is definitely a complicated piece of UI. Ideally it's remains a singular component that can be used across contexts in the UI, meaning it needs to be able to search across all buckets of content:

At the same time, some contexts like navigation items or social links would benefit from strictly filtering the output:

Including search in the navigation item filter makes it a bit taller, but maybe it can thread the needle if we let the field be scrollable:

[getdave]

I agree with everything discussed here. I've renamed to make it a little clearer as to the purpose of the Issue and also added to the Nav block tracking issue.

[kevin940726]

Seems like the order of the results mentioned in the original issue is queried from the search rest API. Perhaps we can add orderBy to the API so that we can customize the suggestions query.

[getdave]

Seems like the order of the results mentioned in the original issue is queried from the search rest API. Perhaps we can add orderBy to the API so that we can customize the suggestions query.

Let me check this.

@kevin940726 I don't see an order parameter available on that endpoint.

See also https://github.com/WordPress/wordpress-develop/blob/541b4eb6462c8f48e235590a9be79ebc4954f66e/src/wp-includes/rest-api/endpoints/class-wp-rest-search-controller.php

[kevin940726]

@getdave Oh, I mean we should add the missing functionality of the orderBy parameter to the rest API to support this. 😅

[annezazu]

Logging this feedback (and ideas) related to this functionality as a part of the FSE Outreach Program's recent exploration, Site Editor Sneak Peek:

Adding a link to Navigation. Currently adding a new link to Navigation one can begin to write the name of the link and various options show up. Example. I write new, and see a post, a page and an URL show up. Scrolling briefly I see + Create draft page: new. Example 2. I do not write anything in the “Search or type url” add a Nav link field. Just below the Search field I see a Recently updated. It shows me 3 posts. What if we have multiple small button toggles (on/off) sitting beside each other below the “Search or type url” Which says Pages Posts Categories Tags … One could click to toggle one or more of these toggles. Clicking Pages toggle one would see a list which one can scroll of pages. Clicking Pages and Posts toggles one would see a list of pages and posts one can scroll through.

[getdave]

Thanks @annezazu. There has been discussion of enhancing and improving the Link UI in regard to providing a better interface for discovering and adding different types of link content. The idea suggested is noted 🙇

[getdave]

This is now being discussed in https://github.com/WordPress/gutenberg/issues/49091

[annezazu]

Noting feedback from the Front Page Fun call for testing for the FSE Outreach Program:

Personally, I miss having all my content (posts, pages, categories) and navigation options visible at once, all the time, without having to see a bunch of other options for other tasks. It’s just easier than clicking three or four times in the site editor UI, in multiple places, just to see what content is available to add as a menu item.

While I understand we've moved away from a bulk add option, it still is a big ask to expect folks to remember the name of the posts/pages/categories they want to add and creates a lot of additional steps.

[getdave]

I agree @annezazu. The means of finding pages via search is a big ask. I think some means of seeing all the Pages on your site in a UI would be a great help.

However the list of priorities for the Nav block for 6.3 is here.

[getdave]

The discussion above reveals this Issue has two possible solutions:

1. Add an orderBy parameter to the Search API to allow for customising the order of the results returned as a secondary listing factor (i.e. keyword first and then sort by X).
2. Create a means to see all your Pages in a single view and pick the one you want rather than having to rely on search.

Of these option 1 seems like the most actionable. Option 2 requires Design/UX exploration.

[getdave]

I've done a little more digging here into why we sometimes get unexpected search results when looking for Posts with similar titles.

For example, why does "Newspapers" take precedence over the second "News" page here?:

Logically as the search string is news it should prefer News over Newspapers in the results but that isn't the case.

I looked into how the REST API works here:

The Search endpoint uses WP_Query under the hood.

When providing the s (search) param to WP_Query then the orderBy parameter is set to relevance.

To quote the docs:

Default is 'date', except when a search is being performed, when the default is 'relevance'.

Therefore the results shown are a product of the search implementation provided by WP_Query which as I understand it matches based on a SQL LIKE operator on the post title, excerpt and content columns and orderBys based on the same.

I therefore believe "relevance" is shorthand for "order based on the match to the search query".

I wonder if anyone with more experience in this area of Core can shed more light on this. Maybe @azaozz or @ironprogrammer would know or at least have an idea of who to ask?

[getdave]

Update: I moved this to a separate Issue in https://github.com/WordPress/gutenberg/issues/38121

---

Create a means to see all your Pages in a single view and pick the one you want rather than having to rely on search.

@richtabor On this point, I wonder whether we could expand upon the UX pattern introduced into the Link UI with the Add block area, to add a new option called Select {Post Type} (i.e. Select Page).

When clicked, it would show a tree representation of all your Pages which you could pick from. I believe that a dedicated view is best because trying to shoehorn in a hierarchical, nested view into the Link UI search results will only add further complication.

In envisage this include a UI similar (but not identical) to that I mocked up in this Issue.

[ironprogrammer]

Logically as the search string is news it should prefer News over Newspapers in the results but that isn't the case.

To quote the docs:

Default is 'date', except when a search is being performed, when the default is 'relevance'.

Hey, @getdave -- I looked into this, and may have an explanation of why you're seeing the news results you've shared.

For a single-term search like news, Core's "relevance" ordering prepends the default ORDER BY wp_posts.post_date DESC to yield ORDER BY wp_posts.post_title LIKE '%news%' DESC, wp_posts.post_date DESC. Since news matches each, the default post_date ordering kicks in, yielding the following:

(Ignore the ID sequence, as I re-dated news-2 for this test, on the presumption that this is similar to your screenshot.)

That post_date DESC ordering is then passed into sortResults(), which scores 1 on everything but "News Travels Fast", which scores 0.3, thus bumping it down to the "least relevant" result. If you were to re-date the other items that score 1, they would correspondingly re-sort per post_date DESC before being passed into sortResults(), but continue to each score 1 and not be re-ordered from what was returned from the REST call.

Perhaps the final sort order could account for exact matches, which would score higher relevance than being part of a word like "Newspaper"?

[getdave]

Perhaps the final sort order could account for exact matches, which would score higher relevance than being part of a word like "Newspaper"?

I think you are right. Optimising the weighting algorithm on the JS side to prefer exact matches could solve this issue. Thank you so much for all that work 🙇

[getdave]

@ironprogrammer I created https://github.com/WordPress/gutenberg/pull/67367 to attempt to resolve this. Reviews much appreciated.