Improve free text search

[aaaaargZombies]

Acceptance Criteria

• [x] I can search for a guide using free text search
  • including external guides from API depends on #31
• [x] When I search, the results make sense
• [x] I can search resources based on the current language
• [x] If I change language, the search is re-run
• [x] There is some indication of what has been searched for (e.g. text at the top of the page)
• [ ] Search results are ordered based on how close of a match they are
• [ ] Search implementation follows accessibility best practices

Implementation Notes

• [x] use fuse search library based on this comparison test

• [x] search result are displayed using one of these methods

  • Existing teaser list is filtered based on results
  • A pop up element appears based on a filtered teaser list
  • a combination of the above

• we want to include Actions in this search, so this needs doing first: #31

Elasticlunr

http://elasticlunr.com/

from the getting started …

var index = elasticlunr(function () {
    this.addField('title');
    this.addField('body');
    this.setRef('id');
});

var doc1 = {
    "id": 1,
    "title": "Oracle released its latest database Oracle 12g",
    "body": "Yestaday Oracle has released its new database Oracle 12g, this would make more money for this company and lead to a nice profit report of annual year."
}

var doc2 = {
    "id": 2,
    "title": "Oracle released its profit report of 2015",
    "body": "As expected, Oracle released its profit report of 2015, during the good sales of database and hardware, Oracle's profit of 2015 reached 12.5 Billion."
}

index.addDoc(doc1);
index.addDoc(doc2);

If all the json has uniform fields this will probably work nicely

• https://github.com/kyrim/get-your-nutrients has elasticlunr as dependency
  • json passed to model and elastic search port set up here https://github.com/kyrim/get-your-nutrients/blob/master/client/js/app.js

some elm packages that do search

• https://package.elm-lang.org/packages/jxxcarlson/elm-text-search/latest/

• https://package.elm-lang.org/packages/jxxcarlson/elm-search/latest/

• https://github.com/elm/package.elm-lang.org/ has some sort of search on the side and on the homepage

Fuse (probably the better option?)

https://fusejs.io/

Looks to be more actively maintained than elasticlunr

Search-index

https://github.com/fergiemcdowall/search-index

External JS in Elm

Ports

• https://guide.elm-lang.org/interop/ports.html

Web component

• https://guide.elm-lang.org/interop/custom_elements.html
• https://htmlwithsuperpowers.netlify.app/get-started/

Examples

• https://github.com/elm-community/js-integration-examples

[aaaaargZombies]

By default elasticlunr is configured to work with english. There are some other language options available here

But no welsh, this means we should think particularly about stop words we need.

[katjam]

Oh that's very annoying about the Welsh.

In the first instance we maybe hide the search form if Welsh is toggled on?

[aaaaargZombies]

I've been looking at alternatives and honestly nothing looks like it's going to have purpose built welsh support. I guess shouldn't be a surprise if so much software isn't even built for huge languages like chinese or spanish.

[aaaaargZombies]

A useful comparison/demo of JS search options https://github.com/fiatjaf/search-comparison

Lunr based looks to be stricter, fuse is more fuzzy. which might be good for people who can't spell plant/animal names perfectly but might also produce more irrelevant results.

[aaaaargZombies]

questionably named http://www.gaddafipsum.com/ provides welsh gibberish, though it might be better to find some real welsh writing to test with.

[kimadactyl]

Worth asking the discord for suggestions here?

[aaaaargZombies]

welsh sample text options

• https://qfp.quaker.org.uk/chapter/1/
• https://cy.m.wikipedia.org/wiki/Hafan

[katjam]

@aaaaargZombies Is any of this ready to merge?

[katjam]

@aaaaargZombies I was attempting to verify this as:

I re-jigged your AC into criteria + plan and have come to think that AC you left un-ticked didn't happen.

I'm not marking this as verified because I think the list returned is not always a very good match, so am wondering if we need to look into this a little more? I've not dug very deep, but for example https://team-wilder-proto.pages.dev/guides/making-sure-everyone-can-access-your-group-or-project comes up in a search for climate and this comes up in a search for woody https://www.wildlifetrusts.org/actions/reduce-food-waste.

I fee like we should think about improvements to search in the next cycle, but wanted you to verify first what the intent is as it is now. So if you can remember what you think you had working, update the AC here and we can verify as a baseline.

[r-ferrier]

Some more intel on this from discord:

hey all, you can adjust that here https://fusejs.io/api/options.html#distance I left the default. also if it finds no options it doesn't filter but maybe that's misleading...

now the plumbings in place it will probably be a quick fix to switch search libraries, I went for this one based on npm downloads, most recent code contributions and eyeballing the results on the live test I listed in the issue

[aaaaargZombies]

@undividual I have this to implement

There is some indication of what has been searched for (e.g. text at the top of the page)

Do you have any suggestions about how you'd like it to look. I was thinking of just adding a little subtitle below guides that says something like

Filtering on: XXXX showing NNN guides

We could update the actual guides title to be like

NN Guides on XXX

where X = query string and N = num guides

[aaaaargZombies]

search accessibility

https://technology.blog.gov.uk/2014/08/14/improving-accessibility-on-gov-uk-search/

[aaaaargZombies]

• [ ] Search happens when I click submit (arrow? + text)

removed based on conversation in meeting today

[aaaaargZombies]

Search results are ordered based on how close of a match they are

resulting JSON is in order, in elm it is displaying in alphabetical order of title