Implement search function using Atlas Search

thekid commented 1 year ago

See https://dialog.sloppy.zone/search?q=Yosemite:

Search screenshot

Suggestions

Suggestions screenshot

Based on titles

[x] Basic autocompleter
[x] Keyboard usage with up and down arrow keys (select suggestion), escape (closes suggestions), enter (goes to selected suggestion)
[x] Accessibility considerations, see https://youtu.be/_w6KvvN9cWw
[x] Indicate more search results are available in full search
[x] Close suggestions when focus leaves search field

Ranking

From highest to lowest:

[x] A direct hit on a location name
[x] A direct hit in the title
[x] Phrase contained in the content
[x] Fuzzy matching on title and content

Additional boost factors:

[x] Journeys should be preferred over their children
[ ] Boosting by date, newer = more relevant
[ ] If the search term contains a number at the end, filter by date

Facetted search

We could include the year as facette

Keywords & synonyms

[x] Any content should be able to provide an arbitrary number of keywords, which are not shown but used for searching (and inside meta tags).
[ ] Videos could automatically have a "video" keyword
[ ] Can we get more details from Google maps / Wikipedia / ... for a location and use English / German / native spelling as synonyms?

Atlas configuration

A search index needs to be created in MongoDB Atlas!

{
  "mappings": {
    "dynamic": true,
    "fields": {
      "title": [
        {
          "dynamic": true,
          "type": "document"
        },
        {
          "type": "autocomplete"
        }
      ]
    }
  }
}

[ ] Can / should we automatically create this at application startup?

Implementation

https://dribbble.com/shots/12682498-Travello-Travel-Web-App-Website-Design/attachments/4281739?mode=media - Design inspiration

thekid commented 1 year ago

We should remove HTML before searching, the highlighting function produces these kinds of fragments every now and then:

HTML fragments

thekid commented 1 year ago

We should remove HTML before searching, the highlighting function produces these kinds of fragments every now and then: [...]

Implemented in thekid/dialog@a589a03. Existing data can be migrated as follows:

use com\mongodb\MongoConnection;
use util\cmd\Console;

$c= new MongoConnection($argv[1]);
$entries= $c->database($argv[2])->collection('entries');
foreach ($entries->find() as $document) {
  Console::writeLine('> ', $document['title'], ' ', $document['is']);
  $entries->update($document->id(), ['$set' => [
    '_searchable' => [
      'content' => strip_tags(strtr($document['content'], ['<br>' => "\n", '</p><p>' => "\n"])),
      'boost'   => isset($document['is']['journey']) ? 2 : 1,
    ]
  ]]);
}

Now looks like this: https://dialog.sloppy.zone/search?q=Video

thekid commented 1 year ago

We need to be able to distinguish journeys and content in this case:

Journey and content with same title

thekid commented 1 year ago

Boosting results by date so that newer entries take precedenc over older ones is not as easy as I thought it would be. It will probably require a gauss function score, see https://www.mongodb.com/docs/atlas/atlas-search/scoring/#function and https://www.elastic.co/de/blog/0-90-4-released/

thekid commented 1 year ago

Released in https://github.com/thekid/dialog/releases/tag/v1.3.0

thekid / dialog