Generate searchindex.js

[jaapio]

The searchindex.js is a very complex file. It looks like it would take a lot of effort to create a working file.

I'm not sure it's worth our time to invest in this as we already have a global search. And It would make more sense to improve that rather than something which is not fully supported by us.

[linawolf]

Let us close this and not support it anymore

[brotkrueml]

That is sad, i like the local search very much, especially when working locally to jump directly to a page instead of clicking (mostly TYPO3 Explained). Maybe we can find another solution.

And: The Elasticsearch results are bloated with different versions and produces too much "noise" (meaning doubled and irrelevant pages) when searching. I avoid to use that when possilble using the local search. But that is another topic.

[jaapio]

Elasticsearch can be improved to add for example a version restriction in the search query. But that's not on us?

[brotkrueml]

Nope, no one in the team has really experience with ES. But ES does not help me on the local documentation. Will make a research if there is some library which can be used to index and search the content locally (with JS). The current way is very convenient.

Additionally, if we want to drop that local search, we should remove it also from DRC. Because: One would select 12.4 and see no search box, switch to 10.4 there is - which might produce bug reports.

[linawolf]

Ok, If you would like to research this, Chris I am also happy.

[brotkrueml]

This library looks promising and was updated recently:

• https://github.com/lucaong/minisearch

There are also some others like

• https://github.com/olivernn/lunr.js (updated 3 years ago)
• https://github.com/krisk/fuse (updated 1,5 years ago)

We need to create a JSON file with the fields and content we need, like "title", "text" and of course the URL path in some kind. Fields can be boosted, to title can get a better relevance.

I don't know what @jaapio did, but AFAIK we need to do (can be done in PHP):

• after rendering: run through all generated HTML documents:
  • get the page title
  • get the body content
  • remove all tags
  • take the text and remove stopwords like "the", "some", "mostly", etc
  • add it to the search array
• jsonify the search array and write it to disk

Some current numbers for the currently used indexsearch.js (which seems to be part of the Sphinx project), the searchindex.js is:

• for the changelog: ~650 kB
• for TYPO3 Explained: ~205 kB

A new implementation should not be much bigger.

[jaapio]

I tried to generate the current searchindex.js,

I think @linawolf can show you how to create a renderer to render the output you need from the internal node structure.

[linawolf]

@jaapio that could be done similar to the interlink renderer?

[jaapio]

Yes

[brotkrueml]

We will have an improved Elasticsearch-based search, so this work can be omitted.