Roadmap to launch

[valencik]

Roadmap

Note: This is a work in progress, some steps are missing.

Goal Provide building blocks for search on Typelevel sites

Summary

• We need to expand the build matrices of textmogrify, lucille, and protosearch to include Scala 2.12
• Need to build an sbt-pluign to index, and modify sbt-typelevel to add search to site

Libraries Overview

• Text Analysis -> textmogrify
  • text analysis is a preprocessing step at both indexing time and query parsing time
• Query Parsing -> lucille
  • parse user input into a query tree, provide common helpers to rewrite/modify queries
• Indexing / Query Execution -> protosearch
  • building an index from raw documents given an analyzer, executing queries against that index given a query parser

Analysis

• Currently we are not leveraging textmogrify
• Currently protosearch uses a minimalist tokenizer split(" ")
• If we want textmogrify to be the analysis library, it needs a pure Scala implementation of at least the StandardAnalyzer
• For an MVP, it could be a some white space tokenizer, probably
• Lucille probably needs some way to leverage an analyzer/tokenizer during parsing
  • ideally this would allow us to remove the QueryAnalyzer hack where we rewrite the query applying the analyzer
  • semi related: https://github.com/cozydev-pink/protosearch/issues/3

sbt-plugin

• Likely the smoothest integration option involves adding search functionality via sbt-typelevel.
• Doing this requires use to support an sbt-plugin which means cross compiling to Scala 2.12
  • [x] Cross compile lucille to Scala 2.12
  • [x] Cross compile protosearch to Scala 2.12
• Build sbt-plugin the indexes documentation

Integrate with laika site

• We'll need to modify the Laika template sbt-typelevel is using to actually show and power a search bar somewhere.
• Should we use any frameworks here like calico, or still to pure ScalaJS?

[valencik]

What kind of compatibility guarantees should we provide on the index's binary representation?

If possible, I effectively want to provide zero guarantees for now. I am pretty confident the index structure will change over time, and if we're optimizing for small document collections, there's no reason we can't be frequently rebuilding the index.

[valencik]

Analysis

To provide a reasonable search over the types of documents we have in cats-effect, http4s, cats, and fs2, we likely want to do some sub-document search, turning every header section into a "sub document". With this we might possibly be able to get away without highlighting support initially.