add keyword-based searching

[bard]

Scenario: user wants to locate a specific project using a specific term (e.g. project name) as opposed to explore available projects using generic terms

Requirements:

• an additional HTTP endpoint that returns projects matching given keyword(s), in order of relevance
• a way of querying the endpoint in the demo UI

Out of scope:

• merging semantic and keyword-based results (criteria on how to do so need to be defined)
• filtering keyword-based results (semantic results aren't currently being filtered either)
• performance and scalability (other than architecting so that it's easy to swap a more efficient indexing engine later)

Design sketch:

1. select indexing engine; criteria: quickest to get started with; in-process, i.e. no dependency external processes or services; candidates: lunr.py, xapian, sphinx
2. wrap existing semantic search in a generic interface/class (e.g. search_engine.load(documents) and search_engine.search(query)
3. implement index and search methods from 2) using engine from 1)
4. validate input documents such that the validation state is part of the data type (aka "parse, don't validate")

Open questions and trade-offs:

• not all search libraries return the document along with the result id (lunr doesn't)
• risk of having multiple in-memory copies of the same data, one for each indexing/searching strategy

[michellema1208]

@bard thank you so much, this is so awesome and thorough!

Basic question: what is the outcome of keyword-based filtering vs. keyword-based search? How do they differ/compare?

[bard]

Basic question: what is the outcome of keyword-based filtering vs. keyword-based search? How do they differ/compare?

It would satisfy the scenario mentioned at the beginning, i.e. would allow to ... "locate a specific project using a specific term (e.g. project name) as opposed to explore available projects using generic terms". Happy to modify the spec if you have suggestions on how to make that clearer.