Overview for "text search" topic

[hpvd]

At this time, it's a little hard to build a complete picture of the field of "text search" topic in Pinot with

• available capabilities,
• ongoing efforts,
• ideas and
• plans

even after having checked the docs

• https://docs.pinot.apache.org/basics/indexing/text-search-support
• https://docs.pinot.apache.org/basics/indexing/native-text-index#searching-native-text-indices

and having went through many issues and PRs....

=> Would be great to keep and maintain a good overview on this "text search" topic by e.g.

• aggregating all text search related topics/issues/ into one master/parent/top-level issue,
• adding a new grouping label (e.g. FTS for full text search, similar to upsert label)
• ...

One can find several ones when searching, e.g. for

• "full text" (5/1): https://github.com/apache/pinot/issues?q=is%3Aissue+%22full+text%22+

  • 10934

  • 7395

  • 7218

  • 2798

• "text search" (14/6): https://github.com/apache/pinot/issues?q=is%3Aissue+%22text+search%22
• "lucene" (22/8): https://github.com/apache/pinot/issues?q=is%3Aissue+lucene
• "embeddings" (3/0): https://github.com/apache/pinot/issues?q=is%3Aissue+embeddings

  • 10919

  • 7218

• ...

There are really many issues/pull requests and some are

• relatively old,
• partly done,
• waiting for merge
• overlapping
• ...

A great full text search is of course complex, but imho pretty rewarding and could be a huge step for Pinot. Having a good overview, could be a valuable step...

[hpvd]

just a twist in POV:

Why not having in a stunning OLAP-system like Pinot

• the same search possibilities and
• same search result quality

like every good Ecommerce store has for its product search??

with

1. fast results
2. well ranked
3. find exact matches
4. even finding a. incomplete words b. other spellings of words (mistakes and grammar) c. other words for the same meaning (similarity)
5. working easily for different languages
6. don't limit search exclusively to one column (ecommerce: search in category name, article name, article description with applied weight factors to each column)
7. enabling set the number of results (if there were not enough results from first search, make a second one with more loose quality regulation)
8. resorting order afterwards (based on values in other columns)
9. slice and dice afterwards (narrow the filter/faceting, based on values in other columns)

10. ...

    see example of a search config setup of an ecommerce backend: https://github.com/apache/pinot/issues/7218#issuecomment-1943661027

[hpvd]

maybe these named items of a great ecommerce search

• could be a possible approach to group the overview of the text search topic
• could help to identify some possible plug points for pluggable variants / extensions