inspire recommendation system

[MJedr]

some inspiration https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch

[MJedr]

Steps:

• [ ] create a workspace in kubeflow to allow collaboration
• [ ] write a custom elasticsearch connector since the one in gpu-index library doesn't match our needs. Our connector should allow reading all the data returned by the search object. Also, it must be possible to query a nested field.
• [ ] investigate request limits of the OpenAI API
• [ ] prepare an index using llamaindex library
• [ ] Evaluate the results - how much time is needed to get the response
• [ ] think how we can evaluate the results - in the past we were using record references to evaluate the algorithm, but I am not sure if that's the best way

[ParthS007]

Finding on llamaindex library and the available indexes options

Context: LlamaIndex provides a set of data structures that allows to index data for various LLM tasks. The usage pattern is here.

Different types of Index available to check with our requirements.

• Nodes: Corresponds to a chunk of text from a Document.

• Response synthesis: Generated response after given nodes in input.

• List Index

  • Query: loads all Nodes in the list into the Response synthesis
  • Example: similar words/keywords filters

• Vector Index

  • Query: An embedding-based query which will fetch the top-k most similar nodes and pass them in response synthesis.
  • Example: search for a text in nodes published in 2019 only.

• Tree Index

  • Query: Querying a tree index involves traversing from root nodes down to leaf nodes. An argument child_branch_factor is there to decide the child nodes to include in response synthesis.
  • Example: Complex querying like search for the description for records published in a specific year and by a specific experiment/collaboration

• Keyword Table Index

  • Query: Extract relevant keywords from the query, and match those with pre-extracted Node keywords to fetch the corresponding Nodes.
  • Example: search for most frequently word used across nodes.

[ParthS007]

https://github.com/cern-sis/inspire-recommendation-system