Key Phrase extraction using KeyBERT

[dvp-git]

TODO:

• Try to optimize the document embeddings using only Keyphrases from the query .
• Embed only keyphrase using keyphrase extracting like keyBert to see if this can potnentially optimize inference time.
• Train an LLM to extract meaning from each sentence in the document. Ask the LLM to summarize each paragraph in 100 words. Use this LLM first to get the summarized documents and then use it as ground truth for the RAG task
• Other techniques such as those discussed from : https://learn.deeplearning.ai/courses/advanced-retrieval-for-ai/lesson/4/query-expansion
  • Re-Ranking or Query expansion would be good to explore.