Learn about auto extracting and separating texts for context ahead of ingestion

[Mael-Abgrall]

Here are some stuff you can check out:

• [x] https://huggingface.co/papers/2410.08815
• [x] Document your findings on this issue + datasets used

[sriram-vinnakota]

StructRAG: Useful only if we know query, not recommend if we don't know the category or topic

Grouping text with same context

Dataset used: tropical chat-bot data

It has over 8k conversations, and each conversation might deviate and get back to the original topic.

Findings:

• could not able to achieve the grouping the topics by clustering
• we might achieve the grouping by relying on embedding similarity as follows:
  • Extract the embedding at current_idx and add it to last_three_embeddings.
  • If last_three_embeddings has more than 3 elements, remove the oldest one.
  • Calculate the mean of the embeddings in last_three_embeddings.
  • Calculate cosine similarity between the mean embedding and all subsequent embeddings in data.
  • Find the index of the first subsequent embedding with similarity above threshold.
  • If such an embedding is found:
  • Update current_idx to the index of the new finding.