[R-263] Roadmap - v0.2

[jjmachan]

• [ ] #1010
  • [ ] Reference free
  • [ ] Generation
    • [ ] Summarisation
    • [ ] Code summary
    • [ ] Textual summary
  • [ ] With Reference
  • [ ] Generation for data types
    • [ ] Text
    • [ ] answer correctness
    • [ ] Code
    • [ ] SQL
• [ ] #1011
  • [ ] make ragas metrics deployable as a server
  • [ ] make testset generation interactive with an API
• [ ] #1018
• [ ] #1012
• [ ] #1015
• [ ] #1016
  • [ ] for RAG
  • [ ] structured data
  • [ ] unstructured data
  • [ ] Agents simulations
  • [ ] Based on predefined task & conditions
  • [ ] State to persist knowledge graphs and results in test generation

_{From SyncLinear.com | R-263}

[rajib76]

We lack chunk quality metrics as of today. It will be good to see some chunk quality evaluation metrics.

[jjmachan]

hey @rajib76, thanks for chipping in 🙂

could you explain a bit more about how you're measuring quality here? maybe an example too if possible?

[rajib76]

One of the hard problem today in RAG is to determine the right size of the chunk. If a chunk talks about multiple concept, it is very difficult to find the most relevant chunk for the question. I was looking for a metrics that will tell that a chunk is atomic and it talks about only one concept. The semantic chunking approach did not work as the embedding model itself has a semantic dissonance.