search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.57k stars 646 forks source link

Swiftide integration #1165

Open timonv opened 1 month ago

timonv commented 1 month ago

Hey everyone, amazing work on Ragas! Recently I've been working on Swiftide, and I'm looking to integrate with an evaluation framework. Rust is great at performance, Python is king at data anytics.

We don't have python bindings yet, but I'd be happy to do everything on the Swiftide side to put down something cool.

jjmachan commented 1 month ago

@timonv thanks for raising this suggestion - would love to do it, Swiftide looks great! How should we do this though? One way could be with #1011 and making some rust bindings?

but I think the easiest would be to create a notebook/documentation with utils code that makes it easier

  1. create the dataframe needed by ragas with swiftide
  2. evaluate with ragas
  3. show results

would love to hear your suggestions 🙂

timonv commented 1 month ago

@jjmachan Is the dataset basically question, contexts (retrieved documents?) and an answer? #1011 would be amazing, my guess is that most users would prefer to start out with a python notebook so they can analyze and compare different setups.

Any ideas on how we would run it from a notebook? Full python bindings are a bit of an effort as we are heavy on the less trivial features. Maybe I can create a tiny wrapper support library that can call a fully constructed pipeline from python.

An even simpler poc approach could be to simply call a compiled binary and have the dataframe output to a file or stdout.

jjmachan commented 1 month ago

alrighty maybe lets start with the notebook then and take it up from there? would love to have a quick chat with you before so we can plan?