search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
385 stars 41 forks source link

Support retrieval as a first-class element #53

Open tekumara opened 2 weeks ago

tekumara commented 2 weeks ago

I'm using inspect_ai to evaluate a RAG pipeline and it would be nice to be able to expose the retrieved docs and metadata (eg: their score) in the results and UI.

For now I've been using a solver that runs retrieval and generation, and adds the retrieved documents and their metadata as state.metadata.

jjallaire commented 2 weeks ago

Thanks for this, you are currently doing it the "right" way but I understand that the the UI display leaves a lot to be desired. We made message histories first class in metadata, @dragonstyle I wonder if there is a way to make documents (and their score) similarly first class?