search

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
http://prompttools.readthedocs.io
Apache License 2.0
2.55k stars 216 forks source link

LlamaIndex Integration #31

Open steventkrawczyk opened 11 months ago

steventkrawczyk commented 11 months ago

🚀 The feature

We need an experiment or example for LlamaIndex. This will be similar to the work we need to do for LangChain and MindsDB.

Motivation, pitch

LlamaIndex is a popular framework for connecting LLMs to data. It would be good to support LlamaIndex testing using our existing vectorDB and LLM evaluations.

Alternatives

N/A

Additional context

N/A

rachittshah commented 11 months ago

hi @steventkrawczyk I'd love to work on this.

My guess is we'd need abstractions over LlamaIndex over core prompttools classes, but we'll need to update it with each release of LlamaIndex.

steventkrawczyk commented 11 months ago

Hey @rachittshah we'd love your help! I think you're right that we'll need to build prompttools abstractions on top of the llamaindex abstractions. I'm hoping there are a few abstractions which are stable enough for us to build once, but if we need to update with each release we can explore that too.

Feel free to open a draft PR with what you're thinking now, and we can work together to iterate. If you want to chat in depth, I can add you to the dev channel of our discord, and we can set up a call to brainstorm if you'd find it useful.

Discord invite: https://discord.gg/72a9xh5Z7P

rachittshah commented 11 months ago

Hi @steventkrawczyk ! Joining discord, I think I'd open a draft PR with a proposed implementation of changing(since this would be a breaking change).

Joining discord, and keen to work on this!

baggiponte commented 7 months ago

+1 on this feature 😊 A client I work uses llama-index and there aren't notable libraries for LLM eval that are compatible. Yesterday deeplearning.ai released a course on RAG and LLM Eval using llama-index and truelens but I feel that promptools design would feel more natural and provide a simpler integration.