search

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
http://prompttools.readthedocs.io
Apache License 2.0
2.65k stars 230 forks source link

Refactor PromptTest module #25

Closed steventkrawczyk closed 1 year ago

steventkrawczyk commented 1 year ago

  1. Change prompttest API

    • Previously, we annotated an evaluation function. Now, we annotate a prompt provider function. This seems like a more natural way to test prompts.
    • expected argument for semantic similarity is still somewhat hacky.
    • See examples/prompttest/test_openai_chat.py or examples/prompttest/test_huggingface_hub.py for examples.

  2. Add evaluate functions to utilities