search

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
http://prompttools.readthedocs.io
Apache License 2.0
2.65k stars 230 forks source link

Allowing custom kwargs to be passed to evaluation function #36

Closed NivekT closed 1 year ago

NivekT commented 1 year ago

This should be compatible with all existing use cases. We might even be able to remove expected as an argument.

github-actions[bot] commented 1 year ago
messages response latency model
0 [{'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': 'Is 17077 a prime number?'}] George Washington 5.4e-06 gpt-3.5-turbo
1 [{'role': 'system', 'content': 'You are a math tutor.'}, {'role': 'user', 'content': 'Is 17077 a prime number?'}] George Washington 2.7e-06 gpt-3.5-turbo
2 [{'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': 'Is 17077 a prime number?'}] George Washington 2.1e-06 gpt-4
3 [{'role': 'system', 'content': 'You are a math tutor.'}, {'role': 'user', 'content': 'Is 17077 a prime number?'}] George Washington 1.7e-06 gpt-4