search

promptfoo / promptfoo

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
https://promptfoo.dev
MIT License
4.31k stars 320 forks source link

Question: Is there a way to separate test runs depending on a given set? #1786

Open Starday2009 opened 2 hours ago

Starday2009 commented 2 hours ago

Question: Is there a way to separate test runs depending on a given set? For example, I want to run a test set daily for one dataset, and once a week for 10 datasets. How can I separate these runs if everything needs to be stored in one file - promptfooconfig.yaml?

sklein12 commented 1 hour ago

@Starday2009 I would create two config files and then you can specify which one you want to run using -c eg: promtfoo eval -c weeklyconfig.yaml