search

promptfoo / promptfoo

Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
https://promptfoo.dev
MIT License
4.64k stars 364 forks source link

v0.45 breaks input data files #530

Closed WilliamHua closed 8 months ago

WilliamHua commented 8 months ago

v0.45 seems to break the ability to load input data

promptfooconfig.yaml

prompts:
  - file://prompt3.json
  - file://prompt7.json

providers:
  # - id: openai:chat:gpt-4-0613
  # - id: anthropic:completion:claude-3-opus-20240229
  - id: openai:chat:gpt-4-vision-preview

tests:
  - vars:
      data: file://data1.json
  - vars:
      data: file://data2.json
  - vars:
      data: file://data3.json
  - vars:
      data: file://data4.json
  - vars:
      data: file://data5.json

npx promptfoo@v0.45 eval --no-cache results in the model not seeing any of the input data npx promptfoo@v0.44 eval --no-cache works as expected

typpo commented 8 months ago

Thanks for the catch. #531 fixes and adds relevant tests.