Investigate genAI evaluation tools

FullFact / health-misinfo-shared

Raphael health misinformation project, shared by Full Fact and Google

MIT License

0 stars 0 forks source link

Investigate genAI evaluation tools #38

Closed dcorney closed 5 months ago

dcorney commented 5 months ago

Overview

There are a number of tools out there that are designed to speed up prompt engineering, by allowing rapid evaluation or comparison of genAI models. We should investigate these to see if they could be useful.

Requirements

[x] Investigate PromptFoo
[x] Investigate Auto side-by-side
[x] Write/share a short report about what they do, how we could use them, practical considerations, strengths/weaknesses etc.

dearden commented 5 months ago

Here's a write up of my work looking at promptfoo and autosxs