feat: support cli runs on github actions - Githubissues

empirical-run / empirical

Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application

https://docs.empirical.run

MIT License

147 stars 12 forks source link

feat: support cli runs on github actions #56

Closed arjunattam closed 5 months ago

arjunattam commented 5 months ago

Before we merge

[x] docs for ci/cd support
[x] should we give an example of what failed? more than just the numbers?
[x] failure scenario: if API key is missing, the results will not show the error (outputs will be 0%)
[x] report numbers on the pull request/commit (as a comment - see action)

Not a priority

[ ] fail the action if numbers are below a threshold
[ ] link to open the web app (share link)

changeset-bot[bot] commented 5 months ago

🦋 Changeset detected

Latest commit: 3ff96c72fd1062a06c7ff6ad7e9831afb9f88cb7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

| Name | Type | | ----------------- | ----- | | @empiricalrun/cli | Minor |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions[bot] commented 5 months ago

Empirical Run Summary

Stats	gpt-3.5-turbo run	gpt-4-turbo-preview run	Run #992a: claude-3-haiku
outputs	100%	100%	0%
is-json	100%	0%	0%

Total dataset samples: 2

Error: Some outputs were not generated successfully AI202: process.env.ANTHROPIC_API_KEY is not set