search

promptfoo / promptfoo-action

MIT License
10 stars 5 forks source link

Unexpected: Git diffing doesn't take into account unstaged files #122

Open jamesbraza opened 3 months ago

jamesbraza commented 3 months ago

I am manually changing my promptfooconfig.yaml in CI. This means the change is not committed, it's a local change.

promptfoo-action v1's git diff here: https://github.com/promptfoo/promptfoo-action/blob/v1/src/main.ts#L101-L105

It doesn't take into account local changes as leading to changed files. Thus my promptfoo-action doesn't invoke promptfoo and instead emits:

No LLM prompt or config files were modified.

Can we make this diff account for local file changes as well?

typpo commented 3 months ago

Interesting. Wouldn't it be pretty unusual for files to be changed locally on a stateless ci runner?

On Thu, May 30, 2024, 11:22 AM James Braza @.***> wrote:

I am manually changing my promptfooconfig.yaml in CI. This means the change is not committed, it's a local change.

promptfoo-action v1's git diff here: https://github.com/promptfoo/promptfoo-action/blob/v1/src/main.ts#L101-L105

It doesn't take into account local changes as leading to changed files. Thus my promptfoo-action doesn't invoke promptfoo and instead emits:

No LLM prompt or config files were modified.

Can we make this diff account for local file changes as well?

— Reply to this email directly, view it on GitHub https://github.com/promptfoo/promptfoo-action/issues/122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACLYJVZLTGFDXTWR5RQGHLZE5U7VAVCNFSM6AAAAABIRNLXHWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZDMMRYGY2DKOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jamesbraza commented 3 months ago

Yeah I think in general you're right that it is unusual, so feel free to close this out.

To share, my use case is generating the promptfooconfig.yaml on the fly in CI: https://github.com/Future-House/LitQA/blob/ba2b3f9f9798b8c7f40632c9ec3d68b3c1a6c67f/.github/create_tests.py