Open jamesbraza opened 3 months ago
Interesting. Wouldn't it be pretty unusual for files to be changed locally on a stateless ci runner?
On Thu, May 30, 2024, 11:22 AM James Braza @.***> wrote:
I am manually changing my promptfooconfig.yaml in CI. This means the change is not committed, it's a local change.
promptfoo-action v1's git diff here: https://github.com/promptfoo/promptfoo-action/blob/v1/src/main.ts#L101-L105
It doesn't take into account local changes as leading to changed files. Thus my promptfoo-action doesn't invoke promptfoo and instead emits:
No LLM prompt or config files were modified.
Can we make this diff account for local file changes as well?
— Reply to this email directly, view it on GitHub https://github.com/promptfoo/promptfoo-action/issues/122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACLYJVZLTGFDXTWR5RQGHLZE5U7VAVCNFSM6AAAAABIRNLXHWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZDMMRYGY2DKOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Yeah I think in general you're right that it is unusual, so feel free to close this out.
To share, my use case is generating the promptfooconfig.yaml
on the fly in CI: https://github.com/Future-House/LitQA/blob/ba2b3f9f9798b8c7f40632c9ec3d68b3c1a6c67f/.github/create_tests.py
I am manually changing my
promptfooconfig.yaml
in CI. This means the change is not committed, it's a local change.promptfoo-action
v1'sgit diff
here: https://github.com/promptfoo/promptfoo-action/blob/v1/src/main.ts#L101-L105It doesn't take into account local changes as leading to changed files. Thus my
promptfoo-action
doesn't invokepromptfoo
and instead emits:Can we make this diff account for local file changes as well?