Generated Robustness Evaluation Sample Scripts

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).

http://prompttools.readthedocs.io

Apache License 2.0

2.71k stars 232 forks source link

Generated Robustness Evaluation Sample Scripts #117

Open RigvedRocks opened 10 months ago

RigvedRocks commented 10 months ago

Hey, I am working on issue #69 and I have generated 2 sample scripts - one involving prompt substitution and the other for auto evaluation. I am using Promptbench in both scripts. Can you please guide me as to how to integrate the scripts into your experiments?

CLAassistant commented 10 months ago

All committers have signed the CLA.

steventkrawczyk commented 10 months ago

Thanks for raising this @RigvedRocks! I'll review it now. When you get a chance, could you fill out the CLA?

steventkrawczyk commented 10 months ago

I'm not very familiar with promptbench but it looks like you want to run attacks as experiments and use your eval function to evaluate the experiment outputs. You can check out experiment examples to see how we feed prompts into LLMs for testing, and how we pass in eval functions too. Let me know if you have any specific questions about it. Thanks again for looking into this issue!

RigvedRocks commented 10 months ago

Sure Mr. Krawcyzk. I will have a look at the implementation of the library and will make changes according to that. When time permits, I will fill out the CLA.

RigvedRocks commented 10 months ago

Hey, I have generated a sample script to evaluate the robustness of the models under the utils folder. Please have a look at it and suggest amends if any.