Open seiggy opened 1 year ago
Thanks for your contribution! And thank you specifically for providing query and response examples.
Which GPT model version are you using?
During the initial implementation I also used system
role for the initial prompt, but found that at least GPT-3.5 tends to loose focus on it. Anyway, I'll test your suggestion later in more detail.
Since I'm not that familiar with PF2e, it would also help me to have a number of queries handy, if somebody is willing to spend time to come up with them.
Oh, I was using GPT 3.5 turbo
Sure thing! I'll build out a bunch of test queries to help. Might need to add a switch to use system
role for PF2e and user
for 5e if it starts to loose focus for 5e. I didn't test much with GPT4 because it's so much slower and more expensive.
Change the Prompt Message to use the "system" role for better accuracy.
I've found (at least with my experiments with PF2e) that if you send nothing for the system message and 2 user messages, you'll get less accurate responses where the system tends to invent rules that don't exist. This change should tighten up rules accuracy a little in PF2e, and hopefully shouldn't negatively impact other systems (a couple cursory tests with D&D 5e questions seemed just fine).
Here's the sample results I have for the 2 different approaches:
Using "user" for both messages, with the question: "What are the rules for grappling?"
Response:
Using "system" for the system prompt and "user" for the user's question:
Response:
The key important differences here are the addition of the immobilized condition, the explanation of that condition, and occasionally when using "user" ChatGPT will claim you can do things like "move" or "pin" a creature, which are not allowed actions in PF2e. I have yet to see said issue when using the "system" approach.