search

646e62 / case-brief

Generates a FIRAC-style case brief from a reported decision
GNU General Public License v3.0
2 stars 0 forks source link

GPT-3 fills in blanks without enough information #24

Closed 646e62 closed 1 year ago

646e62 commented 1 year ago

GPT-3 is doing well when given a fair bit of info and asked to summarize or analyze it. It does not do as well when asked to summarize a single sentence, and tends to make up answers to fill in perceived blanks. This may be resolvable with alternative prompts, more restrained settings when using small text samples, or another means.

bbelderbos commented 1 year ago

You're testing this with openai.Completion.create(...? And the usage of OPENAI_API_KEY incurs the cost?

646e62 commented 1 year ago

Added some min/max targets in the summarizer function to test this out and see what works best.

646e62 commented 1 year ago

My current system supplements these shortcomings with GPT-4. GPT-4 has produced superior results for calls requiring more analysis than summarization. For summarization calls, GPT-3.5 is more than enough and significantly less expensive.