search

aalok-sathe / surprisal

A unified interface for computing surprisal (log probabilities) from language models! Supports neural, symbolic, and black-box API models.
https://aalok-sathe.github.io/surprisal/
MIT License
32 stars 6 forks source link

Add deprecation warning to OpenAI models: OpenAI has deprecated their logprobs API for user input #15

Closed aalok-sathe closed 8 months ago

aalok-sathe commented 12 months ago

OpenAI no longer provides log probs for the prompt, making it impossible to use as a probability over a string function. It does, however, continue to provide logprobs over its own completions.

E.g.:

Originally reported on twitter.

aalok-sathe commented 10 months ago

Reopening, as reported by OpenAI on X, log probs seem to be available again, so it would be nice to re-build and test the (existing) support for this in our package. Thanks @jennhu!

openAI docs

aalok-sathe commented 8 months ago

@jennhu do you have a MWE for this with the new API? I just haven't used openai models in a while so if you have something that already works I wouldn't have to start over:)

aalok-sathe commented 8 months ago

if you have the bandwidth to PR even better but if not I'll take anything to start off of!

jennhu commented 8 months ago

After poking around some more with the chat models, I don't think we can get surprisals after all :(

The logprobs parameter lets us get the logprobs corresponding to the model's generated output, but without the echo parameter (as was previously supported by the legacy Completion models), we can't get the logprobs assigned to an arbitrary input string. It's possible I might be missing something, though!

aalok-sathe commented 8 months ago

I've already added a deprecation warning in README as of https://github.com/aalok-sathe/surprisal/commit/a926f9b; the API should continue working for older models supporting this endpoint (though I haven't tested it recently), but otherwise the broader issue about adding support is going to be a wontfix. Alternatives to OpenAI models include: MPT, Mistral, Llama, Falcon (not an exhaustive list or endorsement).