openai / openai-cookbook

Examples and guides for using the OpenAI API
https://cookbook.openai.com
MIT License
59.54k stars 9.47k forks source link

[PROBLEM] Inconsistent Deterministic Outputs with Seed Parameter in OpenAI API #861

Open hengqujushi opened 11 months ago

hengqujushi commented 11 months ago

Identify the file to be fixed N/A - This issue is related to the OpenAI API's deterministic behavior when using the seed parameter.

Describe the problem I am encountering an issue where I receive inconsistent outputs from the OpenAI API despite setting the same seed parameter for deterministic behavior. This is unexpected as the seed parameter is intended to ensure reproducibility of results. Both the System Fingerprint and the Seed values are identical for different requests, but the responses differ.

Describe a solution A potential solution would be to investigate the determinism functionality within the OpenAI API when the seed parameter is set. Ensuring that the outputs are consistent across multiple requests with the same seed would resolve this issue.

Screenshots image

Additional context

tsilva commented 11 months ago

I confirm that the same issue is happening to me as well.

Getting different responses each time, while getting the same system_fingerprint.

lionelchg commented 11 months ago

Same here using the parameters from the original notebook

jd-codelink commented 11 months ago

Anybody able to reproduce the deterministic results?

SovereignRemedy commented 11 months ago

Anybody able to reproduce(重现) the deterministic results?有人能够重现确定性结果吗?

The same problem was encountered at least two weeks ago, but no logical explanation has been seen so far

SovereignRemedy commented 11 months ago

Some problems that output few tokens may have a high probability of stable output,But the odds are less than 50%

gaborcselle commented 11 months ago

I believe this is actually an API issue, as reported here: https://github.com/openai/openai-python/issues/850

mrbullwinkle commented 10 months ago

I believe this is actually an API issue, as reported here: openai/openai-python#850

If it is an API issue it isn't specific to the Python API as I am unable to get reproducible output when using the Python API as well with equivalent direct calls against the REST API. But I agree the cookbook is probably not the best place to get this issue triaged.

chrispy-snps commented 9 months ago

We have this issue too.

AayushSameerShah commented 9 months ago

I am still looking for the solution/fix/patch

In the mean time I have been following these threads:

  1. https://community.openai.com/t/seed-param-and-reproducible-output-do-not-work/487245
  2. https://community.openai.com/t/the-seed-option-for-gpt-does-not-increase-the-determinism-level/512892

Try subscribing for updates there, who knows when this gets fixed! 🤷🏻‍♂️

yongyaoduan commented 9 months ago

We have this issue too, who knows when this gets fixed?

AayushSameerShah commented 9 months ago

@yongyaoduan let me call Sam Altman, I will let you know.

chrispy-snps commented 9 months ago

Here is a script I wrote to plot the word similarity across N OpenAI responses for one or more LLM parameter configurations:

plot_openai_response_similarity.py

Here is an example plot for four LLM configurations, with N=100 responses for each one:

tell_me_a_story

The X axis indicates how far into the string is considered. The Y axis indicates similarity (higher is more similar).

The gist link has more details.

ross-nordstrom commented 7 months ago

Something to double check: the openai docs point out that seed is only supported on gpt-4-1106-preview and gpt-3.5-turbo-1106 https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter

That said, my anecdotal experience matches what chrispy-snps showed -- the beginning of my response is the same across queries, but I get variety further into the response

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

chrispy-snps commented 5 months ago

This is useful discussion and it would be nice to keep it open a bit longer.

boxabirds commented 5 months ago

Still an issue with gpt-3.5-turbo and both the OpenAI / Microsoft docs say it should work. It's a regression in the underlying API so I guess everyone here would like to know how to log bugs against that if there is one?

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

boxabirds commented 3 months ago

Seems like the seed parameter still needs to be removed or fixed.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

gaborcselle commented 1 month ago

@boxabirds did you still see this bug when you tried a few weeks ago?

boxabirds commented 1 month ago

I got a chat message from support that fingerprints had been accidentally disabled and were going again. But I’ve not tested it recently as I’m on other projects at the moment.

On Sun, 29 Sep 2024 at 03:36, Gabor Cselle @.***> wrote:

@boxabirds https://github.com/boxabirds did you still see this bug when you tried a few weeks ago?

— Reply to this email directly, view it on GitHub https://github.com/openai/openai-cookbook/issues/861#issuecomment-2381077933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABD62KX4RKSDHUQX4QAWDDZY5RTHAVCNFSM6AAAAAA7MKOQEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBRGA3TOOJTGM . You are receiving this because you were mentioned.Message ID: @.***>