[PROBLEM] Inconsistent Deterministic Outputs with Seed Parameter in OpenAI API

[hengqujushi]

Identify the file to be fixed N/A - This issue is related to the OpenAI API's deterministic behavior when using the seed parameter.

Describe the problem I am encountering an issue where I receive inconsistent outputs from the OpenAI API despite setting the same seed parameter for deterministic behavior. This is unexpected as the seed parameter is intended to ensure reproducibility of results. Both the System Fingerprint and the Seed values are identical for different requests, but the responses differ.

Describe a solution A potential solution would be to investigate the determinism functionality within the OpenAI API when the seed parameter is set. Ensuring that the outputs are consistent across multiple requests with the same seed would resolve this issue.

Screenshots

Additional context

• The code is being run using the following notebook: https://github.com/openai/openai-cookbook/blob/main/examples/Deterministic_outputs_with_the_seed_parameter.ipynb
• The issue persists across multiple attempts and different times, suggesting it is not an intermittent issue.
• No concurrent requests were made that could affect the outcome.

[tsilva]

I confirm that the same issue is happening to me as well.

• model: gpt-3.5-turbo-1106
• temperature: 0
• seed: 12345

Getting different responses each time, while getting the same system_fingerprint.

[lionelchg]

Same here using the parameters from the original notebook

[jd-codelink]

Anybody able to reproduce the deterministic results?

[SovereignRemedy]

Anybody able to reproduce(重现) the deterministic results?有人能够重现确定性结果吗？

The same problem was encountered at least two weeks ago, but no logical explanation has been seen so far

[SovereignRemedy]

Some problems that output few tokens may have a high probability of stable output，But the odds are less than 50%

[gaborcselle]

I believe this is actually an API issue, as reported here: https://github.com/openai/openai-python/issues/850

[mrbullwinkle]

I believe this is actually an API issue, as reported here: openai/openai-python#850

If it is an API issue it isn't specific to the Python API as I am unable to get reproducible output when using the Python API as well with equivalent direct calls against the REST API. But I agree the cookbook is probably not the best place to get this issue triaged.

[chrispy-snps]

We have this issue too.

[AayushSameerShah]

I am still looking for the solution/fix/patch

In the mean time I have been following these threads:

1. https://community.openai.com/t/seed-param-and-reproducible-output-do-not-work/487245
2. https://community.openai.com/t/the-seed-option-for-gpt-does-not-increase-the-determinism-level/512892

Try subscribing for updates there, who knows when this gets fixed! 🤷🏻‍♂️

[yongyaoduan]

We have this issue too, who knows when this gets fixed?

[AayushSameerShah]

@yongyaoduan let me call Sam Altman, I will let you know.

[chrispy-snps]

Here is a script I wrote to plot the word similarity across N OpenAI responses for one or more LLM parameter configurations:

plot_openai_response_similarity.py

Here is an example plot for four LLM configurations, with N=100 responses for each one:

The X axis indicates how far into the string is considered. The Y axis indicates similarity (higher is more similar).

The gist link has more details.

[ross-nordstrom]

Something to double check: the openai docs point out that seed is only supported on gpt-4-1106-preview and gpt-3.5-turbo-1106 https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter

That said, my anecdotal experience matches what chrispy-snps showed -- the beginning of my response is the same across queries, but I get variety further into the response

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[chrispy-snps]

This is useful discussion and it would be nice to keep it open a bit longer.

[boxabirds]

Still an issue with gpt-3.5-turbo and both the OpenAI / Microsoft docs say it should work. It's a regression in the underlying API so I guess everyone here would like to know how to log bugs against that if there is one?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[boxabirds]

Seems like the seed parameter still needs to be removed or fixed.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[gaborcselle]

@boxabirds did you still see this bug when you tried a few weeks ago?

[boxabirds]

I got a chat message from support that fingerprints had been accidentally disabled and were going again. But I’ve not tested it recently as I’m on other projects at the moment.

On Sun, 29 Sep 2024 at 03:36, Gabor Cselle @.***> wrote:

@boxabirds https://github.com/boxabirds did you still see this bug when you tried a few weeks ago?

— Reply to this email directly, view it on GitHub https://github.com/openai/openai-cookbook/issues/861#issuecomment-2381077933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABD62KX4RKSDHUQX4QAWDDZY5RTHAVCNFSM6AAAAAA7MKOQEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBRGA3TOOJTGM . You are receiving this because you were mentioned.Message ID: @.***>