search

mingkaid / rl-prompt

Accompanying repo for the RLPrompt paper
MIT License
286 stars 52 forks source link

Reproducibility and randomness #38

Closed YasamanJafari closed 8 months ago

YasamanJafari commented 9 months ago

Hi,

Thank you so much for the great repo. I noticed that even when the random seeds are set to the same value across the runs, the results still differ. Runs start from the same random state but start to deviate from one another. Is there a reason why this behavior happens?

Thanks!

MM-IR commented 8 months ago

Hi, thanks for this question.

Not sure whether you have ensured all random seeds in our implementation are fixed during your experiments. That could be the only mattered key factor, as random seeds should be guaranteed to be reproducible across runs in the same machine.

If you just specify the seeds through our provided command, there may exist some other seeds not fixed, I guess..

In addition, it is really hard for me to judge under your cases, since we are not sure how you pick the prompts for random seed comparisons.