search

uclaml / SPIN

The official implementation of Self-Play Fine-Tuning (SPIN)
https://uclaml.github.io/SPIN/
Apache License 2.0
1.05k stars 92 forks source link

Cannot reproduce generated samples in UCLA-AGI/SPIN_iter0 #32

Open StarDewXXX opened 6 months ago

StarDewXXX commented 6 months ago

I use revision="ac6e600eefcce74f5e8bae1035d4f66019e93190" to generate data for iter0, but cannot reproduce generated samples in UCLA-AGI/SPIN_iter0. Since do_sample is default to be false, the generation should be the same. Below is an example:

######### How does the location of the Sydney Conservatorium of Music impact the academic and professional opportunities available to music students, and how does the conservatorium support student engagement with the music industry in Australia?

[reproduced with generate.py]: The location of the Sydney Conservatorium of Music in the heart of Sydney's central business district provides students with access to a wide range of academic and professional opportunities. The conservatorium is situated in the historic University of Sydney Quadrangle, ........(omitted)

[provided in UCLA-AGI/SPIN_iter0]: he Conservatorium is located in the heart of the city of Sydney, specifically in the central business district. This provides students with unique access to academic and professional opportunities within the city, including the opportunity to learn from top faculty members in the music ........(omitted)

AGTSAAA commented 6 months ago

I also have the same problem. I can not reproduce the generated samples.

angelahzyuan commented 6 months ago

Have you considered that generation results might vary across different machines? Even with identical settings, the outcomes could differ depending on the GPU or the number of GPUs used. Are there any performance differences?

angelahzyuan commented 6 months ago

Also, our dataset posted on hf were generated using the hf generation, not vllm.