Noob question: Why do I get the same text generation despite passing a different seed?

[ModPhoenix]

I tried passing rand::thread_rng() but it didn't help at all 🥲. Is this a bug or an issue with me 😅?

    let mut conversation = vec![
        "This is a conversation between two AI models.".to_string(),
        "Llama AI: Hello, Alpaca AGI! How are you today?".to_string(),
        "Alpaca AI: I'm doing great!".to_string(),
    ];

    loop {
        let mut session = model.start_session(repeat_last_n);

        let current_turn = conversation.len() % 2;
        let prompt = &conversation.join("\n");

        let response_text = RefCell::new(String::new());

        println!("Seed: {}", seed);

        let mut rng = rand::rngs::StdRng::seed_from_u64(seed); // Use a fixed seed for reproducibility

        let res = session.inference_with_prompt::<Infallible>(
            &model,
            &vocab,
            &inference_params,
            &prompt,
            None,
            &mut rng,
            |t| {
                match t {
                    OutputToken::Token(str) => {
                        print!("{t}");

                        response_text.borrow_mut().push_str(str);
                    }
                    OutputToken::EndOfText => {
                        println!("");
                        eprintln!("End of text");
                    }
                }

                std::io::stdout().flush().unwrap();
                Ok(())
            },
        );

The code is here if anyone wants to take a look https://github.com/ModPhoenix/beyond-human/blob/main/src/main.rs#L57

[saites]

You set top_p to 0. As a consequence, you will always choose the token that the model predicts has the highest probability.

Try setting that to 0.9, for example.

[ModPhoenix]

@saites that helped, thank you very much. I experimented with the values and it seems that top_p: 0.3 is enough to have a different dialogue every time, but the topics of the dialogues are quite the same.