search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
MIT License
11.25k stars 1k forks source link

Why not set random seed when running the inference? #367

Open Edwardmark opened 1 month ago

Edwardmark commented 1 month ago

Hi, thanks for your great work. I have found a strange thing, if I set the seed in the for loop, the video will be noise. And why you not setting random seed in inference? https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp_on_npu.py#L173 https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp.py#L210

Edwardmark commented 1 month ago

@LinB203 Looking forward to your reply.

apprivoiser commented 1 month ago

When you use sp, each device will generate noise for part of the frame (because it is split from the frame dimension). If you set the random seed, all devices will get the same noise.

Edwardmark commented 1 month ago

When you use sp, each device will generate noise for part of the frame (because it is split from the frame dimension). If you set the random seed, all devices will get the same noise.

It's not like that. Actually, I solve the problem by setting seed both in right after initialize_sequence_parallel_state (https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp.py#L210 using https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp_on_npu.py#L173 )and in the for loop within the function run_model_and_save_images (https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp.py#L121 using https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/opensora/sample/sample_t2v_sp_on_npu.py#L173). In this way, the same prompt will generate the same video without randomness.