search

threestudio-project / threestudio

A unified framework for 3D content generation.
Apache License 2.0
6.32k stars 480 forks source link

Reproducibility #219

Open thuliu-yt16 opened 1 year ago

thuliu-yt16 commented 1 year ago

Although we have a configured seed for each experiment, I still observe different results from the same configuration. I suspect there exist some modules out of the seed_everything control. Does anyone have any inspiration for this?

shinya7y commented 1 year ago

Other arguments will be needed.

trainer:
    deterministic: True
    benchmark: False

https://github.com/Lightning-AI/lightning/blob/6d888b5ce081277a89dc2fb9a2775b81d862fe54/src/lightning/pytorch/trainer/trainer.py#L239-L248 https://github.com/Lightning-AI/lightning/blob/6d888b5ce081277a89dc2fb9a2775b81d862fe54/src/lightning/pytorch/trainer/connectors/accelerator_connector.py#L655 https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html

Some modules, such as CLIP, may use non-deterministic algorithms. https://github.com/openai/CLIP/issues/13

thuliu-yt16 commented 1 year ago

Thanks, I will try it!

thuliu-yt16 commented 1 year ago

Some CUDA operations, such as upsample2D, do not have a deterministic algorithm so far. So I will just leave the issue open until pytorch has some progress in its deterministic algorithm.