search

willisma / SiT

Official PyTorch Implementation of "SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers"
https://scalable-interpolant.github.io/
MIT License
662 stars 35 forks source link

Wandb issue #7

Closed stefenmax closed 2 months ago

stefenmax commented 8 months ago

I manually set the entity and project name in the train file. And I've encountered an issue with Weights & Biases where the wandb.run.step doesn't seem to reset to 0 between training sessions. This has led to a situation where, after multiple training sessions, I receive warnings indicating that the step value provided is less than the current step, suggesting that wandb.run.step is continuing from the last recorded value rather than starting anew. image Could you please provide guidance on how to ensure that wandb.run.step correctly resets at the start of each training session? Is there a recommended approach to managing the step counter across sessions to avoid this issue?

Thank you for your assistance.

willisma commented 2 months ago

We did not encounter this issue before, so please consult the wandb official documents and let us know if there's been a misconfiguration in our code.