[BUG] GradScaler does not work with torch 2.3.0

learning-at-home / hivemind

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

MIT License

1.95k stars 154 forks source link

Open samsja opened 2 months ago

samsja commented 2 months ago

Describe the bug

GradScaler is not working anymore with pytorch 2.3.0

This

To Reproduce

from torch.cuda.amp.grad_scaler import OptState, _refresh_per_optimizer_state

this is failing now with pytorch 2.3. Which make it impossible to import hivemind GradScaler because of this line

Environment Please list:

rohrl commented 2 months ago