Open the-beee opened 1 year ago
Hi! Thanks for the detailed report! It is indeed a bug, and we'll fix it in the nearest release. In the meantime, i'm afrain that the only override is to keep float32 params with hivemind.Optimizer - while the on-device model is in bfloat16.
Describe the bug
To Reproduce
Please update
config/distributed.yaml
to include the peers address in thehivemind
section, before starting the second peer.Environment