Closed winglian closed 3 weeks ago
Hi @winglian ! Thanks for the great suggestions! This is exactly our next move. Will update in a week or so.
Hi @winglian !
We have done the pip installization. You can now use as follows (as you may see in the updated readme).
git clone https://github.com/zyushun/Adam-mini
pip install -e .
from adam_mini import Adam_mini
optimizer = Adam_mini(
named_parameters = model.named_parameters(),
lr = lr,
betas = (beta1,beta2),
eps = eps,
weight_decay = weight_decay,
model_sharding = True,
dim = model_config.dim,
n_heads = model_config.n_heads,
n_kv_heads = model_config.n_kv_heads,
)
Thanks for your suggestion!
Thank you!
Can you make this pip installable from pypi? Makes it hard to integrate with other libraries with axolotl. Thanks!