`ValueError: algorithm not supported` when running gpt2 example

zyushun / Adam-mini

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

321 stars 10 forks source link

`ValueError: algorithm not supported` when running gpt2 example #20

Open Benjamin-eecs opened 3 months ago

Benjamin-eecs commented 3 months ago

run bash run_gpt2.sh and raise the value error above.

zyushun commented 2 months ago

@Benjamin-eecs Hi! Sorry for the late response.

Perhaps you need to check the name of optimizer in the config file https://github.com/zyushun/Adam-mini/blob/main/examples/gpt2/config/train_gpt2_small.py

Make sure that you are using the following and I think it would be fine.

# optimizer
algorithm = 'adam_mini'

Sorry again for the delay.

Sun2018421 commented 3 weeks ago

elif algorithm == 'adam_mini': optimizer = Adam_mini( named_parameters=model.named_parameters(), lr=lr, betas=(beta1, beta2), weight_decay=weight_decay, model_sharding=False, dim=n_embd, n_heads=n_head ) raise ValueError("algorithm not supported") I commented out the raise part of the code, And it works.