Add deepspeed version of LR range test experiment

numenta / nupic.research

Experimental algorithms. Unsupported.

https://nupicresearch.readthedocs.io

GNU Affero General Public License v3.0

104 stars 60 forks source link

Add deepspeed version of LR range test experiment #522

Closed lscheinkman closed 3 years ago

lscheinkman commented 3 years ago

This lr-range test is based on bert_sparse_100k_kd_lr_range_test and adapted for deepspeed training. With this test the best max_lr value found is 0.0017