Batch size scaling - Githubissues

iejMac / encoder-distill

Align embedding spaces of PyTorch encoders with common input types.

MIT License

4 stars 0 forks source link

Open iejMac opened 2 years ago

iejMac commented 2 years ago

2k batch size -> 4k batch size looks good

iejMac commented 2 years ago

lr 2e-4 -> 3e-4

look into 4k -> 16k lr 3e-4 -> 5e-4

iejMac commented 2 years ago

4k -> 16k lr 3e-4 -> 7e-4

iejMac commented 2 years ago

16k -> 64k on 256 GPU's