huggingface / nanotron

Minimalistic large language model 3D-parallelism training

Apache License 2.0

1.23k stars 122 forks source link

[Feature] DoReMi #34

Closed xrsrke closed 9 months ago

xrsrke commented 10 months ago

[x] DoReMi implementation
[ ] Validate the implementation with wikicorpus dataset

obrienwrite commented 9 months ago

xrsrke commented 9 months ago

@obrienwrite ?

obrienwrite commented 9 months ago

bordingschool

On Sat, Jan 27, 2024, 6:38 PM XλRI-U5 @.***> wrote:

@obrienwrite https://github.com/obrienwrite ?

— Reply to this email directly, view it on GitHub https://github.com/huggingface/nanotron/pull/34#issuecomment-1913421531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAN35N4SFYJHFUSPGVPKRDYQW233AVCNFSM6AAAAABCDZ5V42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTGQZDCNJTGE . You are receiving this because you were mentioned.Message ID: @.***>