Do Cerebras tweaks belong here?

[AngainorDev]

I adapted the scripts to use the newly released, open source Cerebras model.

I trained a LoRA with Cerebras 1.3B so far, on Alpaca cleaned (1h on dual 3090) Base https://huggingface.co/cerebras/Cerebras-GPT-1.3B

I still have a little tokenizer related stuff to get right (eos).

Question: Are you interested in having this integrated into this repo, or does it belong to a distinct fork? There are a few differences, so this could need some refactorization in order to modularize and avoid too many conditions in the main script.

[tloen]

We should probably keep this repo to tracking the instruct-tuning of LLaMA, although a more general command-line PEFT wrapper would be interesting...

Have you gotten good results from the Cerebras weights? They seem frustratingly undertrained relative to e.g. LLaMA.

[AngainorDev]

Ok.

I focused on the smaller models for now, trying to get the tokenizer part right. There seem to be more differences between the different sizes models than with llama so far, explainable by their "optimal" scaling law and dataset for each one.

I'm training a LoRA on 2.7B now, but I'm still unsure I got it fully right.

First impression, yes this model seem below llama, but has the licence advantage and could be a good starting point for specific expert models. Another cons is it's encoder, very english focused, which means other languages do consume a lot of tokens vs english. Not so great for multilingual or localized models.

Still, open architectures and weights are rare enough to be precious, they deserve to get a chance.

[tloen]

Pythia is open-source too. Seems from the graph like it's more parameter-efficient if we're looking for fully FOSS solutions.

[AngainorDev]

Indeed. And with same bias toward english only.

No LoRA fine tuner for Pythia yet?