feat: filter dataset to keep only samples ending with eos token - Githubissues

jina-ai / jerboa

LLM finetuning

Apache License 2.0

42 stars 4 forks source link

feat: filter dataset to keep only samples ending with eos token #95

Open alaeddine-13 opened 1 year ago

alaeddine-13 commented 1 year ago

closes: https://github.com/jina-ai/jerboa/issues/73 training run: https://wandb.ai/jina-ai/jerboa/runs/cxlonnjz?workspace=user-jinaai