search

karpathy / nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.
MIT License
37.49k stars 5.97k forks source link

Data cleaning for openwebtext #529

Open zzkzzkjsw opened 5 months ago

zzkzzkjsw commented 5 months ago

It seems that there are some "" and "\x00\x00" in openwebtext corpus, I find that drop these can get better loss.

def remove_empty_strings(example):
    if example['text']=='' or '\x00\x00' in example['text']:
        return False 
    return True

processed_dataset = dataset.filter(remove_empty_strings)