Open brando90 opened 1 year ago
we don't currently have such a dataset on huggingface, but we will let you know if we decide to do so! One issue is that the weights are on the chunk level, meaning that we are weighting sampling probability for the tokenized examples (not the raw documents).
Is there a data set compatible with HF I may use?
dataset = load_dataset("c4", "en", streaming=True, split="train").with_format("torch") remove_columns = ["text", "timestamp", "url"] but instead have
dataset = load_dataset("doremi", "en", streaming=True, split="train").with_format("torch") remove_columns = ["text", "timestamp", "url"] thus automatically using the doremi weights?