Open xslittlegrass opened 4 months ago
Similar issue in text process
tokenizer=AutoTokenizer.from_pretrained(model_dir[args.model])
train_dataset=datasets.load_from_disk(dataset_dir[args.dataset],keep_in_memory=True)['train']
train_dataset=train_dataset.map(partial(dname2func[args.dataset],tokenizer=tokenizer),batched=True,num_proc =50,remove_columns=train_dataset.features.keys(),desc='tokenize',keep_in_memory=True)
After this train_dataset will be like
Dataset({
features: ['input_ids', 'labels'],
num_rows: 51760
})
In which input_ids and labels are both List[int] However, per iter on dataset cost 7.412479639053345s ……?
for j in tqdm(range(len(train_dataset)),desc='first stage'):
input_id,label=train_dataset['input_ids'][j],train_dataset['labels'][j]
The transform currently replaces the numpy formatting.
So you're back to copying data to long python lists which is super slow.
It would be cool for the transform to not remove the formatting in this case, but this requires a few changes in the lib
Describe the bug
Dataset is 10X slower when applying trivial transforms:
In my real code I'm using set_transform to apply some post-processing on-the-fly for the 2d array, but it significantly slows down the dataset even if the transform itself is trivial.
Related issue: https://github.com/huggingface/datasets/issues/5841
Steps to reproduce the bug
Use code in the description to reproduce.
Expected behavior
Trivial custom transform in the example should not slowdown the dataset iteration.
Environment info
datasets
version: 2.18.0huggingface_hub
version: 0.20.2fsspec
version: 2023.12.2