search

huggingface / datasets

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools
https://huggingface.co/docs/datasets
Apache License 2.0
19.31k stars 2.7k forks source link

Cache only changed columns? #7138

Open Modexus opened 2 months ago

Modexus commented 2 months ago

Feature request

Cache only the actual changes to the dataset i.e. changed columns.

Motivation

I realized that caching actually saves the complete dataset again. This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

Your contribution

Is this even viable in the current architecture of the package? I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?

Modexus commented 2 months ago

so I guess a workaround to this is to simply remove all columns except the ones to cache and then add them back with concatenate_datasets(..., axis=1).

lhoestq commented 2 months ago

yes this is the right workaround. We're keeping the cache like this to make it easier for people to delete intermediate cache files