huggingface / optimum-intel

🤗 Optimum Intel: Accelerate inference with Intel optimization tools
https://huggingface.co/docs/optimum/main/en/intel/index
Apache License 2.0
355 stars 99 forks source link

Remove default trust remove code for predefined datasets #768

Closed echarlaix closed 2 weeks ago

echarlaix commented 2 weeks ago

Removing trust_remote_code=True when loading a dataset from PREDEFINED_SD_DATASETS :

for now loading automatically the dataset conceptual_captions when specified in the quantization config will fail as trust_remote_code=True is needed for datasets with a python loading script for datasets>=v2.20.0

cc @lhoestq

HuggingFaceDocBuilderDev commented 2 weeks ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq commented 2 weeks ago

I opened https://huggingface.co/datasets/google-research-datasets/conceptual_captions/discussions/3 to not require trust_remote_code on conceptual_captions

echarlaix commented 2 weeks ago

Thanks a lot @lhoestq