使用镜像作为环境变量后部分数据在最新版 datasets 上无法正常下载

yucc-leon commented 7 months ago

干净的环境，Python=3.11，只安装了 datasets(==2.18.0)

from datasets import load_dataset
dataset = load_dataset("codeparrot/apps")

会提示

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniforge3/envs/test/lib/python3.11/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniforge3/envs/test/lib/python3.11/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniforge3/envs/test/lib/python3.11/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/home/xxx/miniforge3/envs/test/lib/python3.11/site-packages/datasets/load.py", line 1831, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
                                                                       ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

官方开发人员无法复现，由于该错误仅出现在读取数据文件前向服务器请求数据集信息的阶段，因此猜测问题出在镜像上（参考https://github.com/huggingface/datasets/issues/6760 中的反馈）

halfrot commented 7 months ago

+1 encounter the same issue here

padeoe commented 7 months ago

+1 encounter the same issue here

@halfrot huggingface_hub版本是多少，请尝试最新版这个问题应该修复了

halfrot commented 7 months ago

huggingface_hub 0.22.2 是最新版的

alex-hek commented 7 months ago

我曾经遇到过类似的问题可能是压缩包解码的问题，通过pip install py7zr解决

rangehow commented 5 months ago

大佬有新进展吗？ @padeoe

我的datasets版本是2.19.2 hub版本是2.23.0。加载的数据集是 EleutherAI/drop，也碰到同样的问题

padeoe commented 5 months ago

已经有一些进展了，还在看

rangehow commented 5 months ago

已经有一些进展了，还在看

感谢大佬的无私奉献，你们的工作对国内AI研究起到了很大的帮助~

xuyuzhuang11 commented 5 months ago

pip install py7zr

此法并不能解决该问题

xuyuzhuang11 commented 5 months ago

请参考https://github.com/huggingface/datasets/issues/6760这个issue 把datasets换成2.14.6即可解决 :-)

yucc-leon commented 5 months ago

另一种报错：加载数据集 code_search_net 时，子集下载和校验过程中仍提示超时

Downloading builder script: 8.44kB [00:00, 50.2MB/s]
Downloading readme: 12.9kB [00:00, 73.3MB/s]
Downloading data: 100%|███████████████████████████████████████████████████████████████████| 941M/941M [00:39<00:00, 23.6MB/s]
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
....
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:
......

ReadTimeout: HTTPSConnectionPool(host='cdn-lfs.hf-mirror.com', port=443): Read timed out. (read timeout=100.0)

降级到 2.14.6 及更低版本也可暂时解决此问题。