New failure when loading dataset-- 500 from hub

abertsch72 commented 1 month ago

Describe the bug

A dataset that loaded fine yesterday is now failing on load; I can still load other datasets, and this failure occurs across two different machines I tried. It seems to be getting a 500 from the hub when it's requesting the dataset info.

(Apologies if this is already a known issue, or if this belongs elsewhere! Didn't see any relevant issues when I skimmed, and it seems to be a hub problem not a datasets problem.)

Reproduction

from datasets import load_dataset load_dataset("yaofu/slimpajama-per-source-length-upsample")

Logs

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/datasets/load.py", line 1824, in dataset_module_factory
    raise e
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/datasets/load.py", line 1797, in dataset_module_factory
    dataset_info = hf_api.dataset_info(
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2364, in dataset_info
    hf_raise_for_status(r)
  File "/miniconda3/envs/longcontext/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/yaofu/slimpajama-per-source-length-upsample (Request ID: Root=1-66a3e8cb-10c7ada135a5749b22edee3b;95bd088b-3ef3-4f10-b3e9-5c3ee711ddc2)

Internal Error - We're working hard to fix this as soon as possible!

System info

- huggingface_hub version: 0.23.4
- Platform: Linux-5.15.0-1063-aws-x86_64-with-glibc2.31
- Python version: 3.10.14
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /data/home/abertsch/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: abertsch
- Configured git credential helpers: store
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.3.1
- Jinja2: 3.1.4
- Graphviz: 0.20.3
- keras: N/A
- Pydot: N/A
- Pillow: 10.3.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.4
- pydantic: 2.7.4
- aiohttp: 3.9.5
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /data/home/abertsch/.cache/huggingface/hub
- HF_ASSETS_CACHE: /data/home/abertsch/.cache/huggingface/assets
- HF_TOKEN_PATH: /data/home/abertsch/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10