ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}

[CLL112]

Describe the bug

When I run the code en = load_dataset("allenai/c4", "en", streaming=True), I encounter an error: raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}") ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}. However, running dataset = load_dataset('allenai/c4', streaming=True, data_files={'validation': 'en/c4-validation.00003-of-00008.json.gz'}, split='validation') works fine. What is the issue here?

Steps to reproduce the bug

run code： import os os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com' from datasets import load_dataset

en = load_dataset("allenai/c4", "en", streaming=True)

Expected behavior

Successfully loaded the dataset.

Environment info

• datasets version: 2.18.0
• Platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.17
• Python version: 3.8.19
• huggingface_hub version: 0.22.2
• PyArrow version: 15.0.2
• Pandas version: 2.0.3
• fsspec version: 2024.2.0

[xioatian1]

How do you solve it ?

[zouhuigang]

How do you solve it ?

Please check your Python environment and dataset version. I have just resolved the issue, which was caused by a Python environment switching error