esbatmop / MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化,也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。
MIT License
3.38k stars 231 forks source link

huggingface将下载好的数据下载到本地,通过本地加载的方式报错。 #44

Closed amanyara closed 10 months ago

amanyara commented 10 months ago

image Traceback (most recent call last): File "F:\Data-King\multi_dialogue.py", line 2, in dataset = load_from_disk("MNBVC") File "D:\Anaconda3\envs\zyh\lib\site-packages\datasets\load.py", line 1907, in load_from_disk f"Directory {dataset_path} is neither a Dataset directory nor a DatasetDict directory." FileNotFoundError: Directory MNBVC is neither a Dataset directory nor a DatasetDict directory.

from datasets import load_dataset, load_from_disk dataset = load_from_disk("MNBVC")

next(iter(dataset)) # get the first line

amanyara commented 10 months ago

已解决