按照前序预处理成json格式后,运行tokenizer_dataset_row.py报错:
dataset=datasets.Dataset.from_generator(..)
...
file"/usr/local/python3.8.3/lib/python3.8/lib/python3.8/site-packages/datasets/builder.py", line 1644, in _prepare_split_single datasets.builder.datasetgeneraationerror: an error occurred wile generating the dataset.
按照前序预处理成json格式后,运行tokenizer_dataset_row.py报错: dataset=datasets.Dataset.from_generator(..) ... file"/usr/local/python3.8.3/lib/python3.8/lib/python3.8/site-packages/datasets/builder.py", line 1644, in _prepare_split_single datasets.builder.datasetgeneraationerror: an error occurred wile generating the dataset.