mymusise / ChatGLM-Tuning

基于ChatGLM-6B + LoRA的Fintune方案
MIT License
3.71k stars 443 forks source link

error:raise DatasetGenerationError("An error occured while generating the dataset) #248

Open WindFlowUpTheMoon opened 1 year ago

WindFlowUpTheMoon commented 1 year ago

image xplicitly passing a `revision’is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in revisior 0%| 0/52002 [00:00<?. ?it/s] enerating train split:@ examples [00:03,? examples/s]Traceback (most recent call last): 0/52002 [00:00<?,?it/s] File "/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/builder.py", line 1608, in _prepare_split_single for key, record in generator: File "/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/packaged modules/generator/generator.py", line 30, in _generate_examples for idx, ex in enumerate(self.config.generator(gen_kwargs)): File "tokenize_dataset_rows.py", line 31, in read_jsonl feature = preprocess(tokenizer, config, example, max_seq_length) File "tokenize_dataset_rows.py", line 10, in preprocess prompt = example["context"] teyError: 'context' The above exception was the direct cause of the following exception: fraceback (most recent call last): File "tokenize_dataset_rows.py", line 53, in main() File "tokenize dataset rows.py", line 46, in main dataset = datasets.Dataset.from generatorl File "/root/.conda/envs/tunina/lib/pvthon3.8/site-packaaes/datasets/arrow dataset.oy". line 1012. in from aenerator return GeneratorDatasetInputstream File "/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/io/generator.py", line 47, in read self.builder.download_and_prepare( File "/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/builder.py", line 872, in download_and_prepare self._download_and_prepare( File"/root/.conda/envs/tuning/1ib/python3.8/site-packages/datasets/builder.py",line 1649, in _download_and_prepare super()._download_and_prepare( File "/root/.conda/envs/tuning/1ib/python3.8/site-packages/datasets/builder.py", line 967, in _download_and _prepare self._prepare_split(split_generator, prepare_split_kwargs) File"/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/b ouilder.py", line 1488, in _prepare_split for job_id, done, content in self._prepare_split_single( File"/root/.conda/envs/tuning/lib/python3.8/site-packages/datasets/b ouilder.py", line 1644, in _prepare_split_single raise DatasetGenerationError("An error occurred while generating th he dataset") from datasets.builder.DatasetGenerationError: An error occurred while genera ting the dataset

pengcheng-yan commented 8 months ago

最后解决了没 我也碰到了这个问题