jquesnelle / yarn

YaRN: Efficient Context Window Extension of Large Language Models
MIT License
1.32k stars 115 forks source link

OSError: [Errno 28] No space left on device #18

Closed goog closed 1 year ago

goog commented 1 year ago
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 136/136 [00:00<00:00, 296941.88it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.27s/it]
/root/miniconda3/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
  warnings.warn(
/root/miniconda3/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
  warnings.warn(
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 136/136 [00:00<00:00, 284501.42it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.01it/s]
/root/miniconda3/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
  warnings.warn(
/root/miniconda3/lib/python3.8/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
  warnings.warn(
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 136/136 [00:00<00:00, 121937.87it/s]
Generating train split: 61410 examples [01:49, 561.64 examples/s]
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1940, in _prepare_split_single
    writer.write_table(table)
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/arrow_writer.py", line 577, in write_table
    self.pa_writer.write_table(pa_table, writer_batch_size)
  File "pyarrow/ipc.pxi", line 525, in pyarrow.lib._CRecordBatchWriter.write_table
  File "/root/miniconda3/lib/python3.8/site-packages/fsspec/implementations/local.py", line 365, in write
    return self.f.write(*args, **kwargs)
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "finetune.py", line 193, in <module>
    main(args.parse_args())
  File "finetune.py", line 67, in main
    train_dataset = load_dataset('/root/autodl-tmp/data/emozilla___pg_books-tokenized-bos-eos-chunked-65536/default/0.0.0/9107755b15521c04', split='train',
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 2136, in load_dataset
    builder_instance.download_and_prepare(
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1813, in _prepare_split
goog commented 1 year ago

i make HF_DATASETS_CACHE disk more larger , then solved.