顺利部署,但微调时出错,报错如下:
Traceback (most recent call last):
File "/glm2/code/ChatGLM2-6B-main/ptuning/main.py", line 411, in
main()
File "/glm2/code/ChatGLM2-6B-main/ptuning/main.py", line 98, in main
raw_datasets = load_dataset(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
builder_instance.download_and_prepare(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/datasets/builder.py", line 810, in download_and_prepare
raise OSError(
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 28466) of binary: /root/miniconda3/envs/glm2/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/glm2/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Is there an existing issue for this?
Current Behavior
顺利部署,但微调时出错,报错如下: Traceback (most recent call last): File "/glm2/code/ChatGLM2-6B-main/ptuning/main.py", line 411, in
main()
File "/glm2/code/ChatGLM2-6B-main/ptuning/main.py", line 98, in main
raw_datasets = load_dataset(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
builder_instance.download_and_prepare(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/datasets/builder.py", line 810, in download_and_prepare
raise OSError(
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 28466) of binary: /root/miniconda3/envs/glm2/bin/python
Traceback (most recent call last):
File "/root/miniconda3/envs/glm2/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/glm2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
main.py FAILED
Failures: