微调是否要装nccl?

[MathamPollard]

Traceback (most recent call last): File "finetune_cpm_bee.py", line 4, in import bmtrain as bmt File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 2, in from .init import init_distributed File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 8, in from . import nccl File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/init.py", line 4, in from . import _C as C ImportError: /home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast Traceback (most recent call last): File "finetune_cpm_bee.py", line 4, in import bmtrain as bmt File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 2, in from .init import init_distributed File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 8, in from . import nccl File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/init.py", line 4, in from . import _C as C ImportError: /home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast Traceback (most recent call last): File "finetune_cpm_bee.py", line 4, in import bmtrain as bmt File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 2, in from .init import init_distributed File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 8, in from . import nccl File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/init.py", line 4, in from . import _C as C ImportError: /home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast Traceback (most recent call last): File "finetune_cpm_bee.py", line 4, in import bmtrain as bmt File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 2, in from .init import init_distributed File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/init.py", line 8, in from . import nccl File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/init.py", line 4, in from . import _C as C ImportError: /home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/bmtrain/nccl/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 81486) of binary: /home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/bin/python Traceback (most recent call last): File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/bin/torchrun", line 11, in sys.exit(main()) File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(*args, **kwargs) File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main run(args) File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/mdisk2/tanjunwen/anaconda3/envs/cpmbee2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

[zh-zheng]

和 issue #18 相同