不能调用deepspeed，出现Segmentation Fault

root@dsw-541920-5fd5c64bc4-m25b4:/mnt/workspace/modelscope# xtuner train llama2_7b_chat_qlora_custom_sft_e1_copy.py --deepspeed deepspeed_zero1 [2024-07-01 21:43:15,368] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible 2024-07-01 21:43:16.651177: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variableTF_ENABLE_ONEDNN_OPTS=0. 2024-07-01 21:43:16.689173: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-07-01 21:43:17.345499: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT [2024-07-01 21:43:21,201] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible 2024-07-01 21:43:22.426802: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variableTF_ENABLE_ONEDNN_OPTS=0`. 2024-07-01 21:43:22.464095: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-07-01 21:43:23.116373: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 07/01 21:43:24 - mmengine - INFO -

System environment: sys.platform: linux Python: 3.10.14 (main, May 29 2024, 23:47:02) [GCC 11.4.0] CUDA available: True MUSA available: False numpy_random_seed: 1303059119 GPU 0: NVIDIA A10 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.1, V12.1.105 GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.3.0+cu121 PyTorch compiling details: PyTorch built with:

GCC 11.4
C++ Version: 201703
Intel(R) Math Kernel Library Version 2020.0.4 Product Build 20200917 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX512
CUDA Runtime 12.1
NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
CuDNN 8.9
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.18.0+cu121 OpenCV: 4.9.0 MMEngine: 0.10.4

Runtime environment: launcher: none randomness: {'seed': None, 'deterministic': False} cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: None deterministic: False Distributed launcher: none Distributed training: False GPU number: 1

07/01 21:43:24 - mmengine - INFO - Config: SYSTEM = '' accumulative_counts = 16 batch_size = 1 betas = ( 0.9, 0.999, ) custom_hooks = [ dict( tokenizer=dict( padding_side='right', pretrained_model_name_or_path= 'LLM-Research/Meta-Llama-3-8B-Instruct', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.DatasetInfoHook'), dict( evaluation_inputs=[ '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai', ], every_n_iters=500, prompt_template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', system='', tokenizer=dict( padding_side='right', pretrained_model_name_or_path= 'LLM-Research/Meta-Llama-3-8B-Instruct', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.engine.hooks.EvaluateChatHook'), ] data_files = [ './dataset/BBQ.json', './dataset/CrowS_Paris.json', ] dataloader_num_workers = 0 default_hooks = dict( checkpoint=dict( by_epoch=False, interval=500, max_keep_ckpts=2, type='mmengine.hooks.CheckpointHook'), logger=dict( interval=10, log_metric_by_epoch=False, type='mmengine.hooks.LoggerHook'), param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'), sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'), timer=dict(type='mmengine.hooks.IterTimerHook')) env_cfg = dict( cudnn_benchmark=False, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) evaluation_freq = 500 evaluation_inputs = [ '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai', ] launcher = 'none' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) lr = 0.0002 max_epochs = 1 max_length = 2048 max_norm = 1 model = dict( llm=dict( pretrained_model_name_or_path='LLM-Research/Meta-Llama-3-8B-Instruct', quantization_config=dict( bnb_4bit_compute_dtype='torch.float16', bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, llm_int8_has_fp16_weight=False, llm_int8_threshold=6.0, load_in_4bit=True, load_in_8bit=False, type='transformers.BitsAndBytesConfig'), torch_dtype='torch.float16', trust_remote_code=True, type='transformers.AutoModelForCausalLM.from_pretrained'), lora=dict( bias='none', lora_alpha=16, lora_dropout=0.1, r=64, task_type='CAUSAL_LM', type='peft.LoraConfig'), type='xtuner.model.SupervisedFinetune', use_varlen_attn=False) optim_type = 'torch.optim.AdamW' optim_wrapper = dict( optimizer=dict( betas=( 0.9, 0.999, ), lr=0.0002, type='torch.optim.AdamW', weight_decay=0), type='DeepSpeedOptimWrapper') pack_to_max_length = True param_scheduler = [ dict( begin=0, by_epoch=True, convert_to_iter_based=True, end=0.03, start_factor=1e-05, type='mmengine.optim.LinearLR'), dict( begin=0.03, by_epoch=True, convert_to_iter_based=True, end=1, eta_min=0.0, type='mmengine.optim.CosineAnnealingLR'), ] pretrained_model_name_or_path = 'LLM-Research/Meta-Llama-3-8B-Instruct' prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.llama3_chat' randomness = dict(deterministic=False, seed=None) resume = False runner_type = 'FlexibleRunner' save_steps = 500 save_total_limit = 2 strategy = dict( config=dict( bf16=dict(enabled=True), fp16=dict(enabled=False, initial_scale_power=16), gradient_accumulation_steps='auto', gradient_clipping='auto', train_micro_batch_size_per_gpu='auto', zero_allow_untested_optimizer=True, zero_force_ds_cpu_optimizer=False, zero_optimization=dict(overlap_comm=True, stage=1)), exclude_frozen_parameters=True, gradient_accumulation_steps=16, gradient_clipping=1, sequence_parallel_size=1, train_micro_batch_size_per_gpu=1, type='xtuner.engine.DeepSpeedStrategy') tokenizer = dict( padding_side='right', pretrained_model_name_or_path='LLM-Research/Meta-Llama-3-8B-Instruct', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained') train_cfg = dict(max_epochs=1, type='xtuner.engine.runner.TrainLoop') train_dataloader = dict( batch_size=1, collate_fn=dict( type='xtuner.dataset.collate_fns.default_collate_fn', use_varlen_attn=False), dataset=dict( dataset=dict( data_files=[ './dataset/BBQ.json', './dataset/CrowS_Paris.json', ], path='json', type='datasets.load_dataset'), dataset_map_fn='xtuner.dataset.map_fns.openai_map_fn', max_length=2048, pack_to_max_length=True, remove_unused_columns=True, shuffle_before_pack=True, template_map_fn=dict( template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', type='xtuner.dataset.map_fns.template_map_fn_factory'), tokenizer=dict( padding_side='right', pretrained_model_name_or_path= 'LLM-Research/Meta-Llama-3-8B-Instruct', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.dataset.process_hf_dataset', use_varlen_attn=False), num_workers=0, sampler=dict(shuffle=True, type='mmengine.dataset.DefaultSampler')) train_dataset = dict( dataset=dict( data_files=[ './dataset/BBQ.json', './dataset/CrowS_Paris.json', ], path='json', type='datasets.load_dataset'), dataset_map_fn='xtuner.dataset.map_fns.openai_map_fn', max_length=2048, pack_to_max_length=True, remove_unused_columns=True, shuffle_before_pack=True, template_map_fn=dict( template='xtuner.utils.PROMPT_TEMPLATE.llama3_chat', type='xtuner.dataset.map_fns.template_map_fn_factory'), tokenizer=dict( padding_side='right', pretrained_model_name_or_path='LLM-Research/Meta-Llama-3-8B-Instruct', trust_remote_code=True, type='transformers.AutoTokenizer.from_pretrained'), type='xtuner.dataset.process_hf_dataset', use_varlen_attn=False) use_varlen_attn = False visualizer = None warmup_ratio = 0.03 weight_decay = 0 work_dir = './work_dirs/llama2_7b_chat_qlora_custom_sft_e1_copy'

07/01 21:43:24 - mmengine - WARNING - Failed to search registry with scope "mmengine" in the "builder" registry tree. As a workaround, the current "builder" registry in "xtuner" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmengine" is a correct scope, or whether the registry is initialized. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 07/01 21:43:25 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook
(BELOW_NORMAL) LoggerHook

before_train: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DatasetInfoHook
(LOW ) EvaluateChatHook
(VERY_LOW ) CheckpointHook

before_train_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook

before_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook

after_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(LOW ) EvaluateChatHook
(VERY_LOW ) CheckpointHook

after_train_epoch: (NORMAL ) IterTimerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

before_val: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) DatasetInfoHook

before_val_epoch: (NORMAL ) IterTimerHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_val_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_val: (VERY_HIGH ) RuntimeInfoHook
(LOW ) EvaluateChatHook

after_train: (VERY_HIGH ) RuntimeInfoHook
(LOW ) EvaluateChatHook
(VERY_LOW ) CheckpointHook

before_test: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) DatasetInfoHook

before_test_epoch: (NORMAL ) IterTimerHook

before_test_iter: (NORMAL ) IterTimerHook

after_test_iter: (NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_test_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_test: (VERY_HIGH ) RuntimeInfoHook

after_run: (BELOW_NORMAL) LoggerHook

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Flattening the indices (num_proc=32): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26301/26301 [00:00<00:00, 87979.55 examples/s] Map (num_proc=32): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26301/26301 [00:00<00:00, 28599.51 examples/s] Map (num_proc=32): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1246/1246 [00:00<00:00, 2653.44 examples/s] 07/01 21:43:29 - mmengine - WARNING - Dataset Dataset has no metainfo. dataset_meta in visualizer will be None. low_cpu_mem_usage was None, now set to True since model is quantized. Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.22it/s] 07/01 21:43:33 - mmengine - INFO - Dispatch LlamaFlashAttention2 forward. Due to the implementation of the PyTorch version of flash attention, even when the output_attentions flag is set to True, it is not possible to return the attn_weights. [2024-07-01 21:43:35,021] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.14.4, git-hash=unknown, git-branch=unknown [2024-07-01 21:43:35,021] [INFO] [comm.py:637:init_distributed] cdb=None [2024-07-01 21:43:35,022] [INFO] [comm.py:652:init_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment... [dsw-541920-5fd5c64bc4-m25b4:02519] Process received signal [dsw-541920-5fd5c64bc4-m25b4:02519] Signal: Segmentation fault (11) [dsw-541920-5fd5c64bc4-m25b4:02519] Signal code: Address not mapped (1) [dsw-541920-5fd5c64bc4-m25b4:02519] Failing at address: 0x440000e9 [dsw-541920-5fd5c64bc4-m25b4:02519] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fbb41aa5520] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 1] /usr/lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Comm_set_errhandler+0x47)[0x7fbb2af1e6b7] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 2] /usr/local/lib/python3.10/site-packages/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so(+0x2fcb9)[0x7fb9e9b22cb9] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 3] /usr/local/bin/../lib/libpython3.10.so.1.0(PyModule_ExecDef+0x73)[0x7fbb41e61933] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 4] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1c16bd)[0x7fbb41e626bd] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 5] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1718df)[0x7fbb41e128df] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 6] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x472b)[0x7fbb41e0d65b] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 7] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 8] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x3af6)[0x7fbb41e0ca26] [dsw-541920-5fd5c64bc4-m25b4:02519] [ 9] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [10] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x5c6)[0x7fbb41e094f6] [dsw-541920-5fd5c64bc4-m25b4:02519] [11] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [12] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x787)[0x7fbb41e096b7] [dsw-541920-5fd5c64bc4-m25b4:02519] [13] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [14] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x787)[0x7fbb41e096b7] [dsw-541920-5fd5c64bc4-m25b4:02519] [15] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [16] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1629c6)[0x7fbb41e039c6] [dsw-541920-5fd5c64bc4-m25b4:02519] [17] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1bfaa2)[0x7fbb41e60aa2] [dsw-541920-5fd5c64bc4-m25b4:02519] [18] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_CallMethodIdObjArgs+0x10c)[0x7fbb41e6511c] [dsw-541920-5fd5c64bc4-m25b4:02519] [19] /usr/local/bin/../lib/libpython3.10.so.1.0(PyImport_ImportModuleLevelObject+0x296)[0x7fbb41e19666] [dsw-541920-5fd5c64bc4-m25b4:02519] [20] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1c7fac)[0x7fbb41e68fac] [dsw-541920-5fd5c64bc4-m25b4:02519] [21] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x171942)[0x7fbb41e12942] [dsw-541920-5fd5c64bc4-m25b4:02519] [22] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_Call+0x60)[0x7fbb41e18a80] [dsw-541920-5fd5c64bc4-m25b4:02519] [23] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x472b)[0x7fbb41e0d65b] [dsw-541920-5fd5c64bc4-m25b4:02519] [24] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [25] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x787)[0x7fbb41e096b7] [dsw-541920-5fd5c64bc4-m25b4:02519] [26] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x167493)[0x7fbb41e08493] [dsw-541920-5fd5c64bc4-m25b4:02519] [27] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1629c6)[0x7fbb41e039c6] [dsw-541920-5fd5c64bc4-m25b4:02519] [28] /usr/local/bin/../lib/libpython3.10.so.1.0(+0x1bfaa2)[0x7fbb41e60aa2] [dsw-541920-5fd5c64bc4-m25b4:02519] [29] /usr/local/bin/../lib/libpython3.10.so.1.0(_PyObject_CallMethodIdObjArgs+0x10c)[0x7fbb41e6511c] [dsw-541920-5fd5c64bc4-m25b4:02519] End of error message `

InternLM / xtuner

不能调用deepspeed，出现Segmentation Fault #798