The code stopped running with no errors（Below is my log of running the code. The code stops at the last line and does not run down）：
...
Initializing [clip] visual encoder from ~/.cache/clip/ViT-L-14.pt [cuda]...
Visual encoder initialized.
Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00, 5.86s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:19<00:00, 6.53s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.96s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.71s/it]
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
[2024-01-24 16:43:41,259] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.3, git-hash=4e80e29, git-branch=HEAD
[2024-01-24 16:43:41,259] [INFO] [comm.py:619:init_distributed] Distributed backend already initialized
[2024-01-24 16:44:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/hongzhili/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/hzl/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -UCUDA_NO_HALF_OPERATORS -UCUDA_NO_HALF_CONVERSIONS -UCUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -c /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/hzl/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -DAVX512 -DENABLE_CUDA__ -DBF16_AVAILABLE -c /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.828491926193237 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.78138303756714 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.877806186676025 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.85369896888733 seconds
The code stopped running with no errors（Below is my log of running the code. The code stops at the last line and does not run down）： ... Initializing [clip] visual encoder from ~/.cache/clip/ViT-L-14.pt [cuda]... Visual encoder initialized. Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ... Visual encoder initialized. Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ... Visual encoder initialized. Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ... Visual encoder initialized. Initializing language decoder from /home/hongzhili/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ... Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00, 5.86s/it] Loading checkpoint shards: 100%|██████████| 3/3 [00:19<00:00, 6.53s/it] Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.96s/it] Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.71s/it] trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998 You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565 Language decoder initialized. LLaMa projection layer initialized. DeepSpeedAgent trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998 You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565 Language decoder initialized. LLaMa projection layer initialized. DeepSpeedAgent trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998 You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565 Language decoder initialized. LLaMa projection layer initialized. DeepSpeedAgent trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998 You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565 Language decoder initialized. LLaMa projection layer initialized. DeepSpeedAgent [2024-01-24 16:43:41,259] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.3, git-hash=4e80e29, git-branch=HEAD [2024-01-24 16:43:41,259] [INFO] [comm.py:619:init_distributed] Distributed backend already initialized [2024-01-24 16:44:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root... Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root... Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root... Using /home/hongzhili/.cache/torch_extensions/py310_cu118 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /home/hongzhili/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/hzl/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -UCUDA_NO_HALF_OPERATORS -UCUDA_NO_HALF_CONVERSIONS -UCUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -c /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o [2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/hzl/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -DAVX512 -DENABLE_CUDA__ -DBF16_AVAILABLE -c /data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o [3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/data2/hzl/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/lib64 -lcudart -o cpu_adam.so Loading extension module cpu_adam... Time to load cpu_adam op: 23.828491926193237 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 23.78138303756714 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 23.877806186676025 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 23.85369896888733 seconds